CN109783189B - Static workflow scheduling method and device - Google Patents
Static workflow scheduling method and device Download PDFInfo
- Publication number
- CN109783189B CN109783189B CN201811492446.5A CN201811492446A CN109783189B CN 109783189 B CN109783189 B CN 109783189B CN 201811492446 A CN201811492446 A CN 201811492446A CN 109783189 B CN109783189 B CN 109783189B
- Authority
- CN
- China
- Prior art keywords
- current layer
- virtual machine
- budget
- module
- modules
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a static workflow scheduling method and a static workflow scheduling device. The invention carries out hierarchical division according to the structure of the workflow, considers the type of the virtual machine and allocates the virtual machine based on budget for the workflow module of each layer, and then carries out mapping scheduling of the physical machine aiming at the allocated virtual machine. The method solves the performance bottleneck of the workflow in the heterogeneous cloud environment, and overcomes the defect of insufficient throughput of the workflow in the heterogeneous cloud environment.
Description
Technical Field
The invention belongs to the field of big data and cloud computing, and particularly relates to a static workflow scheduling method and device under a multi-cloud environment.
Background
Currently, many applications in the field of natural science are generating more and more data at a steady rate, and for the processing problem of these large amounts of data, workflow technology in cloud computing is often the basic approach to solve, however, there is still a challenge to analyze and process such large-scale data in a timely and efficient scheme. Especially, with the rapid increase of data dimension and scale, the conventional High-Performance Computing (HPC) system has difficulty in meeting the resource requirements of different degrees due to the complex structure thereof.
With the advent and the gradual maturity of cloud computing, computing resources can be elastically allocated according to requirements, and a user only needs to care about the amount of resources required by a corresponding application and the price of the type of the resources, so that many application workflows are migrated to a cloud environment. There are three main current cloud service modes: infrastructure as a service (IaaS), software and services (SaaS), and platform as a service (PaaS). IaaS is typically used to service scientific workflows due to its resource deployment flexibility.
However, the traditional cloud service providers do not consider the resource use heterogeneity brought by the increase of hardware devices, which greatly reduces the operating efficiency of workflows, and for a workflow structure with a larger scale, one problem that must be faced is to process in a multi-cloud multi-data center, and the hardware heterogeneity characteristics of the data center and the data transmission situation between the data centers greatly limit the performance of the whole workflow, and for a high-throughput system in the field of scientific computing, the overall system performance is limited. On the other hand, from the perspective of cloud consumers, different running instances need different prices, so that how to complete the data to be processed as fast as possible under a certain budget constraint is also one of the problems to be considered, and the traditional method does not consider the budget constraint, and often causes more economic resource waste in order to maximally reduce the overall completion time.
Existing solutions either increase the performance of the running instance, which in turn results in higher economic costs, or decrease the performance of the running instance, which in turn affects the operating efficiency of the overall system.
Disclosure of Invention
Aiming at the defects or shortcomings of the prior art, the invention provides a static workflow scheduling method.
The method provided by the invention is used for allocating virtual machines and selecting physical machines under the environment of a plurality of cloud data centers, and comprises the following steps:
step one, the workflow is layered
To workflow G w (V w ,E w ) Layering, wherein the modules in the same layer have no dependency relationship, V w As a set of modules, E w A set of all module dependencies; after layering, the sum of all module workloads in the first layer is CR l (ii) a L =1,2,3.. L, L ≧ 2, the budget of the L-th layer is
CR is the sum of the workloads of all modules in the workflow;
step two, virtual machine and physical machine allocation
And (3) allocating virtual machines and physical machines to the workflow layer by layer:
(1) The virtual machine allocation to the current layer comprises: allocating virtual machine categories to modules in a current layer according to the worst level to the optimal level of the virtual machine category performances of a plurality of cloud centers, wherein each allocation is performed once, the total calculation cost of the current layer is compared with the budget size of the current layer once, if the total calculation cost of the current layer is less than the budget of the current layer, a new virtual machine category is continuously allocated to the modules in the current layer, if the total calculation cost of the current layer is more than the budget of the current layer, the last allocated virtual machine category is allocated to the modules in the current layer and allocation is completed, and if the total calculation cost of the current layer is equal to the budget of the current layer, allocation is completed; the total calculation cost of the current layer is the sum of the costs required by the modules in the current layer to finish calculation by utilizing the current virtual machine types;
(2) The physical machine selection of the current layer comprises the following steps: allocating all modules to corresponding physical machines of the cloud data centers according to a virtual machine scheme allocated to the current layer, selecting bandwidth for the current layer according to the worst level to the optimal level of bandwidth performance of the plurality of cloud data centers, comparing the total cost of the current layer with the budget size of the current layer once selection, and if the total cost of the current layer is less than the budget of the current layer, continuing to select a new level of bandwidth for the modules in the current layer; if the total cost of the current layer is larger than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection; the total cost of the current layer is the sum of the cost required by each dependency relationship in the current layer to transmit data by using the currently selected bandwidth.
Preferably, the worst level to the best level of the virtual machine classification performance of the multiple cloud centers are sorted and graded according to the sequence of Disk, memory and CPU, and the same parameter is compared and sorted, and the larger the value of the same parameter is, the better the performance is.
Preferably, the worst level to the best level of the bandwidth performance of the plurality of cloud data centers are ranked according to the bandwidth numerical value.
Preferably, the process of allocating the virtual machine to the current layer includes:
2.1 adding a virtual output module to the current layer, wherein the workload and the transmission data size of the virtual output module are both 0;
2.2 allocating the virtual machine instance of the worst virtual machine type to all the modules of the current layer, and calculating any module w in the current layer i Completion time T to virtual output module i I is not less than 1,Ti = T ime +T idt ， T ime Is a module w i Ratio of workload to calculated force of worst virtual machine type, T idt Is a module w i Inputting the time of data transmission; the module with the longest completion time in all modules of the current layer is a key module;
2.3, carrying out priority sequencing on the modules in the current layer, wherein the priority of the key module is the highest, and sequencing the rest modules in sequence, wherein the higher the workload is, the higher the priority is;
2.4 calculate the total completion time T needed by all modules of the current layer to calculate by using the worst virtual machine type l And calculating the total calculation cost C l ；
2.5 Total computation cost C if Current layer l If the total calculation cost of the current layer after upgrading is less than the budget of the current layer, updating the key module according to the ratio of the workload of each module in the current layer to the calculation power of each current virtual machine type, wherein the module with the largest ratio is a new key module; then, carrying out virtual machine type upgrading on the new key module, then comparing the total calculation cost of the current layer after upgrading with the budget size of the current layer again, if the total calculation cost of the current layer after upgrading is still smaller than the budget of the current layer, continuing to update the key module until the total calculation cost of the current layer is larger than the budget given by the current layer, and carrying out virtual machine type degradation on the current key module; complete virtual machine allocation, or total computational cost up to the current layer, etcThe virtual machine allocation is completed when the budget given by the current layer.
Preferably, the process of selecting the physical machine for the current layer includes:
2.6 distributing all modules to the physical machines of the corresponding cloud data center according to the selected virtual machine distribution scheme;
2.7 calculating the cost required by each dependency relationship in the current layer to transmit data by using the worst performance bandwidth to obtain the total cost of the current layer;
2.8 comparing the total cost of the current layer with the budget of the current layer, if the total cost of the current layer is less than the budget of the current layer, selecting a new level of bandwidth for the current layer and then continuously comparing the sizes of the current layer and the budget of the current layer, if the total cost of the current layer is greater than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection.
Preferably, in the physical machine selection process, modules with dependency relationships are placed in the same cloud under the condition that cloud data center resources are met.
In some embodiments, in step 2.5, if the virtual machine type is upgraded to the highest level and the total computation cost of the current layer is still less than the budget given by the current layer, the virtual machine type is upgraded for the current non-critical module, the current non-critical module selects a next-level module of the current critical module for the first time, and if the total computation cost of the current layer is less than the budget of the current layer after the upgrade, the virtual machine type upgrade continues for the current non-critical module, the total computation cost of the current layer after the upgrade is compared with the budget given by the current layer, and if the total computation cost of the current layer is still less than the budget given by the current layer, the upgrade and the comparison continue until the total computation cost of the current layer is equal to or greater than the budget given by the current layer.
In some embodiments, in step 2.8, if the cost exceeds the budget during the transmission with the worst bandwidth, the non-critical modules are prioritized virtual machine type downgraded based on the result obtained in step 2.5, the virtual machine type of the module with the lowest priority among the non-critical modules is downgraded first and the module priority is updated in time until the virtual machine type is worst or the budget requirement is met, and then step 2.8 is executed.
Preferably, the method of the present invention comprises:
step one, the workflow is layered
To workflow G w (V w ,E w ) Layering is carried out, modules in the same layer have no dependency relationship, V w As a set of modules, E w Is the set of all module dependencies; the sum of all module workloads in the first layer after layering is CR l (ii) a L =1,2,3.. L, L ≧ 2, the budget of the L-th layer is
CR is the sum of the workloads of all modules in the workflow;
step two, sorting the virtual machine types
Sorting the types of the virtual machines of the multi-cloud data center according to the priority and the value of the CPU, the Memory and the Disk;
step three, distributing the virtual machine and the physical machine
And (3) allocating virtual machines and physical machines to the workflow layer by layer:
3.1 adding a virtual output module to the current layer, wherein the workload and the transmission data size of the virtual output module are both 0;
3.2 distributing virtual machine examples of the worst virtual machine type to the self modules of the current layer, and calculating any one module w in the current layer i Completion time T to virtual output module i I is not less than 1,Ti = T ime +T idt ， T ime Is a module w i Ratio of workload to calculated force of worst virtual machine type, T idt Is a module w i Inputting the time of data transmission; the module with the longest completion time in all modules of the current layer is a key module;
3.3, carrying out priority sequencing on the modules in the current layer, wherein the priority of the key module is the highest, and sequencing the rest modules in sequence, wherein the higher the workload is, the higher the priority is;
3.4 calculate the total completion time T needed by all modules of the current layer to calculate with the worst virtual machine type l And total calculation cost C l ；
3.5 Total cost of current layer C l If the total calculation cost of the current layer after upgrading is less than the budget of the current layer, updating the key module according to the ratio of the workload of each module in the current layer to the calculation power of each current virtual machine type, wherein the module with the largest ratio is a new key module; then, carrying out virtual machine type upgrading on the new key module, then comparing the total calculation cost of the current layer after upgrading with the budget size of the current layer again, if the total calculation cost of the current layer after upgrading is still smaller than the budget of the current layer, continuing to update the key module until the total calculation cost of the current layer is larger than the budget given by the current layer, and carrying out virtual machine type degradation on the current key module; completing virtual machine allocation or completing virtual machine allocation until the total computational cost of the current layer is equal to the budget given by the current layer;
3.6 distributing all modules to the physical machines of the corresponding cloud data center according to the virtual machine distribution scheme selected in the step 3.5; under the condition that cloud data center resources are met, modules with dependency relationships are placed in the same cloud as much as possible;
3.7, performing performance sequencing on the available bandwidths of the cloud data centers; arranging worst bandwidth for data transmission among all modules of the current layer;
3.8 calculating the cost required by each dependency relationship in the current layer for transmitting data by using the worst bandwidth to obtain the total cost of the current layer;
and 3.9 comparing the total cost of the current layer with the budget of the current layer, if the total cost of the current layer is less than the budget of the current layer, selecting a new level of bandwidth for the current layer and then continuously comparing the total cost of the current layer with the budget of the current layer, if the total cost of the current layer is greater than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection.
The invention also provides a static workflow scheduling device, which comprises:
to workflow G w (V w ,E w ) Layering is carried out, and the modules in the same layer do not have workflow layering modules, dependency relationships and V w As a set of modules, E w Is the set of all module dependencies; the sum of all module workloads in the first layer after layering is CR l (ii) a L =1,2,3.. L, L ≧ 2, the budget of the L-th layer is
CR is the sum of the workloads of all modules in the workflow;
the virtual machine and physical machine allocation module is used for allocating the virtual machine and the physical machine to the workflow layer by layer:
in the process of distributing virtual machines to a current layer, distributing virtual machine categories to modules in the current layer according to the worst level to the optimal level of the performance of the virtual machine categories of a plurality of cloud centers, each distributing is performed once, the total calculation cost of the current layer is compared with the budget of the current layer once, if the total calculation cost of the current layer is smaller than the budget of the current layer, a new virtual machine category is continuously distributed to the modules in the current layer, if the total calculation cost of the current layer is larger than the budget of the current layer, the last distributed virtual machine category is distributed to the modules in the current layer and distribution is completed, and if the total calculation cost of the current layer is equal to the budget of the current layer, distribution is completed; the total calculation cost of the current layer is the sum of the costs required by all the modules in the current layer to finish the calculation by utilizing the current virtual machine types;
in the process of selecting a physical machine for a current layer, all modules are allocated to the physical machines of corresponding cloud data centers according to a virtual machine scheme allocated to the current layer, bandwidth is selected for the current layer according to the worst level to the optimal level of bandwidth performance of a plurality of cloud data centers, the total expense of the current layer is compared with the budget of the current layer once the bandwidth is selected, and if the total expense of the current layer is smaller than the budget of the current layer, a new level of bandwidth is continuously selected for the modules in the current layer; if the total cost of the current layer is larger than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection; the total cost of the current layer is the sum of the cost required by each dependency relationship in the current layer to transmit data by using the currently selected bandwidth.
The workflow mapping scheduling strategy based on the multi-cloud environment has the following advantages:
(1) According to the invention, the workflow is layered according to the dependency of the workflow modules, so that the situation of mutual influence does not exist among all the modules, the time of module calculation waiting phase of large-scale scientific calculation is reduced, and the working efficiency of the workflow of each layer is improved.
(2) For the whole workflow, the virtual machine and the physical machine are selected according to the key module by considering the multi-cloud situation and the heterogeneous characteristics of the physical machine in the data center in a combined manner, so that the influence of the heterogeneous performance on the efficiency of the workflow is reduced.
(3) The invention reduces the occurrence probability of the over-budget situation by considering the performance and the price of different virtual machines and bandwidths, improves the throughput value of the whole workflow under the budget situation, and further improves the performance of the whole high-throughput system.
Drawings
FIG. 1 is a workflow module division of the present invention;
fig. 2 is a diagram illustrating throughput improvement values under the fixed budget condition of the present invention compared to the existing algorithm.
Fig. 3 is a graph illustrating throughput improvement values of the present invention compared to existing algorithms. Wherein (a) is the throughput improvement of the present invention relative to the B-RATE method under 20 different problem scales and 20 different budget constraints; (b) The throughput capacity of the method is improved relative to that of a Greedy-LDP method under the conditions of 20 different problem scales and 20 different budget constraints; (c) The throughput of the present invention is improved relative to the MCWM method in 20 different problem scales and 20 different budget constraints.
Detailed Description
All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. In addition, the following definitions apply to the terms used in this specification unless otherwise limited by special circumstances.
The workflow in this field is usually expressed as a directed acyclic graph DAG, where each edge represents the connection or a dependency relationship between a previous node and a next node, after the previous node finishes executing, the previous node transmits data to the next node according to the weight of the edge, i.e., the size of the transmitted data, and the next node starts executing. Specifically, a workflow G represented by a directed acyclic graph can be generated according to specific application requirements submitted by a user or a cloud provider and a data placement position w (V w ,E w ) The modules in the workflow specifically refer to basic data operations such as programs and code segments.
Workflow G w (V w ,E w ) Layering is the formation of multiple layers in a workflow, with no dependencies on modules in the same layer. For example to workflow G w (V w ,E w ) The modules with dependency relationship (i.e. modules with edges connected) are divided by dotted lines, and the modules without adjacent edges are placed in the same layer to generate the structure shown in fig. 1, so that the modules in the same layer have no dependency relationship, and the modules in the next layer need to wait until the modules in the previous layer and the previous layer are operated (i.e. there is dependency relationship).
The virtual machine type refers to a selectable item provided by a multi-cloud provider and capable of running a virtual machine instance, and the method and the device are applicable to the existing virtual machine type. The worst level to the optimal level of the type performance of the virtual machine can be sorted and graded according to the sequence of Disk, memory and CPU and the value size of each, and the same parameter is compared and sorted, and the performance is better when the same parameter value is larger. That is to say, the virtual machines are firstly sorted according to the size of the CPU resource, if the CPU resource is the same, the Memory resource is sorted according to the index of the Memory resource, if the CPU resource is the same, the Memory resource is sorted according to the size of the available space of the Disk, and after the sorting is finished, the virtual machine type and the price table are obtained according to the corresponding price.
The bandwidth refers to options provided by a multi-cloud provider for running a virtual machine instance, and the poor performance and the excellent performance of the bandwidth can be ranked according to the numerical value of the bandwidth.
The virtual machine allocation scheme of the invention adopts the steps of sorting the types of the given virtual machines according to performance and obtaining the virtual machine selection scheme under the condition of not exceeding budget by continuously selecting the modules and the types of the virtual machines at the current layer based on the principle of the highest priority of key modules. The updating in the process refers to the reselection of the key module in the current layer according to the method of the invention; the upgrading in the process refers to upgrading the type of the virtual machine from a poor-performance level to a good-performance level, preferably from the poor-performance level to a superior-level; the comparison in the process refers to comparing the total calculation cost of the current layer in the method with the budget of the current layer. The total calculation cost of the previous layer is the sum of the calculation costs of the modules in the current layer by using the respective current virtual machine types, and the calculation cost of the modules by using the respective current virtual machine types is multiplied by the calculation time of the modules by using the respective current virtual machine types and the price of the corresponding virtual machine types, wherein the price of the virtual machine types is provided by a cloud provider.
The physical machine selection is to select a physical machine to generate a virtual machine instance according to the type of the virtual machine of each layer, select the position of the cloud and a specific physical machine in the cloud according to the budget of the current layer, and select the most available scheme under the current cost based on the principle of the same cloud and the minimum bandwidth increase. The upgrading in the process refers to upgrading the bandwidth from a poor-performance level to a good-performance level, preferably from the poor-performance level to a superior level; the comparison in the process refers to comparing the total cost of the current layer with the budget of the current layer.
The following are specific examples provided by the inventors to further explain the technical solutions of the present invention.
Example 1:
step 1: workflow layering
Generating a workflow structure G represented by a directed acyclic graph according to specific application requirements and data placement positions submitted by users or cloud providers w (V w ,E w ) Wherein Vw represents the set of modules in the workflow and Ew represents the set of computational dependencies resulting from data placement and application requirements; the workflow is layered, the modules in the same layer do not have dependency relationship, and the total work amount of the first layer is CR l Budget is
And 2, step: virtual machine type ordering
For n (n is greater than or equal to 1) virtual machine types provided by a multi-cloud provider, the expression is VT = { VT = 0 ,vt 1 ,...,vt n-1 Sorting the performance of the virtual machines according to the priority of the CPU, the Memory and the Disk, namely sorting according to the size of CPU resources, if the CPU resources are the same, sorting according to the index of the Memory resources, and if the CPU resources are the same, sorting according to the size of the available space of the Disk;
and 3, step 3: distributing virtual machines
Selecting a proper virtual machine instance for the workflow module of each layer, considering the constraints of cost and performance, and updating the performance index of the current layer in real time, wherein the specific method comprises the following steps:
1) Adding a virtual output module for the current layer, wherein the workload and the size of the transmission data of the virtual output module are both 0 (namely the assumed module);
2) Distributing the virtual machine instance of the worst virtual machine type for the self module (not including the virtual output module) of the current layer, and calculating any one module w in the current layer i Completion time T to virtual output module i I is greater than or equal to 1,T i ＝T ime +T idt And T is ime Is the ratio of the module workload to the calculated power of the selected worst virtual machine type, T idt Is a module w i The time of input data transmission, the module with the longest completion time among all modulesDefining as a key module;
3) The priority of the key module is highest, the priority is distributed to the rest modules in the layer according to the workload, and the higher the workload is, the higher the priority is;
4) Calculating the completion time T of the current layer under the worst virtual machine type l The computational cost of each module of the layer C i The total cost of the layer C l (the calculation method is the product of time and the price of the corresponding virtual machine type);
5) If the total computation cost of the current layer C L If the total calculation cost of the current layer after upgrading is less than the budget given by the current layer, updating the key module according to the ratio of the workload of all modules of the current layer to the calculation power of each current virtual machine type, wherein the module with the largest ratio is a new key module; then, upgrading the virtual machine type of the new key module to a third level, continuously comparing the total calculation expense of the current layer with the given budget size of the current layer, if the total calculation expense of the current layer is smaller than the given budget size of the current layer, continuously updating and comparing until the total calculation expense of the current layer is larger than the given budget of the current layer, and downgrading the virtual machine type of the current key module; completing virtual machine allocation, or completing virtual machine allocation when the total calculation cost of the current layer is equal to the budget given by the current layer;
6) According to the mapping relation between the modules selected in the step 5) and the virtual machines, further considering data transmission between the cloud environment and the cloud data center to select physical machines, and distributing all the modules to the physical machines of the corresponding cloud data center according to the virtual machine distribution scheme selected in the step 5); under the condition that cloud data center resources are met, modules with dependency relationships are placed in the same cloud as much as possible;
7) Performing performance sequencing on available bandwidths provided by a multi-cloud environment, and arranging a bandwidth with the worst performance for the transmission among clouds required by a current layer;
8) Calculating the unit cost required by each edge (dependency between modules and data transmission) by using the worst bandwidth, and summing to obtain the total cost of the current layer;
9) Comparing the total cost of the current layer with the budget of the current layer, if the total cost of the current layer is less than the budget of the current layer, selecting a new level of bandwidth for the current layer and then continuously comparing the sizes of the current layer and the budget of the current layer, if the total cost of the current layer is greater than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection.
In order to verify the feasibility of the method, the method for dispatching the workflow in the multi-cloud environment is verified by a software programming simulation experiment:
firstly, a virtual machine type is set in a cloud data center of a cloud supplier in simulation reality, wherein specific parameters of the virtual machine type are that Cpu frequency is in a range of [1Ghz,4Ghz ], I/O is set to be 20M/s, and a hard disk space is randomly set between [500MB,40GB ]; the available type of data transmission bandwidth between data centers is randomly assumed, and the range is [10M,100M ].
And a random workflow structure is generated according to the parameters shown in the following table, 20 problems with different specifications are defined, and in order to ensure that the workflow can be reasonably scheduled, the setting of the budget value is determined according to the corresponding problem specification. The lower limit is the cost required when the types of the virtual machine and the bandwidth are all worst, and the upper limit is the cost required when the types of the virtual machine and the bandwidth are all optimal. The details are shown in Table 1.
TABLE 1 problem Scale actual parameters
In the table, m represents the number of blocks, | E w I represents the specific data transmission size of the data dependency relationship, and the range is [5,500 ]]In the above-mentioned manner,n represents the number of available virtual machine types, and c represents the number of cloud data centers at the scale of the problem.
Fig. 2 shows that, in the case where the budget is constant, the throughput of the present invention increases significantly at a constant budget as the problem scale changes, and the average throughput increase rate changes more greatly than the throughput of another algorithm. FIG. 3 is a graph comparing throughput at different problem scales and budgets, and the throughput of the present invention is significantly improved over other methods, with large variation fluctuations, whether the budget value or the problem scale is changed.
Example 2:
for a workflow and a multi-cloud environment different from that of embodiment 1, on the basis of the execution of the method in embodiment 1, there may be a case that in the virtual machine allocation process, the virtual machine type has been upgraded to the highest level, and the total computation cost of the current layer is still less than the budget given by the current layer, in this case, the virtual machine type upgrade is performed on the next-level module of the current key module, and the upgrade is performed to the next-level module of the current virtual machine type, and if the total computation cost of the current layer is less than the budget given by the current layer, the virtual machine type upgrade is performed on the key module for the second level, and the total computation cost of the current layer after the upgrade is compared with the budget given by the current layer until the total computation cost of the current layer is greater than the budget given by the current layer.
Example 3:
for other different workflows and cloud environments, on the basis of the scheme in embodiment 1 or 2, in the physical machine selection process, if the total cost of the current layer exceeds the budget when the current layer is transmitted with the worst bandwidth, the virtual machine allocation scheme is adjusted again, specifically, the priority virtual machine type degradation is performed on the non-critical modules on the basis of the allocated virtual machine scheme, the virtual machine type of the module with the lowest priority in the non-critical modules is degraded first, the module priority is updated in time, and the upgrading bandwidth and the comparison are continuously updated until the virtual machine type is reduced to the worst level or the budget requirement is met.
Claims (9)
1. A static workflow scheduling method for virtual machine allocation and physical machine selection in a plurality of cloud data center environments is characterized by comprising the following steps:
step one, the workflow is layered
To workflow G w (V w ,E w ) Layering is carried out, modules in the same layer have no dependency relationship, V w As a set of modules, E w Is the set of all module dependencies; the sum of all module workloads in the first layer after layering is CR l (ii) a L =1,2,3 \ 8230l, L is more than or equal to 2, and the budget of the first layer is
CR is the sum of the workloads of all modules in the workflow;
step two, virtual machine and physical machine allocation
And (3) distributing the virtual machine and the physical machine to the workflow layer by layer:
(1) The virtual machine allocation to the current layer comprises: allocating virtual machine types to modules in a current layer according to the worst level to the optimal level of the virtual machine type performances of a plurality of cloud centers, wherein each allocation is performed once, the total calculation cost of the current layer is compared with the budget size of the current layer once, if the total calculation cost of the current layer is less than the budget of the current layer, a new virtual machine type is continuously allocated to the modules in the current layer, if the total calculation cost of the current layer is more than the budget of the current layer, the last allocated virtual machine type is allocated to the modules in the current layer and allocation is completed, and if the total calculation cost of the current layer is equal to the budget of the current layer, allocation is completed; the total calculation cost of the current layer is the sum of the costs required by the modules in the current layer to finish calculation by utilizing the current virtual machine types;
the virtual machine allocation process for the current layer comprises the following steps:
2.1 adding a virtual output module to the current layer, wherein the workload and the transmission data size of the virtual output module are both 0;
2.2 allocating the virtual machine instance of the worst virtual machine type to the self modules of the current layer, and calculating any module w in the current layer i Completion time T to virtual output module i I is greater than or equal to 1,T i ＝T ime +T idt ，T ime Is a module w i Ratio of workload to calculated force of worst virtual machine type, T idt Is a module w i Inputting the time of data transmission; the module with the longest completion time in all modules of the current layer is a key module;
2.3, carrying out priority sequencing on the modules in the current layer, wherein the priority of the key module is the highest, and sequencing the rest modules in sequence, wherein the higher the workload is, the higher the priority is;
2.4 calculate the total completion time T required for all modules in the current layer to calculate by using the worst virtual machine type l And calculating the total calculation cost C l ；
2.5 Total cost of current layer C l If the total calculation cost of the current layer after upgrading is less than the budget of the current layer, updating the key module according to the ratio of the workload of each module in the current layer to the calculation power of each current virtual machine type, wherein the module with the largest ratio is a new key module; then, carrying out virtual machine type upgrading on the new key module, then comparing the total calculation cost of the current layer after upgrading with the budget size of the current layer again, if the total calculation cost of the current layer after upgrading is still smaller than the budget of the current layer, continuing to update the key module until the total calculation cost of the current layer is larger than the budget given by the current layer, and carrying out virtual machine type degradation on the current key module; completing virtual machine allocation, or completing virtual machine allocation until the total calculation cost of the current layer is equal to the budget given by the current layer;
(2) The physical machine selection of the current layer comprises the following steps: allocating all modules to corresponding physical machines of the cloud data centers according to a virtual machine scheme allocated to the current layer, selecting bandwidth for the current layer according to the worst level to the optimal level of the bandwidth performance of the plurality of cloud data centers, comparing the total cost of the current layer with the budget size of the current layer once selection, and if the total cost of the current layer is less than the budget of the current layer, continuing to select a new level of bandwidth for the modules in the current layer; if the total cost of the current layer is larger than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection; the total cost of the current layer is the sum of the cost required by each dependency relationship in the current layer to transmit data using the currently selected bandwidth.
2. The static workflow scheduling method of claim 1 wherein the worst to the best performance of the virtual machine classes of the plurality of cloud centers are ranked in order of Disk, memory, and CPU, and the same parameter is compared and ranked, and the performance is better the larger the value of the same parameter is.
3. The static workflow scheduling method of claim 1 wherein the worst to best ranking of bandwidth performance of the plurality of cloud data centers is a ranking by bandwidth value size.
4. The static workflow scheduling method of claim 1 wherein performing a physical machine selection process on the current layer comprises:
2.6 distributing all modules to the physical machines of the corresponding cloud data center according to the selected virtual machine distribution scheme;
2.7 calculating the cost required by each dependency relationship in the current layer to transmit data by using the worst performance bandwidth to obtain the total cost of the current layer;
2.8 comparing the total cost of the current layer with the budget of the current layer, if the total cost of the current layer is less than the budget of the current layer, selecting a new level of bandwidth for the current layer and then continuously comparing the sizes of the current layer and the budget of the current layer, if the total cost of the current layer is greater than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection.
5. The static workflow scheduling method of claim 1 wherein in the physical machine selection process, modules having a dependency relationship are placed in the same cloud if cloud data center resources are satisfied.
6. The method according to claim 1, wherein in step 2.5, if the virtual machine type is upgraded to the highest level and the total computation cost of the current layer is still less than the budget given by the current layer, the virtual machine type is upgraded for the current non-critical module, the current non-critical module first selects a next-level module of the current critical module, and if the total computation cost of the current layer is less than the budget of the current layer after the upgrade, the virtual machine type upgrade continues for the current non-critical module, the total computation cost of the current layer after the upgrade is compared with the budget given by the current layer, and if the total computation cost of the current layer is still less than the budget of the current layer, the upgrade continues and the comparison until the total computation cost of the current layer is equal to or greater than the budget given by the current layer.
7. The method according to claim 1, wherein in step 2.8, if the budget is exceeded when the cost is spent on the transmission with the worst bandwidth, the non-critical module is downgraded with the priority virtual machine type based on the result obtained in step 2.5, the virtual machine type of the module with the lowest priority in the non-critical module is downgraded first and the module priority is updated in time until the virtual machine type is worst or the budget requirement is met, and then step 2.8 is executed.
8. A static workflow scheduling method is used for virtual machine allocation and physical machine selection in a plurality of cloud data center environments, and is characterized by comprising the following steps:
step one, the workflow is layered
To workflow G w (V w ,E w ) Layering, wherein the modules in the same layer have no dependency relationship, V w As a set of modules, E w Is the set of all module dependencies; the sum of all module workloads in the first layer after layering is CR l ；l＝1,2,3…L，L≥2, budget of layer I
CR is the sum of the workloads of all modules in the workflow;
step two, sorting the virtual machine types
Sorting the types of the virtual machines of the multi-cloud data center according to the priority and the value of the CPU, the Memory and the Disk;
step three, distributing the virtual machine and the physical machine
And (3) distributing the virtual machine and the physical machine to the workflow layer by layer:
3.1 adding a virtual output module to the current layer, wherein the workload and the transmission data size of the virtual output module are both 0;
3.2 distributing the virtual machine instance of the worst virtual machine type to the self modules of the current layer, and calculating any module w in the current layer i Completion time T to virtual output module i I is not less than 1,Ti = T ime +T idt ，T ime Is a module w i Ratio of workload to calculated force of worst virtual machine type, T idt Is a module w i Inputting the time of data transmission; the module with the longest completion time in all modules of the current layer is a key module;
3.3, carrying out priority sequencing on the modules in the current layer, wherein the priority of the key module is the highest, and sequencing the rest modules in sequence, wherein the higher the workload is, the higher the priority is;
3.4 calculate the total completion time T needed by all modules of the current layer to calculate with the worst virtual machine type l And total calculation cost C l ；
3.5 Total cost of current layer C l If the total calculation cost of the current layer after upgrading is less than the budget of the current layer, updating the key module according to the ratio of the workload of each module in the current layer to the calculation power of each current virtual machine type, wherein the module with the largest ratio is a new key module; then, the new key module is upgraded by the virtual machine type, and thenComparing the total calculation cost of the current layer after upgrading with the budget size of the current layer, if the total calculation cost of the current layer after upgrading is still smaller than the budget of the current layer, continuing to update the key module until the total calculation cost of the current layer is larger than the given budget of the current layer, and performing degradation virtual machine type on the current key module; completing virtual machine allocation, or completing virtual machine allocation until the total calculation cost of the current layer is equal to the budget given by the current layer;
3.6 distributing all modules to the physical machines of the corresponding cloud data center according to the virtual machine distribution scheme selected in the step 3.5; under the condition that the cloud data center resources are met, modules with dependency relationships are placed in the same cloud as much as possible;
3.7, performing performance sequencing on the available bandwidths of the cloud data centers; arranging worst bandwidth for data transmission among all modules of the current layer;
3.8 calculating the cost required by each dependency relationship in the current layer to transmit data by using the worst bandwidth to obtain the total cost of the current layer;
and 3.9 comparing the total cost of the current layer with the budget of the current layer, if the total cost of the current layer is less than the budget of the current layer, selecting a new level of bandwidth for the current layer and then continuously comparing the total cost of the current layer with the budget of the current layer, if the total cost of the current layer is greater than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection.
9. A static workflow scheduling apparatus, the apparatus is used for virtual machine allocation and physical machine selection in a plurality of cloud data center environments, and the apparatus comprises:
to workflow G w (V w ,E w ) Layering is carried out, the modules in the same layer do not have the workflow layering module,
dependence, V w As a set of modules, E w Is the set of all module dependencies; after layering, the sum of all module workloads in the first layer is CR l (ii) a L =1,2,3 \ 8230l, L is more than or equal to 2, and the budget of the first layer is
CR is the sum of the workloads of all modules in the workflow;
the virtual machine and physical machine distribution module is used for distributing the virtual machine and the physical machine layer by layer to the workflow:
in the process of distributing virtual machines to a current layer, distributing virtual machine categories to modules in the current layer according to the worst level to the optimal level of the performance of the virtual machine categories of a plurality of cloud centers, each distributing is performed once, the total calculation cost of the current layer is compared with the budget of the current layer once, if the total calculation cost of the current layer is less than the budget of the current layer, a new virtual machine category is continuously distributed to the modules in the current layer, if the total calculation cost of the current layer is greater than the budget of the current layer, the last distributed virtual machine category is distributed to the modules in the current layer and distribution is completed, and if the total calculation cost of the current layer is equal to the budget of the current layer, distribution is completed; the total calculation cost of the current layer is the sum of the costs required by the modules in the current layer to finish calculation by utilizing the current virtual machine types;
the virtual machine allocation process for the current layer comprises the following steps:
2.1 adding a virtual output module to the current layer, wherein the workload and the transmission data size of the virtual output module are both 0;
2.2 distributing the virtual machine instance of the worst virtual machine type to the self modules of the current layer, and calculating any module w in the current layer i Completion time T to virtual output module i I is greater than or equal to 1,T i ＝T ime +T idt ，T ime Is a module w i Ratio of workload to calculated force of worst virtual machine type, T idt Is a module w i Inputting the time of data transmission; the module with the longest completion time in all modules of the current layer is a key module;
2.3, carrying out priority sequencing on the modules in the current layer, wherein the priority of the key module is the highest, and sequencing the rest modules in sequence, wherein the higher the workload is, the higher the priority is;
2.4 calculate the total completion time T needed by all modules of the current layer to calculate by using the worst virtual machine type l And calculating the total calculation cost C l ；
2.5 Total computation cost C if Current layer l If the total calculation cost of the current layer after upgrading is less than the budget of the current layer, updating the key module according to the ratio of the workload of each module in the current layer to the calculation power of each current virtual machine type, wherein the module with the largest ratio is a new key module; then, carrying out virtual machine type upgrading on the new key module, then comparing the total calculation cost of the current layer after upgrading with the budget size of the current layer again, if the total calculation cost of the current layer after upgrading is still smaller than the budget of the current layer, continuing to update the key module until the total calculation cost of the current layer is larger than the budget given by the current layer, and carrying out virtual machine type degradation on the current key module; completing virtual machine allocation, or completing virtual machine allocation until the total calculation cost of the current layer is equal to the budget given by the current layer;
in the process of selecting a physical machine for a current layer, all modules are allocated to the physical machines of corresponding cloud data centers according to a virtual machine scheme allocated to the current layer, bandwidth is selected for the current layer according to the worst level to the optimal level of the bandwidth performance of a plurality of cloud data centers, the total cost of the current layer is compared with the budget size of the current layer once the bandwidth is selected, and if the total cost of the current layer is smaller than the budget of the current layer, a new level of bandwidth is continuously selected for the modules in the current layer; if the total cost of the current layer is larger than the budget of the current layer, selecting the last bandwidth for the module in the current layer and completing the selection, and if the total cost of the current layer is equal to the budget of the current layer, completing the selection; the total cost of the current layer is the sum of the cost required by each dependency relationship in the current layer to transmit data using the currently selected bandwidth.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811492446.5A CN109783189B (en) | 2018-12-07 | 2018-12-07 | Static workflow scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811492446.5A CN109783189B (en) | 2018-12-07 | 2018-12-07 | Static workflow scheduling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109783189A CN109783189A (en) | 2019-05-21 |
| CN109783189B true CN109783189B (en) | 2022-11-22 |
Family
ID=66495757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811492446.5A Active CN109783189B (en) | 2018-12-07 | 2018-12-07 | Static workflow scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109783189B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609736B (en) * | 2019-07-30 | 2021-04-20 | 中国人民解放军国防科技大学 | Deadline constraint scientific workflow scheduling method in cloud environment |
| US11995479B2 (en) * | 2020-01-02 | 2024-05-28 | International Business Machines Corporation | Implementing workloads in a multi-cloud environment with data migration between cloud vendors |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108469988A (en) * | 2018-02-28 | 2018-08-31 | 西北大学 | A kind of method for scheduling task based on isomery Hadoop clusters |
| CN108628665A (en) * | 2018-05-16 | 2018-10-09 | 天津科技大学 | Task scheduling based on data-intensive scientific workflow and virtual machine integration method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10528396B2 (en) * | 2016-11-14 | 2020-01-07 | King Abdulaziz University | Temporal task scheduling in a hybrid system |
-
2018
- 2018-12-07 CN CN201811492446.5A patent/CN109783189B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108469988A (en) * | 2018-02-28 | 2018-08-31 | 西北大学 | A kind of method for scheduling task based on isomery Hadoop clusters |
| CN108628665A (en) * | 2018-05-16 | 2018-10-09 | 天津科技大学 | Task scheduling based on data-intensive scientific workflow and virtual machine integration method |
Non-Patent Citations (4)
| Title |
|---|
| Distributed Workflow Scheduling under Throughput and Budget Constraints in Grid Environments;Cao F;《Job Scheduling Strategies for Parallel Processing》;20140611;第62-80页 * |
| Optimizing the Performance of Big Data Workflows in Multi-cloud Environments Under Budget Constraint;Chase Q. Wu;《2016 IEEE International Conference on Services Computing (SCC)》;20160901;全文 * |
| 多个云环境下大数据工作流性能优化算法的设计与实现;石萌;《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》;20180215;第2018年卷(第2期);正文 * |
| 面向数据密集型工作流的QoS-aware调度算法;万聪等;《系统仿真学报》;20160308(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109783189A (en) | 2019-05-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11275609B2 (en) | Job distribution within a grid environment | |
| Bittencourt et al. | HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds | |
| US9141432B2 (en) | Dynamic pending job queue length for job distribution within a grid environment | |
| Samadi et al. | E-HEFT: enhancement heterogeneous earliest finish time algorithm for task scheduling based on load balancing in cloud computing | |
| US20050081208A1 (en) | Framework for pluggable schedulers | |
| JP4185103B2 (en) | System and method for scheduling executable programs | |
| CN111381950A (en) | Task scheduling method and system based on multiple copies for edge computing environment | |
| Thaman et al. | Green cloud environment by using robust planning algorithm | |
| Wu et al. | Optimizing the performance of big data workflows in multi-cloud environments under budget constraint | |
| CN109784663B (en) | Workflow scheduling method and device | |
| US9298500B2 (en) | Information processing system and control method of information processing system for managing jobs in a distributed multi-node environment | |
| CN109783189B (en) | Static workflow scheduling method and device | |
| CN109815009B (en) | Resource scheduling and optimizing method under CSP | |
| Wang et al. | An iterative optimization framework for adaptive workflow management in computational clouds | |
| CN110084507B (en) | Scientific workflow scheduling optimization method based on hierarchical perception in cloud computing environment | |
| Jalalian et al. | A hierarchical multi-objective task scheduling approach for fast big data processing | |
| Stavrinides et al. | Orchestrating bag-of-tasks applications with dynamically spawned tasks in a distributed environment | |
| Pujiyanta et al. | Planning and scheduling jobs on grid computing | |
| Cao et al. | Distributed workflow scheduling under throughput and budget constraints in grid environments | |
| He et al. | Hybrid performance-based workload management for multiclusters and grids | |
| Xie et al. | A novel independent job rescheduling strategy for cloud resilience in the cloud environment | |
| EP1630671A1 (en) | Framework for pluggable schedulers | |
| Singh et al. | Prediction-Based Resource Assignment Scheme to Maximize the Net Profit of Cloud Service Providers | |
| Kumar et al. | Instance and value (IVH) algorithm and dodging dependency for scheduling multiple instances in hybrid cloud computing | |
| CN114816720B (en) | Scheduling method and device of multi-task shared physical processor and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |