CN115080236A - Workflow deployment method based on graph segmentation - Google Patents
Workflow deployment method based on graph segmentation Download PDFInfo
- Publication number
- CN115080236A CN115080236A CN202210730454.9A CN202210730454A CN115080236A CN 115080236 A CN115080236 A CN 115080236A CN 202210730454 A CN202210730454 A CN 202210730454A CN 115080236 A CN115080236 A CN 115080236A
- Authority
- CN
- China
- Prior art keywords
- task
- tasks
- workflow
- graph
- representing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 230000011218 segmentation Effects 0.000 title abstract description 4
- 238000005192 partition Methods 0.000 claims abstract description 54
- 230000005540 biological transmission Effects 0.000 claims abstract description 34
- 238000010586 diagram Methods 0.000 claims abstract description 15
- 238000013507 mapping Methods 0.000 claims abstract description 3
- 238000012546 transfer Methods 0.000 claims description 16
- 238000000638 solvent extraction Methods 0.000 claims description 8
- KDYFGRWQOYBRFD-UHFFFAOYSA-N succinic acid Chemical compound OC(=O)CCC(O)=O KDYFGRWQOYBRFD-UHFFFAOYSA-N 0.000 claims description 8
- 238000004422 calculation algorithm Methods 0.000 abstract description 31
- 238000004891 communication Methods 0.000 abstract description 29
- 230000008569 process Effects 0.000 abstract description 13
- 238000005457 optimization Methods 0.000 abstract description 4
- 238000004088 simulation Methods 0.000 description 11
- 238000004364 calculation method Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 7
- 230000008901 benefit Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 2
- 230000002035 prolonged effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a workflow deployment method based on graph segmentation, which mainly solves the problem that the existing workflow deployment algorithm based on clustering realizes the minimization of communication overhead at the cost of sacrificing the parallel execution efficiency of tasks in workflows, so that the parallel execution efficiency of the tasks is lower. The implementation scheme is as follows: 1) establishing a DAG model G of the workflow directed acyclic graph; 2) calculating task execution time and data transmission time among tasks in the workflow; 3) merging serial structures in the workflow model G to obtain a new workflow model diagram G'; 4) segmenting the new workflow model diagram G' to obtain an optimal task partition; 5) and mapping the optimal task partition to the virtual machine according to the minimum execution time to complete the deployment of the workflow. The invention reduces the completion time of the workflow, improves the execution efficiency of the workflow, and can be used for joint optimization of data overhead and task parallel execution efficiency in the workflow execution process.
Description
Technical Field
The invention belongs to the technical field of cloud computing, and particularly relates to a workflow deployment method which can be used for joint optimization of data overhead and task parallel execution efficiency in a workflow execution process.
Background
In a cloud computing environment, a workflow refers to an associated task consisting of a set of dependent tasks, typically described using a directed acyclic graph DAG. Workflow is much larger and more structurally complex than independent tasks. The deployment of the workflow needs to consider not only the resource allocation of each task in the workflow, but also the data transmission and execution sequence among the tasks, thereby greatly increasing the complexity of task deployment. How to more scientifically and reasonably deploy workflows in a distributed and heterogeneous environment remains a research hotspot in the academic community at present.
At present, cluster computing architectures such as MapReduce and Spark are widely applied to data center networks to analyze and process continuously increasing computing and network tasks, decompose complex large-scale tasks into simpler tasks and model them into workflows, and deliver the workflows to cloud data centers with powerful parallel processing capability. A workflow comprises a plurality of tasks with interdependencies, the tasks are connected with each other according to a certain priority order, and the deployment of the workflow must consider the data transmission among the tasks. Research shows that in the MapReduce application, the time used for intermediate data transmission accounts for more than 30% of the time for completing the whole workflow, and for some large-scale commercial data centers, such as yahoo data center clusters, the process data transmission of the workflow is the most important component of the network traffic, and the time occupied by the process data transmission is close to 60% of the total time for completing the workflow. Meanwhile, the process data transmission of the workflow is also a key cause of network congestion. Therefore, the workflow is optimized and deployed, so that the process communication overhead is reduced, the flow pressure of a data center is relieved, and the task completion time is shortened.
A representative heuristic workflow deployment algorithm mainly includes: list-based deployment algorithms, cluster-based deployment algorithms, replication-based deployment algorithms. The workflow deployment algorithm based on the clustering aims at reducing communication overhead among workflow tasks, tasks in the workflow are firstly mapped to different clusters, and then each cluster is mapped to the same computing node as a whole. The core idea is to divide a plurality of tasks with edge connection (data dependency relationship) into the same cluster, thereby saving the communication overhead among the tasks in the cluster.
For example, Ahmad SG et al, in its published paper "Data-intensive workflow optimization based on application task graph in the hierarchy-new computing systems" (IEEE Fourth International Conference on Big Data and Cloud computing. IEEE 2014: 129-. In this algorithm, the workflow is partitioned into task partitions of a specified size and number to minimize the data transfer overhead between partitions. The PDWA defines the maximum number of tasks allowed to be contained in each task partition, the value is calculated by multiplying the total number of tasks in the workflow by a coefficient smaller than 1, and then each task partition is mapped to the computing node which enables the partition to be shortest in execution time. The method has the disadvantages that the workflow is clustered according to the data dependency relationship among the workflow tasks, so that certain tasks which have stronger data dependency relationship and can be executed in parallel can be divided into the same cluster, the parallel execution performance of the tasks is poor, and the workflow completion time is influenced finally.
Disclosure of Invention
The invention aims to provide a workflow deployment method based on graph partitioning aiming at the defects of the prior art, so as to realize the balance between the minimization of communication overhead and the maximization of parallelism in the workflow clustering process and improve the execution efficiency of workflows.
The technical idea of the invention is as follows: from the perspective of graph theory, the dependency and the parallelism among tasks in the workflow are fully mined, a classic graph partitioning algorithm, namely a community discovery algorithm, is improved, and the joint optimization of data overhead and task parallelism in the workflow task partitioning process is realized.
According to the above thought, the technical scheme of the invention comprises the following steps:
(1) according to a task set T, a data dependency and time sequence relation E among tasks, a task complexity set L and a data transmission quantity set D in a workflow, establishing a DAG model of the workflow directed acyclic graph: g ═ { T, E, L, D };
(2) assigning a set of virtual machines S ═ S to the workflow k 1,2,3, a. Computing execution time w of tasks in a workflow on different virtual machines i,k And data transfer time c between tasks having data dependency i,j And average execution time of the task across all virtual machines
And average data transfer time between tasks
(3) Two tasks with serial structures in the workflow model G are determined and merged:
in the workflow model diagram, if one task only has one subtask and the subtask only has one father task, the subtask and the father task form a serial structure;
will have task t in serial structure i And task t i+1 Cancel the data transmission between them, and add the complexity of the two tasks and combine them into a new task t i '；
(4) Segmenting the workflow model diagram:
(4a) dividing the workflow model graph G into n sub-graphs, wherein each sub-graph comprises a vertex;
(4b) searching and trying two sub-graphs combined with edge connection in sequence, and calculating modularity increment delta Q after each combination according to tasks contained in each sub-graph and connection relation among the tasks:
if the tasks of the same layer exist in the two subgraphs, calculating the sum of the average execution time sum of the tasks of the same layer in the new subgraph after the two subgraphs are merged, and comparing the sum with the maximum value maxW of the average execution time of all the tasks of the layer:
if sum > maxW α, Δ Q ═ e i,j +e j,i -2a i a) j ＝-2(e i,j -a i a j )；
Otherwise, Δ Q ═ e i,j +e j,i -2a i a j ＝2(e i,j -a i a j ) (ii) a Wherein alpha is a comparison coefficient, the value is a number less than 1, e i,j Representing the proportion of the weight of the connecting edge between the ith sub-graph and the jth sub-graph to the sum of the weights of the total connecting edges in the graph G, a i And the proportion of the sum of the link edge weights of all tasks in the ith sub-graph to the sum of the total link edge weights in the graph G is represented.
If there are no tasks of the same layer in both subgraphs, Δ Q ═ e i,j +e j,i -2a i a j ＝2(e i,j -a i a j )；
(4c) Merging the two sub-graphs with the maximum delta Q value, and updating the modularity Q to be Q + max delta Q;
(4d) repeating the steps (4a) to (4c) until the whole graph G is combined into a sub-graph, and finding a corresponding graph division result when the modularity value is maximum, namely the optimal task partition P ═ P 1 ,p 2 ,...p x ,...,p h In which p is x Representing the x-th task partition, and h representing the number of partitions;
(5) deploying the optimal task partition to the virtual machine:
(5a) average execution time according to task
And average data transfer time between tasks
Calculating the priority rank (t) of each task i )：
Wherein succ (t) i ) Representing a task t i A set of subtasks of (2);
(5b) according to the priority rank (t) of each task i ) Calculating the priority of the task partition:
rank(p x )＝max(rank(t i )，t i ∈p x )
(5c) pressing rank (p) x ) Sorting all partitions in descending order, selecting rank (p) each time x ) Traversing all the virtual machines and calculating the sum of the total execution time of the task partition on the current virtual machine and the execution time of the deployed task on the virtual machine to find the virtual machine s with the minimum value k ；
(5d) Deploying all tasks in a task partition as a whole together to a virtual machine s k In the above, a plurality of tasks in the partition are arranged according to rank (t) i ) The values are sorted in descending order and the virtual machine will perform these tasks in sequence.
Compared with the prior art, the invention has the following advantages:
1. according to the invention, when the workflow model diagram is divided, the partitioning process of the tasks is constrained by comparing the sum of the average execution times sum of the tasks on the same layer in the partition with the maximum maxW of the average execution times of all the tasks on the same layer, so that the problem that the task parallel execution efficiency is lower due to the fact that the same task partition contains too many tasks on the same layer which can be parallel is effectively avoided, the execution time balance of the tasks on the same layer among different partitions is guaranteed, and the problem that the communication overhead is minimized at the cost of sacrificing the parallel execution efficiency of the tasks in the workflow in the existing workflow deployment algorithm based on the clustering is solved, so that the parallel execution efficiency can be effectively improved while the data transmission overhead is reduced, and the smaller workflow completion time is obtained.
2. According to the invention, when the optimal task partition is deployed on the virtual machine, the priority is calculated and sequenced for each task partition, and then the appropriate virtual machine is selected for the task partition to be deployed on the basis of the minimum execution time of the task partition, so that the completion time of the workflow is further reduced, and the execution efficiency of the workflow is improved.
Drawings
FIG. 1 is a flow chart of an implementation of the present invention;
FIG. 2 is an exemplary diagram of a workflow DAG model in the present invention;
FIG. 3 is a merging diagram of DAG serial structure in the present invention;
FIG. 4 is a schematic diagram of a workflow model graph partitioning according to the present invention;
FIG. 5 is a simulation comparison of the performance variation of the present invention and existing workflow deployment algorithms as the workflow scale increases;
FIG. 6 is a comparison graph of performance change simulation of the present invention and existing workflow deployment algorithms as the communication computation ratio increases;
FIG. 7 is a comparison graph of performance change simulation of the present invention and existing workflow deployment algorithms as the number of available virtual machines increases.
Detailed Description
Embodiments and effects of the present invention will be described in further detail below with reference to the accompanying drawings.
Referring to fig. 1, the implementation steps for the example are as follows:
The set of tasks T in the workflow is represented as: t ═ T i 1,2, ·, n }, where t is ═ i ═ 1,2 i The ith task is represented, and n is the number of tasks contained in the workflow;
the data dependency and timing relationship E between tasks is expressed as: e ═ E i,j |t i ,t j E.g., T }, wherein e i,j Value is 0 or 1, when e i,j When the value is 0, the task t is represented i And task t j There is no dependency/edge present between them when e i,j When the value is 1, the task is representedt i And task t j There is a dependency/presence edge between them;
the task complexity set L is represented as: l ═ L i |t i E.g., T }, where l i Representing a task t i The computational complexity of (2);
the set of data transmission amounts D is represented as: d ═ D i,j |t i ,t j E.g., T }, wherein d i,j Representing a task t i And task t j The amount of data transfer in between;
and according to the four element combinations, obtaining a workflow directed acyclic graph model:
G＝{T,E,L,D}，
the task set T is a vertex set of the graph G, the data dependency and time sequence relation E between tasks is an edge set of the graph G, the task complexity set L is a vertex weight set of the graph G, and the data transmission quantity set D is an edge weight set of the graph G.
The workflow directed acyclic graph model is shown in FIG. 2, where there is a directed edge e i,j Connecting two tasks t i And t j Let t be i Is t j Parent task of, t j Is t i A task without any parent task is called an entry task t enter A task without any subtasks is called an egress task t exit The number of layers of each task is determined by the maximum distance between the node and the ingress task, wherein the ingress task t 1 Is located in the first layer.
Without loss of generality, the embodiment assumes that a workflow has only one entry task and only one exit task, when a workflow has a plurality of exit tasks and entry tasks, a virtual exit task vertex or a virtual entry task vertex is added to the workflow, so that the workflow has only one exit task and one entry task, the complexity of the virtual task is set to be zero, the data transmission amount between the virtual task and other vertices is also zero, and therefore, the deployment result of the workflow is not affected by adding the virtual task vertex, and t in fig. 2 10 I.e. a virtual egress task.
And 2, calculating the execution time of each task and the data transmission time among the tasks in the workflow.
2.1) assigning a set of virtual machines S ═ S to the workflow k 1,2, 3., q }, wherein q represents the number of virtual machines, and the physical machine corresponding to each virtual machine is different;
2.2) calculating working parameters of tasks in the workflow on different virtual machines:
2.2.1) computing task t in workflow i Execution time w on different virtual machines i,k And task t i Average execution time across all virtual machines
Wherein l i Representing a task t i Of the computational complexity, v k Representing virtual machines s k Q represents the number of available virtual machines;
2.2.2) computing a task t with data dependency i And task t j Data transmission time c between i,j And average data transfer time between tasks
Wherein, the task t i And task t j Respectively deployed to virtualSimulation machine s k1 And virtual machine s k2 Upper execution, d i,j Representing a task t i And task t j Amount of data transmission between r k1,k2 Representing virtual machines s k1 And virtual machine s k2 The data transmission rate therebetween;
representing the average data transfer rate between all virtual machines; when two tasks with precedence dependency relationship are placed in the same virtual machine, the data transmission overhead between the two tasks is negligible, namely c i,j Is 0.
When task t is shown in FIG. 2 1 And task t 2 When the virtual machines are placed in the same virtual machine, extra resources are not needed for data transmission, so the data transmission overhead can be marked as 0; when task t 1 And task t 2 When the data is respectively placed in two different virtual machines, the data needs additional network resources for transmission, and it is assumed that the transmission rate r between the two virtual machines k1,k2 When the task t is equal to 1 1 And task t 2 The required data transfer time in between is 8.
And 3, merging the serial structures of the workflow model diagrams.
In the workflow model diagram, if one task only has one subtask and the subtask only has one father task, the subtask and the father task form a serial structure;
will have task t in serial structure i And task t i+1 The two tasks are added and merged into a new task t' i And updating the calculation complexity and the workflow model diagram to form a new model diagram G'.
As shown in FIG. 3, task t i And task t i+1 Two tasks with serial structures are combined to form a new task t' i And updating the new task computational complexity and the workflow model diagram as follows:
w′ i ＝w i +w i+1
pre(t′ i )＝pre(t i )
succ(t′ i )＝succ(t i+1 )
wherein, w' i Represents a new task t' i Execution time of (w) i Representing a task t i Execution time of (w) i+1 Representing a task t i+1 Execution time of pre (t' i ) Represents a new task t' i Parent task set of, succ (t' i ) Represents a new task t' i Is a set of subtasks, pre (t) i ) Representing a task t i Of the parent task set, succ (t) i+1 ) Representing a task t i+1 Is performed on the set of subtasks.
And 4, segmenting a new workflow model graph G' formed after the serial structures are combined.
4.1) dividing the workflow model graph G' into n sub-graphs, wherein each sub-graph comprises a vertex;
4.2) searching and trying to combine two subgraphs with edge connection in sequence, and calculating modularity increment delta Q after each combination according to tasks contained in each subgraph and connection relation among the tasks:
if the two subgraphs have the tasks of the same layer, calculating the sum of the average execution time of the tasks of the same layer in the new subgraph after the two subgraphs are merged, and comparing the sum with the maximum value maxW of the average execution time of all the tasks of the layer:
if sum > maxW α, Δ Q ═ e i,j +e j,i -2a i a) j ＝-2(e i,j -a i a j )；
Otherwise, Δ Q ═ e i,j +e j,i -2a i a j ＝2(e i,j -a i a j ) (ii) a Wherein alpha is a comparison coefficient, the value is a number less than 1, e i,j Representing the proportion of the weight of the connecting edge between the ith sub-graph and the jth sub-graph to the sum of the total weight of the connecting edges in the graph G', a i Representing the proportion of the sum of the link edge weights of all tasks in the ith sub-graph to the sum of the total link edge weights in the graph G';
if there are no tasks of the same layer in both subgraphs, Δ Q ═ e i,j +e j,i -2a i a j ＝2(e i,j -a i a j )；
Wherein e is i,j The ratio of the weight of the connecting edge between the ith sub-graph and the jth sub-graph to the sum of the total weight of the connecting edges in the graph G', a i Calculating the proportion of the sum of the link edge weights of all tasks in the ith sub-graph to the sum of the total link edge weights in the graph G' as follows:
a i ＝k i /2m
wherein, A is i,j And m is the sum of all the continuous edge weights in the graph G'. k is a radical of i Is the sum of all the connected edge weights connected with the ith task.
4.3) repeat 4.1) and 4.2) until the entire graph G' merges into a community. Finding out the community structure corresponding to the maximum modularity value, i.e. finding out the optimal partitioning scheme P ═ P 1 ,p 2 ,...p h }。
FIG. 4 shows an example of workflow task segmentation, which is a workflow consisting of five tasks, where t 2 ,t 3 ,t 4 The three tasks belong to the same layer and can be executed in parallel, the execution time of the three tasks is respectively 15, 10 and 25, when sub-graph division is carried out, the three tasks are divided into the same partition to be executed in sequence, obviously, the parallelism of workflow execution is reduced, and the completion time is prolonged; another possible result is to assign t 3 And t 4 Division into the same partition, t 2 The task execution time of the same layer between the two partitions is not balanced, and the two partitions need to wait for each other in actual execution, so that the workflow completion time is prolonged; on the contrary, if t is to be 2 And t 3 Division into the same partition, t 4 When the task is divided into another partition, the execution time of the task in the same layer of the two partitions is distributed uniformly, and the redundant waiting time can be prevented from being increased.
And 5, mapping the optimal task partition to the virtual machine to complete the deployment of the workflow.
5.1) average execution time according to task
And average data transfer time between tasks
Calculating the priority rank (t) of each task i )：
Wherein succ (t) i ) Representing a task t i A set of subtasks of (2);
5.2) priority rank (t) according to each task i ) Calculating the priority of the task partition:
rank(p x )＝max(rank(t i )，t i ∈p x )
5.3) priority rank (p) of partition by task x ) All partitions are sorted in descending order, and rank (p) is selected each time x ) Traversing all the virtual machines and calculating the sum of the total execution time of the task partition on the current virtual machine and the execution time of the deployed task on the virtual machine, and finding out the virtual machine s with the minimum value k ；
5.4) deploying all tasks in the task partition as a whole together to the virtual machine s k And the plurality of tasks in the partition are arranged according to rank (t) i ) The values are arranged in a descending order, and the virtual machine can sequentially execute the tasks according to the order to complete the deployment of the workflow.
The effect of the present invention is further explained by combining the simulation experiment as follows:
1. setting simulation parameters:
randomly generating a workflow directed acyclic graph DAG, and setting simulation parameters as shown in Table 1:
table 1 workflow deployment simulation parameter settings
| Parameter(s) | Taking values: fixed value (variance value) |
| Workflow size N | 50(20,40,60,80,100) |
| Communication calculation ratio CCR | 0.5(0.4,0.8,1.2,1.6,2) |
| Number of available virtual machines K | 5(1,2,3,4,5,6) |
In table 1, the workflow scale refers to the number of tasks included in the workflow; the communication computation ratio CCR refers to the ratio of the sum of average data transmission time among tasks to the sum of average execution time of all tasks, a higher CCR means that the workflow is a communication-intensive application, and conversely, a lower CCR means that the workflow is a computation-intensive application;
2. simulation content and result analysis thereof:
the method comprises the steps of selecting two workflow deployment algorithms, namely an existing list-based deployment algorithm HEFT and a clustering-based deployment algorithm PDWA, from three aspects of influence of workflow scale on algorithm performance, influence of communication calculation ratio on algorithm performance and influence of virtual machine number on algorithm performance, and evaluating and comparing deployment results of the method.
fig. 5(a) is a comparison graph of changes of the scheduling length ratio SLR of the three methods with the increase of the workflow scale, wherein the abscissa is the number of tasks in the workflow, the ordinate is the scheduling length ratio SLR representing the completion time of the workflow, and the larger the SLR value is, the larger the completion time of the workflow is, the worse the performance is. As can be seen from fig. 5(a), compared with the PDWA based on the clustering deployment algorithm, the workflow completion time of the present invention is reduced by 75% on average, because the PDWA artificially specifies the size of the workflow partition, the number of the partitions is relatively fixed, and the parallelism of task execution is not considered, which results in the prolongation of the workflow completion time.
Fig. 5(b) is a comparison graph of the speed change of the three methods along with the increase of the workflow scale, the abscissa of the comparison graph is the number of tasks in the workflow, the ordinate of the comparison graph is the speed representing the parallel execution efficiency of the workflow, and the larger the speed value is, the higher the parallel execution efficiency of the deployment method is, the better the performance is. As can be seen from fig. 5(b), with the gradual increase of the workflow scale, the speedup value of the method of the present invention is always larger compared with the cluster-based deployment algorithm PDWA. The method fully considers the dependency and the parallelism of the tasks in the workflow when the tasks are partitioned, reduces the data transmission overhead among communities, simultaneously considers the parallel execution efficiency of the tasks in the workflow, and enables the tasks at the same layer to be efficiently and parallelly executed.
Fig. 5(c) is a comparison graph of the communication overhead variation of the three methods with the increase of the workflow scale, wherein the abscissa is the number of tasks in the workflow and the ordinate is the communication overhead. As can be seen from fig. 5(c), as the size of the workflow gradually increases, the communication overhead tends to increase, and the method and the cluster-based deployment algorithm PDWA always have lower communication overhead compared with the list-based algorithm het, which, in conjunction with fig. 5(a) and 5(b), illustrates that the method can achieve reduction of the completion time of the workflow and significant improvement of the parallel execution efficiency while maintaining smaller communication overhead.
fig. 6(a) is a graph comparing changes of the scheduling length ratio SLR with the increase of CCR in the three methods, where the abscissa is the communication calculation ratio CCR and the ordinate is the scheduling length ratio SLR. As can be seen from fig. 6(a), under the condition that CCR is continuously increased, workflow completion time of the method of the present invention is always smaller than that of the cluster-based deployment algorithm PDWA, and with the change of CCR value, the performance of the method of the present invention and the list-based algorithm het is relatively stable, while SLR value of PDWA increases relatively significantly with the increase of CCR value, because with the increase of proportion of data transmission time, the cluster-based deployment algorithm PDWA is gradually out of advantage in the policy of only minimizing communication overhead on the basis of specifying the size of workflow partition.
FIG. 6(b) is a comparison graph of speed change with increasing CCR for the three methods, with the abscissa being the number of tasks in the workflow and the ordinate being speed. As can be seen from fig. 6(b), under the condition that CCR is continuously increased, the speedup value of the method of the present invention has obvious advantages compared with the deployment algorithm PDWA based on clustering, and is highly adaptive, because parallelism of the workflow task is fully considered when the workflow partition is performed by the present invention.
Fig. 6(c) is a graph comparing the change in communication overhead with the increase in CCR for the three methods, with the abscissa being the number of tasks in the workflow and the ordinate being the communication overhead. As can be seen from fig. 6(c), the inventive method and the cluster-based deployment algorithm PDWA have a lower communication overhead compared to the list-based algorithm het. With reference to fig. 6(a) and fig. 6(b), it can be verified that the method of the present invention can effectively improve the execution efficiency of the workflow while maintaining a small communication overhead.
fig. 7(a) is a graph comparing changes of the scheduling length ratio SLR with an increase in the number of available virtual machines in the three methods, where the abscissa is the number of available virtual machines and the ordinate is the scheduling length ratio SLR. As can be seen from fig. 7(a), as the number of available virtual machines increases, the value of the workflow SLR decreases, and the advantages of the method of the present invention are also gradually obvious, because as the number of virtual machines increases, more tasks are allowed to be processed in parallel on different virtual machines during the workflow execution process, and when the task is partitioned, the method of the present invention considers the task parallelism and the execution time balance and the data transmission overhead between partitions at the same time, so that when a plurality of virtual machines are executed in parallel, the excessive mutual waiting time between different partitions is avoided, and the completion time of the workflow is effectively reduced.
Fig. 7(b) is a comparison graph of speed change of the three methods with the increase of the number of available virtual machines, wherein the abscissa is the number of available virtual machines and the ordinate is the speed. As can be seen from fig. 7(b), as the number of available virtual machines increases, the speed value also increases, and the advantages of the method of the present invention are gradually obvious. The reason is that when the method is used for partitioning, not only is the communication overhead among the partitions minimized, but also the parallel execution efficiency of the tasks in the workflow execution process is considered, and with the increase of the number of available virtual machines, more partitions are allowed to be processed in parallel on different virtual machines in the workflow execution process, so that the speed value can be increased with the increase of the number of the available virtual machines.
Fig. 7(c) is a comparison graph of the communication overhead variation of the three methods with the increase of the number of available virtual machines, wherein the abscissa represents the number of available virtual machines and the ordinate represents the communication overhead. As can be seen from fig. 7(c), the inventive method and the cluster-based deployment algorithm PDWA have a lower communication overhead compared to the list-based algorithm het. As can be seen from fig. 7(a) and 7(b), the method of the present invention can effectively improve the parallel execution efficiency and reduce the workflow completion time while maintaining low communication overhead.
Claims (7)
1. A workflow deployment method based on graph partitioning is characterized by comprising the following steps:
(1) according to a task set T, a data dependency and time sequence relation E among tasks, a task complexity set L and a data transmission quantity set D in a workflow, establishing a DAG model of the workflow directed acyclic graph: g ═ { T, E, L, D };
(2) assigning a set of virtual machines S ═ S to the workflow k 1,2,3, a. Computing execution time w of tasks in a workflow on different virtual machines i,k And data transfer time c between tasks having data dependency i,j And average execution time of the task across all virtual machines
And average data transfer time between tasks
(3) Determining two tasks with serial structures in the workflow model G, and merging the two tasks to obtain a new workflow model diagram G':
in the workflow model diagram, if one task only has one subtask and the subtask only has one father task, the subtask and the father task form a serial structure;
will have task t in serial structure i And task t i+1 Cancels the data transmission between the two tasks and adds and combines the two tasks into a new task t' i ；
(4) And (3) dividing a new workflow model graph G' formed after the serial structures are combined:
(4a) dividing the workflow model graph G' into n sub-graphs, wherein each sub-graph comprises a vertex;
(4b) searching and trying two sub-graphs combined with edge connection in sequence, and calculating modularity increment delta Q after each combination according to tasks contained in each sub-graph and connection relations among the tasks:
if the tasks of the same layer exist in the two subgraphs, calculating the sum of the average execution time sum of the tasks of the same layer in the new subgraph after the two subgraphs are merged, and comparing the sum with the maximum value maxW of the average execution time of all the tasks of the layer:
if sum > maxW α, Δ Q ═ e i,j +e j,i -2a i a) j ＝-2(e i,j -a i a j )；
Otherwise, Δ Q ═ e i,j +e j,i -2a i a j ＝2(e i,j -a i a j ) (ii) a Wherein alpha is a comparison coefficient, the value is a number less than 1, e i,j Representing the proportion of the weight of the connecting edge between the ith sub-graph and the jth sub-graph to the sum of the weights of the total connecting edges in the graph G, a i And the proportion of the sum of the link edge weights of all tasks in the ith sub-graph to the sum of the total link edge weights in the graph G is represented.
If there are no tasks of the same layer in both subgraphs, Δ Q ═ e i,j +e j,i -2a i a j ＝2(e i,j -a i a j )；
(4c) Merging the two sub-graphs with the maximum delta Q value, and updating the modularity Q to be Q + max delta Q;
(4d) repeating the steps (4a) to (4c) until the whole graph G' is merged into a sub-graph, and finding a corresponding graph division result when the modularity value is maximum, namely the optimal task partition P ═ P 1 ,p 2 ,...p x ,...,p h In which p is x Representing the x-th task partition, and h representing the number of partitions;
(5) mapping the optimal task partition to a virtual machine to complete the deployment of the workflow:
(5a) average execution time according to task
And average data transfer time between tasks
Calculating the priority ran of each taskk(t i )：
Wherein succ (t) i ) Representing a task t i A set of subtasks of (a);
(5b) according to the priority rank (t) of each task i ) Calculating the priority of the task partition:
rank(p x )＝max(rank(t i )，t i ∈p x )
(5c) according to rank (p) x ) Sorting all partitions in descending order, selecting rank (p) each time x ) Traversing all the virtual machines and calculating the sum of the total execution time of the task partition on the current virtual machine and the execution time of the deployed task on the virtual machine to find the virtual machine s with the minimum value k ；
(5d) Deploying all tasks in a task partition as a whole together to a virtual machine s k In the above, a plurality of tasks in the partition are arranged according to rank (t) i ) The values are sorted in descending order and the virtual machine will perform these tasks in sequence.
2. The method of claim 1, wherein: in the step (1), a workflow directed acyclic graph model G is established, and the following is realized:
(1a) the set of tasks T in the workflow is represented as: t ═ T i 1,2, ·, n }, where t is ═ i ═ 1,2 i Representing the ith task, wherein n is the number of tasks contained in the workflow;
(1b) the data dependency and timing relationship E between tasks is expressed as: e ═ E i,j |t i ,t j E.g., T }, wherein e i,j Value is 0 or 1, when e i,j When the value is 0, the task t is represented i And task t j There is no dependency/edge present between them when e i,j When the value is 1, the task t is represented i And task t j There is a dependency/existing edge between them, there is a directed edge e i,j Connecting two tasks t i And t j Let t be i Is t j Parent task of, t j Is t i The sub task of (2), the task without any parent task is called an entry task;
(1c) the task complexity set L is represented as: l ═ L i |t i E.g., T }, where l i Representing a task t i The computational complexity of (2);
(1d) the set of data transmission amounts D is represented as: d ═ D i,j |t i ,t j E.g., T }, wherein d i,j Representing a task t i And task t j The amount of data transfer in between;
(1e) and combining the four elements to obtain a workflow directed acyclic graph model G ═ T, E, L and D.
3. The method of claim 1, wherein: in the step (2), the execution time w of the tasks in the workflow on different virtual machines is calculated i,k And average execution time of tasks on all virtual machines
The formula is as follows:
wherein w i,k Representing a task t i In virtual machines s k The execution time of the first time slot is greater than the execution time of the second time slot,
representing a task t i Average execution time over all virtual machines, l i Representing a task t i Of the computational complexity, v k Representing virtual machines s k Q represents the number of available virtual machines.
4. The method of claim 1, wherein: calculating the data transmission time c between tasks with data dependency relationship in step (2) i,j Average data transfer time between tasks
The formula is as follows:
wherein, c i,j Representing a task t i And task t j Time of data transmission between, task t i And task t j Respectively deployed to virtual machines s k1 And virtual machine s k2 Upper execution, d i,j Representing a task t i And task t j Amount of data transmission between r k1,k2 Representing virtual machines s k1 And virtual machine s k2 The data transmission rate therebetween;
representing the ith task t i And the jth task t j The average data transfer time in between,
representing the average data transfer rate between all virtual machines; when two tasks with sequential dependency relationship are placed in the same virtual machine, the data transmission overhead between the tasks is negligible, namely c i,j Is 0.
5. The method of claim 1, wherein: in the step (4b), the number of layers in which each task is located is determined by the maximum distance between the node and the ingress task.
6. The method of claim 1, wherein: step (4b) calculating the proportion e of the connecting edge weight between the ith sub-graph and the jth sub-graph related in the modularity increment delta Q formula to the sum of the total connecting edge weight in the graph G i,j The following were determined:
wherein, A i,j And m is the sum of all the continuous edge weights in the graph G'.
7. The method of claim 1, wherein: step (4b) calculating the proportion a of the sum of the link edge weights of all tasks in the ith sub-graph related in the modularity increment delta Q formula to the sum of the total link edge weights in the graph G i The following were determined:
a i ＝k i /2m
wherein k is i M is the sum of all the connected edge weights connected with the ith task, and m is the sum of all the connected edge weights in the graph G'.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210730454.9A CN115080236B (en) | 2022-06-24 | 2022-06-24 | Workflow deployment method based on graph segmentation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210730454.9A CN115080236B (en) | 2022-06-24 | 2022-06-24 | Workflow deployment method based on graph segmentation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115080236A true CN115080236A (en) | 2022-09-20 |
| CN115080236B CN115080236B (en) | 2024-04-16 |
Family
ID=83255512
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210730454.9A Active CN115080236B (en) | 2022-06-24 | 2022-06-24 | Workflow deployment method based on graph segmentation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115080236B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107193658A (en) * | 2017-05-25 | 2017-09-22 | 重庆工程学院 | Cloud computing resource scheduling method based on game theory |
| US20170371709A1 (en) * | 2016-06-23 | 2017-12-28 | International Business Machines Corporation | Optimizing simultaneous startup or modification of inter-dependent machines with specified priorities |
| CN108108225A (en) * | 2017-12-14 | 2018-06-01 | 长春工程学院 | A kind of method for scheduling task towards cloud computing platform |
| CN109298930A (en) * | 2017-07-24 | 2019-02-01 | 西安电子科技大学昆山创新研究院 | A kind of cloud workflow schedule method and device based on multiple-objection optimization |
| CN109634742A (en) * | 2018-11-15 | 2019-04-16 | 华南理工大学 | A kind of time-constrain scientific workflow optimization method based on ant group algorithm |
| CN110008026A (en) * | 2019-04-09 | 2019-07-12 | 中国科学院上海高等研究院 | Job scheduling method, device, terminal and the medium divided equally based on additional budget |
| WO2021056787A1 (en) * | 2019-09-23 | 2021-04-01 | 苏州大学 | Hybrid cloud service process scheduling method |
| CN113032155A (en) * | 2021-05-25 | 2021-06-25 | 深圳大学 | Cloud edge resource cooperative scheduling method driven by time-space data visualization task |
-
2022
- 2022-06-24 CN CN202210730454.9A patent/CN115080236B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170371709A1 (en) * | 2016-06-23 | 2017-12-28 | International Business Machines Corporation | Optimizing simultaneous startup or modification of inter-dependent machines with specified priorities |
| CN107193658A (en) * | 2017-05-25 | 2017-09-22 | 重庆工程学院 | Cloud computing resource scheduling method based on game theory |
| CN109298930A (en) * | 2017-07-24 | 2019-02-01 | 西安电子科技大学昆山创新研究院 | A kind of cloud workflow schedule method and device based on multiple-objection optimization |
| CN108108225A (en) * | 2017-12-14 | 2018-06-01 | 长春工程学院 | A kind of method for scheduling task towards cloud computing platform |
| CN109634742A (en) * | 2018-11-15 | 2019-04-16 | 华南理工大学 | A kind of time-constrain scientific workflow optimization method based on ant group algorithm |
| CN110008026A (en) * | 2019-04-09 | 2019-07-12 | 中国科学院上海高等研究院 | Job scheduling method, device, terminal and the medium divided equally based on additional budget |
| WO2021056787A1 (en) * | 2019-09-23 | 2021-04-01 | 苏州大学 | Hybrid cloud service process scheduling method |
| CN113032155A (en) * | 2021-05-25 | 2021-06-25 | 深圳大学 | Cloud edge resource cooperative scheduling method driven by time-space data visualization task |
Non-Patent Citations (4)
| Title |
|---|
| GO学堂: ""Go实战 | 基于有向无环图的并发执行流的实现"", Retrieved from the Internet <URL:《https://segmentfault.com/a/1190000041267350》> * |
| HEBA SALEH: ""IPSO Task Scheduling Algorithm for Large Scale Data in Cloud Computing Environment"", 《IEEE ACCESS》, vol. 7, 28 December 2018 (2018-12-28), pages 5412 - 5420, XP011705567, DOI: 10.1109/ACCESS.2018.2890067 * |
| 曹书锦: ""云环境下截止期约束的科学工作流优化调度算法研究"", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 2022, 15 February 2022 (2022-02-15), pages 138 - 3 * |
| 马英红: ""通信计算联合优化的图分割工作流部署方法"", 《西安电子科技大学学报（网络首发）》, 4 January 2024 (2024-01-04), pages 1 - 16 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115080236B (en) | 2024-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108260169B (en) | QoS guarantee-based dynamic service function chain deployment method | |
| CN108566659B (en) | 5G network slice online mapping method based on reliability | |
| CN111722910B (en) | Cloud job scheduling and resource allocation method | |
| US7752628B2 (en) | Method and apparatus for reassigning objects to processing units | |
| CN112988345B (en) | Dependency task unloading method and device based on mobile edge calculation | |
| CN112380008A (en) | Multi-user fine-grained task unloading scheduling method for mobile edge computing application | |
| CN112364590B (en) | Construction method of practical logic verification architecture-level FPGA (field programmable Gate array) wiring unit | |
| Zhang et al. | The real-time scheduling strategy based on traffic and load balancing in storm | |
| CN110191155B (en) | Parallel job scheduling method, system and storage medium for fat tree interconnection network | |
| Schlag et al. | Scalable edge partitioning | |
| CN114996001A (en) | Distributed machine learning task GPU resource scheduling and distributing method and system | |
| Ahmad et al. | Task assignment using a problem‐space genetic algorithm | |
| CN107360031B (en) | Virtual network mapping method based on optimized overhead-to-revenue ratio | |
| Chai et al. | A parallel placement approach for service function chain using deep reinforcement learning | |
| CN111159859B (en) | Cloud container cluster deployment method and system | |
| CN113672391A (en) | Parallel computing task scheduling method and system based on Kubernetes | |
| CN114500560A (en) | Edge node service deployment and load balancing method for minimizing network delay | |
| Lin et al. | Column generation based service function chaining embedding in multi-domain networks | |
| Elsharkawey et al. | Mlrts: multi-level real-time scheduling algorithm for load balancing in fog computing environment | |
| CN110138830B (en) | Cross-data center task scheduling and bandwidth allocation method based on hypergraph segmentation | |
| CN111131447A (en) | Load balancing method based on intermediate node task allocation | |
| CN110048966A (en) | The Coflow dispatching method of minimum overhead based on deadline | |
| CN111245701B (en) | Link priority virtual network mapping method based on maximum weighted matching | |
| CN108833295B (en) | SDN-oriented virtual network reconfiguration method based on tabu search | |
| CN112596910A (en) | Cloud computing resource scheduling method in multi-user MEC system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |