CN107656799B - Workflow scheduling method considering communication and computing cost under multi-cloud environment - Google Patents

Abstract

The invention relates to a workflow scheduling method considering communication and computing cost in a multi-cloud environment. The method is based on the self structure and the execution characteristics of the workflow, the related factors of the current cloud resource environment communication and execution cost, and the thought of random two-point cross operation and random single-point variation operation based on the genetic algorithm, so that the diversity in the population evolution process is improved, the virtualized resources are integrated, the data communication cost and the task calculation cost are considered, the resource utilization rate is optimized, and the execution cost is reduced on the premise of meeting the workflow deadline. The method has good performance in terms of meeting the workflow deadline and controlling the execution cost under the condition of existence of fluctuation factors, and reduces the execution cost on the premise of meeting the workflow deadline as much as possible.

Description

Workflow scheduling method considering communication and computing cost under multi-cloud environment

Technical Field

The invention belongs to a workflow scheduling method in the field of parallel and distributed high-performance computing, and particularly relates to a workflow scheduling method considering communication and computing cost in a multi-cloud environment.

Background

With the continuous development of cloud computing technology, a 'multi-public cloud' situation in which a plurality of cloud service providers coexist appears in the current cloud market. The nature of cloud computing to flexibly provision virtual resources and pay as needed facilitates the processing of large-scale scientific workflows (hereinafter referred to as 'workflows'). However, task scheduling in a cloud heterogeneous environment is an NP-hard problem, complex time dependency and data dependency relationship exist between subtasks of a workflow itself, and a plurality of differences (such as asking price mechanism, instance type, communication bandwidth, etc.) exist between a plurality of cloud Service providers, so that an appropriate scheduling method is needed to reduce the execution cost of the workflow on the premise of satisfying the Quality of Service (QoS) of the workflow as much as possible. Most of the workflow scheduling methods in the current cloud environment are improved on the basis of workflow scheduling algorithms in the traditional distributed computing environment (such as grids), and the characteristics of the cloud environment are not considered. Or some scheduling methods only consider in a static single cloud environment, and simply pursue the objective of minimizing the execution time, and do not research the problem of optimizing and scheduling the cost of the workflow with constraints (such as due date).

In recent years, workflow scheduling in a conventional distributed environment has been widely studied. The workflow scheduling scheme in the grid environment generally achieves the purpose of optimizing the execution time of the workflow and meeting the QoS requirement through a heuristic or meta-heuristic scheduling algorithm. Improving the resource utilization rate in the grid environment, and the like. However, cloud computing environments and grid environments have a large difference in resource provisioning and resource pricing mechanisms. Since the motivation, the scheme for deployment, and the services provided by the two are different, the workflow scheduling method in the grid environment cannot be simply applied to the cloud computing environment. Cost-driven workflow scheduling algorithm with preparation time and deadline constraints in a cloud environment is partially researched and designed, however, mostly only one virtual machine type is considered, and the cloud environment is not in accordance with the reality. In addition, related research works propose a single workflow scheduling scheme based on a PSO algorithm, which respectively develop research on a cost optimization problem of deadline constraints and an execution time optimization problem of budget constraints. But the type and the number of the virtual machines for working are fixed, and the property of cloud environment elastic supply is not met. At present, a single cloud service provider is considered in workflow scheduling work in a cloud environment, and research is not carried out on a multi-cloud environment.

Regarding the cost-driven scheduling problem of the workflow with deadline constraint in the cloud environment, no relevant research work exists at home and abroad at present. Among them, the most relevant research work is workflow scheduling research based on deadline constraints with fluctuating factors about single cloud environments, which deals with global workflow subtask scheduling schemes using traditional PSO scheduling methods. However, in the current research work, a scientific workflow scheduling method with deadline constraint for considering communication and calculation cost under a multi-public cloud environment is not yet involved.

Disclosure of Invention

The invention aims to provide a workflow scheduling method considering communication and calculation cost in a multi-cloud environment, which has good performance in the aspects of meeting workflow deadline date and controlling execution cost under the condition of existence of fluctuation factors, and reduces the execution cost on the premise of meeting the workflow deadline date as far as possible.

In order to achieve the purpose, the technical scheme of the invention is as follows: a workflow scheduling method considering communication and computing costs in a multi-cloud environment is based on the structure and execution characteristics of a workflow and relevant factors of communication and execution costs of a current cloud resource environment, and based on the idea of random two-point cross operation and random single-point variation operation of a genetic algorithm, diversity in a population evolution process is improved, virtualized resources are integrated, data communication costs and task computing costs are considered, resource utilization rate is optimized, and the execution costs are reduced on the premise that the deadline of the workflow is met.

In an embodiment of the present invention, the workflow scheduling method is specifically implemented as follows,

defining the workflow scheduling method as S ═ Re, Map, T_total,C_total) Where Re represents a set of virtual machine resources Re ═ { vm) that need to be enabled₁,vm₂,...,vm_r}，Map＝{(t_i,vm_j)|t_i∈V,vm_jE.g. Re represents the mapping relation of the subtask corresponding to the virtual machine resource Re in the workflow, T_totalRepresents the execution completion time of the workflow, and C_totalThen the total workflow execution cost is represented; the workflow is represented by a directed acyclic graph G (V, E), where V represents a set of vertices { t } containing n tasks₁,t₂,...,t_nE represents the data dependency relationship between tasks E₁₂,e₁₃,...,e_ij}; each data dependent edge e_ij＝(t_i,t_j) Representing a subtask t_iAnd subtask t_jThere is a data dependency relationship between, where the subtask t_iIs a subtask t_jOf the direct predecessor node, and sub-task t_jThen it is the subtask t_iThe direct successor node of (1);

each virtual machine has a corresponding virtual machine type s_piAnd corresponding switch-on times Tls (vm)_i) And a closing time Tle (vm)_i) (ii) a When the subtasks are scheduled to be completed, there is a corresponding set of actual execution start times AST (t)_i) And actual execution completion time AET (t)_i) And no more data will be generated and communicated upon tasking; thus, the workflow execution completion time T_totalAnd a corresponding total execution cost C_totalAs shown in the following equations, respectively:

the first half of the formula (2) represents the execution cost of the virtual machine, and the second half represents the data communication cost; lambda [ alpha ]_pThe cloud service provider p sets specific asking price unit time and subtask t for the service provided by the cloud service provider p_kIs a subtask t_jP (t) of_j) And p (t)_k) Respectively represent execution t_jAnd t_kThe service provider of (2); when t is_jAnd t_kWhen executed by the same cloud service provider, s_jkIs 0, i.e. no inter-cloud data communication is generated, otherwise s_jkIs 1;

based on the related definitions, the workflow scheduling problem with the deadline constraint in the multi-cloud environment can be formally expressed as a formula (3), and the core idea is to pursue the execution cost C_totalAt the lowest, make the execution time T_totalLess than or equal to workflow expiration date D (w);

then, the following algorithm is executed to achieve the execution cost C_totalAt the lowest, make the execution time T_totalWorkflow expiration date D (w) or less:

s1: the initialization related parameters are as follows: population size 100, maximum iteration number 1000, inertia weight factor w and cognition factor parameter c₁_start＝0.9，c₁_end＝0.2，c₂_start＝0.4，c₂0.9, generating an initial population;

s2: calculating the fitness values of the particles under different conditions according to a particle mapping strategy and fitness functions of the particles, namely formulas (4), (5) and (6), selecting the particle with the minimum fitness value as a population global optimal particle, and setting each particle in the first generation as a self historical optimal particle;

wherein, formula (4) represents the fitness function of the particle when one particle is a feasible solution and the other particle is an infeasible solution, formula (5) represents the fitness function of the particle when both particles are feasible solutions, and formula (6) represents the fitness function of the particle when both particles are infeasible solutions;

s3: updating the particles according to the particle updating formulas (7) to (10);

wherein, formula (7) represents the update mode of the particle i at time t, M_u() Denotes mutation operation, C_g() And C_p() It is indicated that the cross-over operation,

and gBest^tRespectively obtaining the self historical optimal position of the particle i and the historical optimal position of the whole population after t iterations; equation (8) represents the update mode of the inertial part, r₁Is a random number between 0 and 1; equations (9) and (10) represent the update methods of the individual recognition and social recognition parts, respectively, and r₂And r₃Is a random number between 0 and 1;

s4: recalculating the fitness value of each particle, and if the fitness value of the current particle is smaller than the historical optimal value of the current particle, updating the new particle into the historical optimal particle of the current particle;

s5: if the fitness value of the current particle is smaller than that of the population global optimal particle, updating the particle into the population global optimal particle;

s6: checking whether an algorithm termination condition is met, and if so, terminating the algorithm; otherwise, go to S3.

Compared with the prior art, the invention has the following beneficial effects: the method has good performance in terms of meeting the workflow deadline and controlling the execution cost under the condition of existence of fluctuation factors, and reduces the execution cost on the premise of meeting the workflow deadline as much as possible.

Drawings

FIG. 1 is a workflow scheduling framework diagram.

FIG. 2 is a particle code pattern.

FIG. 3 is a diagram of the operation of the mutation operator in the inertial portion.

FIG. 4 is a diagram of the operation of crossover operators in the human (social) cognitive segment.

Fig. 5 is an algorithmic flow chart of a scheduling method.

Detailed Description

The technical scheme of the invention is specifically explained below with reference to the accompanying drawings.

The workflow scheduling method considering communication and computing costs in a multi-cloud environment improves diversity in a population evolution process, integrates virtualized resources, considers data communication costs and task computing costs, optimizes resource utilization rate and reduces the execution costs on the premise of meeting workflow deadline date, based on the self structure and execution characteristics of the workflow and relevant factors of the communication and execution costs of the current cloud resource environment and based on the ideas of random two-point cross operation and random single-point variation operation of a genetic algorithm; the concrete implementation is as follows,

defining the workflow scheduling method as S ═ Re, Map, T_total,C_total) Where Re represents a set of virtual machine resources Re ═ { vm) that need to be enabled₁,vm₂,...,vm_r}，Map＝{(t_i,vm_j)|t_i∈V,vm_jE.g. Re represents the mapping relation of the subtask corresponding to the virtual machine resource Re in the workflow, T_totalRepresenting execution of a workflowCompletion time, and C_totalThen the total workflow execution cost is represented; the workflow is represented by a directed acyclic graph G (V, E), where V represents a set of vertices { t } containing n tasks₁,t₂,...,t_nE represents the data dependency relationship between tasks E₁₂,e₁₃,...,e_ij}; each data dependent edge e_ij＝(t_i,t_j) Representing a subtask t_iAnd subtask t_jThere is a data dependency relationship between, where the subtask t_iIs a subtask t_jOf the direct predecessor node, and sub-task t_jThen it is the subtask t_iThe direct successor node of (1);

each virtual machine has a corresponding virtual machine type s_piAnd corresponding switch-on times Tls (vm)_i) And a closing time Tle (vm)_i) (ii) a When the subtasks are scheduled to be completed, there is a corresponding set of actual execution start times AST (t)_i) And actual execution completion time AET (t)_i) And no more data will be generated and communicated upon tasking; thus, the workflow execution completion time T_totalAnd a corresponding total execution cost C_totalAs shown in the following equations, respectively:

the first half of the formula (2) represents the execution cost of the virtual machine, and the second half represents the data communication cost; lambda [ alpha ]_pThe cloud service provider p sets specific asking price unit time and subtask t for the service provided by the cloud service provider p_kIs a subtask t_jP (t) of_j) And p (t)_k) Respectively represent execution t_jAnd t_kThe service provider of (2); when t is_jAnd t_kWhen executed by the same cloud service provider, s_jkIs 0, i.e. no inter-cloud data communication is generated, otherwise s_jkIs 1;

based on the related definitions, the workflow scheduling problem with the deadline constraint in the multi-cloud environment can be formally expressed as a formula (3), and the core idea is to pursue the execution cost C_totalAt the lowest, make the execution time T_totalLess than or equal to workflow expiration date D (w);

then, the following algorithm is executed to achieve the execution cost C_totalAt the lowest, make the execution time T_totalWorkflow expiration date D (w) or less:

s1: the initialization related parameters are as follows: population size 100, maximum iteration number 1000, inertia weight factor w and cognition factor parameter c₁_start＝0.9，c₁_end＝0.2，c₂_start＝0.4，c₂0.9, generating an initial population;

s2: calculating the fitness values of the particles under different conditions according to a particle mapping strategy and fitness functions of the particles, namely formulas (4), (5) and (6), selecting the particle with the minimum fitness value as a population global optimal particle, and setting each particle in the first generation as a self historical optimal particle;

wherein, formula (4) represents the fitness function of the particle when one particle is a feasible solution and the other particle is an infeasible solution, formula (5) represents the fitness function of the particle when both particles are feasible solutions, and formula (6) represents the fitness function of the particle when both particles are infeasible solutions;

s3: updating the particles according to the particle updating formulas (7) to (10);

wherein, formula (7) represents the update mode of the particle i at time t, M_u() Denotes mutation operation, C_g() And C_p() It is indicated that the cross-over operation,

and gBest^tRespectively obtaining the self historical optimal position of the particle i and the historical optimal position of the whole population after t iterations; equation (8) represents the update mode of the inertial part, r₁Is a random number between 0 and 1; equations (9) and (10) represent the update methods of the individual recognition and social recognition parts, respectively, and r₂And r₃Is a random number between 0 and 1;

s4: recalculating the fitness value of each particle, and if the fitness value of the current particle is smaller than the historical optimal value of the current particle, updating the new particle into the historical optimal particle of the current particle;

s5: if the fitness value of the current particle is smaller than that of the population global optimal particle, updating the particle into the population global optimal particle;

s6: checking whether an algorithm termination condition is met, and if so, terminating the algorithm; otherwise, go to S3.

FIG. 1 is a diagram of a workflow scheduling framework defined by the present invention. It mainly includes workflows, a cloudy environment, and a cost driven scheduler.

The workflow w is represented by a directed acyclic graph G (V, E), where V represents a set of vertices { t } containing n tasks₁,t₂,...,t_nE represents the data dependency relationship between tasks E₁₂,e₁₃,...,e_ij}. Each data dependent edge e_ij＝(t_i,t_j) Representative task t_iAnd task t_jThere is a data dependency relationship between them, where the task t_iIs task t_jIs directly preceding (parent) node, and task t_jThen it is task t_iIs directly succeeding (child) node. In the workflow scheduling process, a task must be executed after all the predecessor nodes of the task are executed. In a given directed acyclic graph representing a workflow, a task without a predecessor node is referred to as an 'in task', and similarly, a task without a successor node is referred to as an 'out task'. Each workflow w has a corresponding expiration date d (w), and a scheduling method is said to be a feasible solution if it can be executed before the corresponding expiration date.

In a multi-cloud environment, a plurality of cloud service providers P ═ { P, q.. multidata, r }, exist, and the service providers P provide a plurality of virtual machine instance types S_p＝{s_p1,s_p2,...,s_pk}. Each type of virtual machine instance has its specific computing and storage capabilities, and the present invention assumes that the virtual machine has sufficient storage space to store the transfer data during the execution of the subtasks, and therefore the present document focuses primarily on the virtual machine computing capabilities (i.e., the number of CPUs). Subtask t_iIn virtual machine vm_pijThe estimated execution time of (c) is Exe _ T (T)_i,vm_pij) The performance cost of a given task on different types of virtual machines is different.

When the virtual machine is started for the first time, a certain initial starting time T is needed_boot(vm_pij) To perform an initialization configuration. In the workflowIn the scheduling process, such virtual machine initialization time should be considered as important influence on the formation of the workflow scheduling scheme. Similarly, when all the subtasks on the virtual machine are executed, the corresponding virtual machine is not immediately closed, but the corresponding virtual machine waits until all the subtasks on the virtual machine completely communicate the output data of the corresponding subtasks to the virtual machine corresponding to the next generation task. In a cloud environment, a cloud service provider p sets a specific asking price unit time λ for a service provided by the cloud service provider p_pEach virtual machine type s_piAll have corresponding unit time prices c_pi。

The infrastructure of the same cloud service provider is typically concentrated in a smaller area, while the infrastructure of different cloud service providers are far apart, thus assuming that the intra-cloud bandwidth of a single cloud is faster than the inter-cloud bandwidth between different clouds. Within the cloud service provider p, data is driven from subtasks t_iTransfer to subtask t_jHas an intra-cloud communication time of T_intra(e_ijP) and the communication time of the data between the cloud service providers p and q is T_inter(e_ijP, q) as shown in the following formula.

Subtask t_iTransfer to subtask t_jThe Data size of (a) is Data (e)_ij)，B_intra(p) is the intra-cloud bandwidth of cloud p, and B_inter(p, q) is the inter-cloud bandwidth between cloud p and cloud q. Assuming infinite bandwidth on a single virtual machine, T is therefore assigned to the same virtual machine when two subtasks are executed_intra(e_ijP) has a value of 0.

The cost of data communication between different clouds will affect the final scheduling decision, c_p,qRepresenting the unit price required to communicate 1GB of data from cloud p to cloud q. The present invention does not discloseThe costs generated by services such as resource monitoring, data storage and load balancing are considered, because the low costs are negligible compared with the calculation cost or the data communication cost.

The purpose of the scheduler is to minimize workflow execution costs, including computation costs for the virtual machines and data communication costs between subtasks, while satisfying deadline constraints. The definition of the entire scheduling scheme is S ═ (Re, Map, T)_total,C_total) Where Re represents a set of virtual machine resources Re ═ { vm) that need to be enabled₁,vm₂,...,vm_r}，Map＝{(t_i,vm_j)|t_i∈V,vm_jE.g. Re represents the mapping relation of the subtask corresponding to the virtual machine resource Re in the workflow, T_totalRepresents the execution completion time of the workflow, and C_totalThe total workflow execution cost is indicated. Each virtual machine has a corresponding virtual machine type s_piAnd corresponding switch-on times Tls (vm)_i) And a closing time Tle (vm)_i). When the subtasks are scheduled to be completed, there is a corresponding set of actual execution start times AST (t)_i) And actual execution completion time AET (t)_i) And the outbound task no longer generates and communicates data. Thus, the workflow execution completion time T_totalAnd a corresponding total execution cost C_totalAs shown in the following equations, respectively.

The first half of equation (4) represents the execution cost of the virtual machine, and the second half represents the data communication cost. Lambda [ alpha ]_pThe cloud service provider p sets specific asking price unit time and subtask t for the service provided by the cloud service provider p_kIs a subtask t_jP (t) of_j) And p (t)_k) Respectively represent execution t_jAnd t_kThe service provider of (1). When t is_jAnd t_kWhen executed by the same cloud service provider, s_jkIs 0 (i.e. no inter-cloud data communication is generated), otherwise s_jkIs 1.

Based on the related definitions, the workflow scheduling problem with the deadline constraint in the multi-cloud environment can be formally expressed as a formula (5), and the core idea is to pursue the execution cost C_totalAt the lowest, make the execution time T_totalLess than or equal to workflow deadline D (w).

The PSO algorithm is an animal evolutionary computing technology based on social behavior of a bird flock, and is proposed by Eberhart and Kennedy in 1995. Particles are particularly important in PSO algorithms, where each particle represents a candidate solution to the optimization problem, and they can move throughout the problem space. Each particle moves with a speed that is influenced by the conditions of the particle itself, the optimal historical position of the particle itself, and the historical optimal position of the entire population. In order to determine the superiority and inferiority of the solution generated by each particle at different positions in the problem space, a fitness function is introduced to evaluate the solution quality of each particle. Each particle is determined by its own position and velocity, which are iteratively updated and adjusted in the problem search space based on surrounding particles and its own experience. Wherein the velocity is updated according to equation (6) and the position is updated according to equation (7).

Where, t represents the current number of iterations,

and

respectively representing the velocity and position of the ith particle at the t-th iteration, it is usually necessary to define a maximum velocity V_maxTo limit the particle velocity so that the search results are within the problem solution space.

And gBest^tThe self-history optimal position of the particle i and the history optimal position of the whole population are respectively obtained after t iterations. w is an inertia weight which determines the influence of the previous iteration speed on the current speed and is important for the convergence of the algorithm. c. C₁And c₂The cognition factors represent the cognition learning ability of the current particles to the self historical optimal value and the population global historical optimal value. r is₁And r₂Are two random variables ranging from 0 to 1 to enhance randomness during the iterative search process.

The method provided by the invention is mainly used in the scheduler of fig. 1. The method mainly comprises the parts of problem coding, resource pool initialization, fitness function, particle updating strategy, particle to scheduling result mapping, parameter setting and the like, and is specifically discussed through the following contents.

Problem coding

Improving the algorithm search efficiency and performance requires a good coding scheme. The evaluation criterion of the coding strategy mainly considers three basic principles of soundness, completeness and non-redundancy. The workflow scheduling problem is coded in a nesting mode of cloud providers, instance types and specific instances. One particle represents one scheduling scheme of workflow in multi-cloud environment, and the position of the particle i at the time t

As shown in equation (8).

Wherein

Indicates the distribution position of the kth subtask at time t, as shown in equation (9). (p, s)_pj,vm_pjr) The type of the instance representing that the subtask is allocated to the cloud p is s_pjOn the r-th concrete example. Each node bit on the particle is nested and divided into 3 decimal bits, which respectively represent the cloud service provider, the instance type and the specific instance, so that the size of the coding space is 3 times of the number of the subtasks. When initializing the seed group, the node decimal places of the particles are respectively initialized randomly to an integer value between 0 and the corresponding maximum value. Fig. 2 is a particle encoding diagram showing a scheduling of a particle encoding policy including 8 subtask workflows, assuming that a multi-cloud environment includes 3 cloud service providers, and each cloud service provider provides 8 instance types. Thus, the p coordinate value is from 0 to 2, s_pjThe coordinate values are from 0 to 7. As can be seen from FIG. 2, the subtask t₁Type s assigned to cloud 0₀₀Vm of a virtual machine₀₀₀。

Initializing a resource pool

And due to the flexible supply mode of the resources in the cloud environment, the initial resource set cannot be obtained by the algorithm. For the PSO algorithm, the size of the initialization resource pool will determine the scope of the search space, playing a key role in algorithm complexity and workflow execution performance. When the initialized resource pool is too small, a workflow may occur that could have been completed by the expiration date, and cannot be completed in time due to lack of resources. When the initialization resource pool is too large, the potential solution of the PSO code is too large, so that the algorithm cannot be converged in time. A simple and feasible initialization resource allocation scheme is that one virtual machine of all categories in a multi-cloud environment is allocated to each subtask, so that diversity and integrity of a search space can be guaranteed. However, the initialized resource pool R for this scheme_intialIs n × Num_type(vm)The search space is large, and the algorithm complexity is increased.

Where n is the number of subtasks in the workflow w, Num_type(vm)The sum of the number of instance types for all cloud service providers is defined as formula (10). Num_vm(p)Number of instance types provided for cloud service provider p.

To further compress the search space while preserving the diversity of the original potential solutions, the present invention designs an initialization resource pool R_intialIs of size | S_par(w)|*Num_type(vm)In which S is_par(w)Is the largest set of parallelizable subtasks in the workflow w. Due to addition of S_par(w)The other subtasks are all equal to S_par(w)The subtasks in the method have direct and indirect dependency relations, so the initialization resource strategy can ensure that each subtask has the opportunity of selecting one type of instance, thereby ensuring the diversity of potential solutions and reducing the search space.

Fitness function

The fitness function of a particle is used to evaluate the superiority or inferiority of two compared particles, with a smaller fitness function value generally corresponding to a better particle. Since the previous particle coding strategy does not satisfy the integrity principle, that is, the execution time of the workflow exceeds the corresponding deadline, it is necessary to define the fitness function of the feasible solution and the infeasible solution exceeding the deadline. The fitness function for judging the quality of two particles is defined in three different conditions.

Case 1 one particle is a feasible solution and the other particle is an infeasible solution. The feasible solution is chosen without any controversy, and its fitness function is defined as shown in equation (11).

Case 2 both particles are feasible solutions. Selecting the particles with lower execution cost, wherein the fitness function is defined as follows:

case 3 both particles are not feasible solutions. Particles with smaller execution times are selected because the particles are more likely to become viable solutions after evolution. The fitness function is defined as shown in formula (13).

Particle update strategy

As shown in equation (6), the PSO includes three core components: an inertial component, an individual cognitive component, and a social cognitive component. In order to overcome the defect of premature convergence of the traditional PSO algorithm, the ADPSOGA algorithm introduces mutation and cross operation of a genetic algorithm and updates corresponding parts in the formula (6). The particle i is updated at time t in the manner shown in equation (14), where M_u() Denotes mutation operation, C_g() And C_p() Indicating a crossover operation.

The inertia part in the formula (6) is combined with the variation operation idea in the genetic algorithm, and the updating mode of the inertia part is shown as a formula (15), wherein r₁Is a random number between 0 and 1. M_u() Randomly selecting a fraction of the particles, irregularly changing the fraction value, and the new values must all be within the corresponding threshold values. Fig. 3 shows the variation operation on the encoded particle of fig. 2, a quantile mp1 is randomly selected, and the value at the position mp1 is updated from (0,1,2) to (1,2,0), and the variation meets the scheduling criteria.

The individual cognition and social cognition parts in the formula (6) are combined with the genetic algorithm cross-operation idea, and the updating results are respectively shown in formulas (16) and (17). r is₂And r₃Is a random number between 0 and 1, C_p() (or C)_g() Two bins of the particle are randomly selected, and the values between the bins are interleaved with the values between the corresponding pBest (or gBest) bins. Fig. 4 shows the crossover operation of the cognitive part of the individual (or society), randomly generating two crossover positions (i.e. cp1 and cp2), replacing the value between the positions of the particles cp1 and cp2 with the value of pBest (or gBest) in this interval.

Mapping of particles to scheduling results

Pseudo code is designed that maps from the encoded particles to the workflow scheduling process. The algorithm inputs include workflow w, initialization resource pool R_intialAnd a coded particle X. First, for scheduling S ═ Re, Map, T_total,C_total) Initializing four elements. After initialization, the estimated execution time matrix Exe _ T [ | w | × | R for each subtask corresponding to different types of instances is calculated_intial|]Element Exe _ T [ i ] in the matrix][j]Representing a subtask t_iIn virtual machine mv_jTo the estimated execution time. Computing estimated communication time, T, between single and multiple clouds of data volume between subtasks_intra[i][j]Representing a single cloud sub-task t_iThe amount of data generated is communicated to the subtask t_jEstimated time required, T_inter[i][j][p][q]Representing a subtask t_iSubtask t for communicating generated data volume from cloud p to cloud q_jThe estimated time required.

Through the above operations, all information to obtain a candidate solution from the encoded particle has been obtained. Progressively scanning each position of particle X to generate pairsSet of Re and Map for response. Based on 'problem coding', the coding bits of the particles correspond to subtasks, and the value of the bit corresponds to instance resources, thereby determining the subtask t_iIs assigned to instance r_X(i)The above. Requiring a computation subtask t_iIs estimated starting time STt_iHere, two cases are distinguished:

a) subtask t_iIs a true incoming task, i.e. it has no direct predecessor subtasks. When the virtual machine r_X(i)When available, the subtask t_iStarting execution immediately, it estimates the start time STt_iFor a virtual machine r_X(i)Rented time LETr_X(i). In addition, the virtual machine r needs to be judged_X(i)Whether the virtual machine is started or not, if not, the virtual machine needs to be started, and the rented time LETr of the virtual machine_X(i)I.e. the initialization time T of the virtual machine_boot(r_X(i))。

b) Subtask t_iIt is not an inbound task, i.e., it has one or more parent tasks. Subtask t_iNot only can be executed when the resources are free, but also all the father tasks of the virtual machine r can be executed, and the generated data is communicated to the virtual machine r_X(i)Can be executed. Dispatching operator task t_iPseudo code of latency and data communication cost while considering virtual machine r_X(i)Whether it has already been activated.

Has calculated the subtask t_iIs estimated starting time STt_iThe subtask t needs to be calculated according to its estimated execution time and data communication time on the virtual machine_iIs estimated to be the end time ETt_i. For the calculation of data communication time, the calculation is needed according to the descendant subtasks t_cWhether or not to cooperate with subtask t_iThe allocation is determined in the same cloud, and there are three cases:

a)t_cand t_iIf the virtual machine executes on the same virtual machine, the communication time transfer is 0.

b)t_cAnd t_iIf the virtual machine is executed in the same cloud but on different virtual machines, the communication time transfer is T_intra[i][c]。

c)t_cAnd t_iRespectively executing on different clouds (such as cloud p and cloud q), the communication time transfer is T_inter[i_p][c_q]。

Subtask t_iDispatch to virtual machine r_X(i)Its start time STt_iAnd an end time ETt_iAnd adding the mapping relation of the equal four elements into the Map set. Subsequently judging the virtual machine r_X(i)Whether it has already been added to the rental resource Re, and if not, the addition is made. Virtual machine r_X(i)Is equal to the subtask t_iThe estimated completion time of (2). And finally, respectively calculating the total execution time and the total execution cost of the workflow according to the formula (3) and the formula (4). And outputting the scheduling scheme S corresponding to the coded particles.

Designing a calculation subtask t_iLatency and communication cost. First, the waiting time T is initialized_waitAnd a data communication cost C_tranfer. Subtask t_iThe latency is equal to the maximum data communication time among all its parent tasks. Current subtask t_iAnd its parent task t_pThe data communication costs are taken into account when assigning to different clouds.

Parameter setting

The inertial weight factor w of equation (6) can determine the convergence and search capability of the PSO algorithm. When w is small, the algorithm has strong local search capability; otherwise, the algorithm has stronger global search capability. In the initial stage of algorithm execution, the diversity of problem space search and the particle global search capability are emphasized more, and as the search is deep, the local search capability is emphasized more in the later stage. Therefore, the weight of the inertia weight factor w should gradually decrease as the number of iterations of the algorithm increases. Equation (18) is a classical inertial weight factor adjustment strategy. Wherein, w_maxAnd w_minRespectively, a maximum value and a minimum value set at the time of w initialization, iters_curAnd iters_maxRespectively representing the iteration number of the current algorithm and the maximum iteration number set by initialization.

In the above classical inertia weight factor adjustment strategy, the change of w is only related to the number of iterations, and the nonlinear, complex and variable characteristics of the practical problem cannot be well satisfied. The weight value of the inertia weight factor w should be continuously evolved along with the evolution of the population particles, so that an inertia weight factor adjusting strategy which is adaptively adjusted according to the quality of the current population particles is constructed. The strategy adjusts the inertial weight factor size based on the degree of difference between the current particle and the global historical optimal particle, as shown in equation (19). Wherein div (X)^t-1,gBest^t-1) Represents a particle X^t-1And global history optimal particle gBest^t-1The bit number of different quantiles, and T is the number of subtasks in the workflow.

When div (X)^t-1) When the value is smaller, it represents particle X^t-1And gBest^t-1The difference degree is small, so the weight of w should be reduced to ensure that the particles can be better searched in a small range to find an optimal solution; otherwise, the weight of w should be increased to make the search space of the particles larger, so as to find the optimized solution space more quickly. Therefore, the weight calculation formula of the inertia weight factor w is updated as follows:

in addition, two cognitive factors c of the algorithm₁And c₂And setting by adopting a linear increase and decrease strategy. The updating method is shown as formula (21) and formula (22), wherein c₁A start and c₂Respectively denotes parameter c₁And c₂Initial value of iteration, c₁End and c₂_ end represents the parameter c, respectively₁And c₂The final value of the iteration.

Fig. 5 is an algorithm flowchart of the scheduling method of the present invention, which specifically comprises the following steps:

step 1: the relevant parameters in the initial scheduling method are as follows: population size 100, maximum iteration number 1000, inertial weight factor, cognition factor and other parameters c₁_start＝0.9，c₁_end＝0.2，c₂_start＝0.4，c₂End is 0.9, generating an initial population.

Step 2: calculating the fitness values of the particles under different conditions according to the particle mapping strategy and the formulas (11), (12) and (13), selecting the particle with the minimum fitness value as a population global optimal particle, and setting each particle in the first generation as a history optimal particle.

And step 3: the particles are updated according to the particle update formulas (14) to (17).

And 4, step 4: and recalculating the fitness value of each particle, and if the fitness value of the current particle is smaller than the historical optimal value of the current particle, updating the new particle into the historical optimal particle of the current particle.

And 5: and if the fitness value of the current particle is smaller than that of the population global optimal particle, updating the particle into the population global optimal particle.

Step 6: checking whether an algorithm termination condition is met, and if so, terminating the algorithm; otherwise, go to step 3.

The above are preferred embodiments of the present invention, and all changes made according to the technical scheme of the present invention that produce functional effects do not exceed the scope of the technical scheme of the present invention belong to the protection scope of the present invention.

Claims (1)

1. A workflow scheduling method considering communication and computing cost under a multi-cloud environment is characterized in that: based on the structure and execution characteristics of the workflow and relevant factors of current cloud resource environment communication and execution cost, and based on the idea of random two-point cross operation and random single-point variation operation of a genetic algorithm, the diversity in the population evolution process is improved, virtualized resources are integrated, data communication cost and task calculation cost are considered, the resource utilization rate is optimized, and the execution cost is reduced on the premise of meeting the workflow deadline; the workflow scheduling method is specifically implemented as follows,

defining the workflow scheduling method as S ═ Re, Map, T_total,C_total) Where Re represents a set of virtual machine resources Re ═ { vm) that need to be enabled₁,vm₂,...,vm_r}，Map＝{(t_i,vm_j)|t_i∈V,vm_jE.g. Re represents the mapping relation of the subtask corresponding to the virtual machine resource Re in the workflow, T_totalRepresents the execution completion time of the workflow, and C_totalThen the total workflow execution cost is represented; the workflow is represented by a directed acyclic graph G (V, E), where V represents a set of vertices { t } containing n tasks₁,t₂,...,t_nE represents the data dependency relationship between tasks E₁₂,e₁₃,...,e_ij}; each data dependent edge e_ij＝(t_i,t_j) Representing a subtask t_iAnd subtask t_jThere is a data dependency relationship between, where the subtask t_iIs a subtask t_jOf the direct predecessor node, and sub-task t_jThen it is the subtask t_iThe direct successor node of (1);

each virtual machine has a corresponding virtual machine type s_piAnd corresponding switch-on times Tls (vm)_i) And a closing time Tle (vm)_i) (ii) a When the subtasks are scheduled to be completed, there is a corresponding set of actual execution start times AST (t)_i) And actual execution completion time AET (t)_i) And new communication data cannot be generated after the task is taken out; thus, the workflow execution completion time T_totalAnd a corresponding total execution cost C_totalAs shown in the following equations, respectively:

the first half of the formula (2) represents the execution cost of the virtual machine, and the second half represents the data communication cost; lambda [ alpha ]_pThe cloud service provider p sets specific asking price unit time and subtask t for the service provided by the cloud service provider p_kIs a subtask t_jP (t) of_j) And p (t)_k) Respectively represent execution t_jAnd t_kThe service provider of (2); when t is_jAnd t_kWhen executed by the same cloud service provider, s_jkIs 0, i.e. no inter-cloud data communication is generated, otherwise s_jkIs 1;

based on the related definitions, the workflow scheduling problem with the deadline constraint in the multi-cloud environment can be formally expressed as a formula (3), and the core idea is to pursue the execution cost C_totalAt the lowest, make the execution time T_totalLess than or equal to workflow deadline D (W);

then, the following algorithm is executed to achieve the execution cost C_totalAt the lowest, make the execution time T_totalWorkflow deadline D (W):

s1: the initialization related parameters are as follows: population size 100, maximum iteration number 1000, inertia weight factor w and cognition factor parameter c₁_start＝0.9，c₁_end＝0.2，c₂_start＝0.4，c₂0.9, generating an initial population;

s2: calculating the fitness values of the particles under different conditions according to a particle mapping strategy and fitness functions of the particles, namely formulas (4), (5) and (6), selecting the particle with the minimum fitness value as a population global optimal particle, and setting each particle in the first generation as a self historical optimal particle;

wherein, formula (4) represents the fitness function of the particle when one particle is a feasible solution and the other particle is an infeasible solution, formula (5) represents the fitness function of the particle when both particles are feasible solutions, and formula (6) represents the fitness function of the particle when both particles are infeasible solutions;

s3: updating the particles according to the particle updating formulas (7) to (10);

wherein, formula (7) represents the update mode of the particle i at time t, M_u() To representMutation operation, C_g() And C_p() It is indicated that the cross-over operation,

and gBest^t-1Respectively obtaining the self historical optimal position of the particle i and the historical optimal position of the whole population after t-1 iterations; equation (8) represents the update mode of the inertial part, r₁Is a random number between 0 and 1; equations (9) and (10) represent the update methods of the individual recognition and social recognition parts, respectively, and r₂And r₃Is a random number between 0 and 1;

s4: recalculating the fitness value of each particle, and if the fitness value of the current particle is smaller than the historical optimal value of the current particle, updating the new particle into the historical optimal particle of the current particle;

s5: if the fitness value of the current particle is smaller than that of the population global optimal particle, updating the particle into the population global optimal particle;

s6: checking whether an algorithm termination condition is met, and if so, terminating the algorithm; otherwise, go to S3.

Images