CN107341041B

CN107341041B - Cloud task multidimensional constraint backfill scheduling method based on priority queue

Info

Publication number: CN107341041B
Application number: CN201710500564.5A
Authority: CN
Inventors: 程维文; 袁佳欣; 陈建新; 李连万; 闫午阳
Original assignee: Nanjing University of Posts and Telecommunications
Current assignee: Nanjing University of Posts and Telecommunications
Priority date: 2017-06-27
Filing date: 2017-06-27
Publication date: 2020-08-11
Anticipated expiration: 2037-06-27
Also published as: CN107341041A

Abstract

The invention discloses a cloud task multidimensional constraint backfill scheduling method based on a priority queue, which overcomes the defects of poor backfill performance and large duty ratio of virtual machine resources caused by the fact that task indexes are considered to be too single in the traditional backfill scheduling algorithm and the improved backfill scheduling algorithm in cloud computing; the two dimensions of the number of the application processor cores and the expected execution duration of the independent tasks in the batch tasks are comprehensively considered for balancing, a priority queue based on the minimum binary heap is constructed by utilizing the backfill weight coefficient to preferentially backfill the tasks with less application resources and longer expected execution time, and the constraint conditions such as memory occupation and bandwidth occupation are increased. The invention improves the utilization rate of virtual computer computing resources in unit time, reduces the final completion time of batch tasks and reduces the average waiting time delay of the tasks.

Description

Cloud task multidimensional constraint backfill scheduling method based on priority queue

Technical Field

The invention relates to the technical field of cloud computing, in particular to a resource management technology applied to a cloud computing platform, and particularly relates to a cloud task multidimensional constraint backfill scheduling method based on a priority queue.

Background

Nowadays, in the era of information explosion, the development of computer technology is changing day by day. When the amount of users is increasing, the conventional data center has been unable to meet the existing demand. The cloud computing technology is widely applied, and the birth of the cloud computing technology is a product of the information technology revolution. Cloud computing is generally oriented to heterogeneous components, and in the face of massive task requests, how to schedule processing tasks in the shortest time in limited computing resources by a cloud computing data center is important in the aspects of shortening the final completion time of the tasks, reducing the average waiting time delay of batch tasks and increasing the utilization rate of server resources.

Some existing cloud computing-based task scheduling methods are also various, and some are based on heuristic algorithms such as an ant colony algorithm, an annealing algorithm, a genetic algorithm and the like, for example, CN103345657A discloses a task scheduling method based on heredity and ant colony in a cloud computing environment, and iterative solution is performed by combining the advantages of the genetic algorithm and the ant colony algorithm. For another example, CN103870317A discloses a task scheduling method and system in cloud computing, in which a bacterial foraging algorithm is used to calculate an optimal working node, and tasks and resources are scheduled and allocated. The heuristic model with high abstraction is exposed to slow iteration in the actual high-timeliness and quick-response cloud computing task scheduling process, and is easy to fall into a local optimal solution in the face of a large number of different tasks, so that the system response is slowed, and the user experience is reduced.

Some scheduling methods simply consider the priority weight of the task, and do not consider the relationship between the task and the computing resources such as the virtual machine in many aspects. For example, CN103324525A discloses a scheduling method in a cloud computing environment, which performs scheduling by calculating a priority index and a waiting index of each task. CN103914754A discloses a task scheduling method for workflow, which combines the average predicted time of each task running on all processors and the average time of data transfer as a weight value to be sorted, and then traverses the compute nodes to bind the maximum weight value to the virtual machine with the best performance. Therefore, the bearing of the computing resources of each machine and the resource requirements of independent tasks are not considered, the tasks are easily bound in the virtual machines to generate larger virtual machine idleness, and the resource utilization rate of the virtual machines cannot be increased to the maximum extent.

Still other scheduling methods simply consider task scheduling from performance triggers of the compute nodes. For example, CN101986272A discloses a task scheduling method in a cloud computing environment. The method does not consider the performance of the nodes in task allocation, and only considers the health degree of the nodes. After the distributed tasks exceed the set threshold value and are not processed, the tasks are recovered and distributed again. The resulting default rate of the task is high, and the average delay of the task is increased. In contrast, CN105740077A discloses a task allocation method using cloud computing, which is to establish loads of respective computing nodes in a data center, and allocate tasks to nodes with loads from high to low in sequence, so that the load of the nodes is too large, and the overall scheduling performance is impaired.

For example, CN103841208A discloses a cloud computing task scheduling method based on response time optimization, which calculates an objective function according to the transmission time of task fragments, the average processing time of tasks at each node, and the total processing time of task fragments, and models the task response time as a target. However, this makes it easy to prioritize tasks with a smaller execution time, and makes the response time of tasks with a longer execution time longer. The scheduling relationship between tasks of different lengths cannot be balanced.

In addition, some traditional backfill algorithms and the existing improved backfill scheduling-based method only consider the single index of the number of processor cores waiting for the task application when selecting the backfill task, but do not combine the applied resources with the index of the estimated execution duration of the independent task. In addition, an improved method based on backfill scheduling, which is efficient and comprehensively considers multidimensional performance index constraints, is not provided in the existing task scheduling methods under the cloud computing environment.

Disclosure of Invention

The invention provides a cloud task multidimensional constraint backfill scheduling method based on a priority queue, aiming at the defects and shortcomings of the existing improved backfill scheduling method, the method comprehensively considers two dimensions of task submission processor resources and the predicted execution time of an independent task to calculate a backfill weight coefficient, then constructs the backfill weight coefficient of each task submitted to a virtual machine into a task priority queue by using a data structure of a minimum binary heap, carries out backfill scheduling processing on batch waiting tasks, and simultaneously adds constraints of resources such as memory, bandwidth and the like during backfill so as to improve backfill efficiency and backfill success rate. Therefore, the aim of reserving and backfilling the idle resources of the virtual machine in time to the maximum extent is achieved, the final completion time of the task is shortened, and QoS indexes such as the average waiting time delay of the task are reduced.

The technical scheme adopted by the invention is as follows: a cloud task multidimensional constraint backfill scheduling method based on a priority queue comprises the following steps:

step 1: the tasks in the waiting task queue are arranged in a descending order by taking the ratio of the task instruction length to the number of the applied processor cores as a task processing time factor and are stored in a common queue; the processing time factor is used for representing the predicted processing time of the independent task;

step 2: arranging virtual machines in the data center in a descending order according to the magnitude of the computing processing capacity of the virtual machines, and establishing a corresponding relation between the serial number of the virtual machine and the computing processing capacity value of the virtual machine in the data center;

and step 3: binding the task in the common queue in the step 1 to the virtual machine queue in the step 2 according to the estimated execution time, wherein the longer the execution time is, the better the performance of the bound virtual machine is, if the estimated execution time of the task distributed to the current virtual machine is less than or equal to the next virtual machine, the task is bound to the current virtual machine, otherwise, the task is bound to the current virtual machine, and the primary binding of the virtual machine is completed;

and 4, step 4: step 3, batch tasks on each virtual machine are entered, and a backfill weight coefficient obtained by dividing the square of the number of processor cores applied by each task by the ratio of the task instruction length is used as a key to establish a minimum binary heap to form a task priority queue; the backfill weight coefficient is characterized by the ratio of the square of the number of processor cores applied by each task divided by the length of the task instruction;

and 5: popping up a first task in the minimum binary heap, scanning a running task in the virtual machine, and backfilling if constraint conditions of the total processor core number, the total memory size and the total bandwidth size of the virtual machine are met at the same time, namely the task does not exceed the processing capacity, the memory and the bandwidth of the virtual machine after backfilling; if not, the tail of the binary heap queue is pressed again to float again;

step 6: and repeatedly executing the step 5 until all tasks in the binary heap are scheduled.

The computing processing capacity of the virtual machine is characterized by MIPS (Million Instructions Per Second, average execution speed of single-length fixed-point Instructions).

The invention has the beneficial effects that: the initial binding is finished by preferentially distributing the uncommitted tasks to the fastest virtual machine from large to small according to the predicted execution time, so that the maximum completion time of the tasks can be reduced; and establishing a task priority queue of a minimum binary heap for the tasks entering the virtual machine according to BFV (Backfill Values, BackFill weight coefficients), and adopting the data structure to quickly push the tasks with the minimum BFV value and the most urgent need to be refilled to the root node. The task with the smaller BFV is backfilled preferentially, so that the vacancy of the virtual machine can be fully reduced, the resource utilization rate is improved, and meanwhile, the average waiting time delay of the task can be reduced.

Drawings

FIG. 1 is a cloud task multidimensional constraint backfill scheduling architecture diagram based on a priority queue;

FIG. 2 is a flowchart of a cloud task multidimensional constraint backfill scheduling method based on a priority queue;

FIG. 3 is a diagram of a minimum binary heap structure of a task queue after a virtual machine is selected according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Fig. 1 is a diagram of a cloud task backfill scheduling architecture according to the present invention, where the whole architecture is composed of a waiting task queue and a data center, and the data center is composed of a data center agent (datacentrerbaker), a host and a virtual machine built on the host. Wherein the role of the data center agent is the cloud task scheduler.

Firstly, initializing a data center, and firstly starting a plurality of hosts by the data center; then, the task is submitted, and the data center can establish a corresponding number of virtual machines on the host; at the moment, tasks are already run on the virtual machine, and a waiting task queue is arranged outside the data center and is stored by using a common queue.

FIG. 2 is a detailed flow of the whole cloud task multidimensional constraint backfill scheduling method:

step 1: firstly, according to T in formula 1, aiming at task waiting queue which does not enter data center_ratioAnd the tasks are sorted in descending order, so that the tasks with longer expected execution time are preferentially sorted in front of the queue.

Step 2: sequentially establishing an index table for the virtual machines on all the hosts in the data center, wherein key is the id of each virtual machine; value is MIPS, i.e. the processing power of the virtual machine processor core. And then the MIPS values of the virtual machines are arranged according to a descending order, so that the virtual machine with the best performance is arranged at the forefront of the index table. The statistically good index table is then submitted to a data center broker (datacentrerbower) and saved.

And step 3: and (3) binding the tasks in the common task queue in the step (1) to the virtual machines in the index table in the step (2) in sequence, calculating the accumulated execution time of the current virtual machine and the next virtual machine while binding, and selecting the virtual machine with the shortest accumulated execution time to bind.

And 4, step 4: and 3, establishing a minimum binary heap to form a priority queue according to the value of BFV by the batch tasks submitted to each virtual machine in the step 3.

After the selected virtual machines enter the data center, heap construction operation is performed on batch tasks on each virtual machine, a minimum binary heap data structure shown in fig. 3 is used, and the minimum binary heap is a complete binary tree with ordered heap. The nodes T1-T9 in FIG. 3 store BFV values of tasks, respectively. Wherein T1 is less than T2/T3, T2 is less than T4/T5, T3 is less than T6/T7, and T4 is less than T8. T1 is the root node with the minimum BFV value in the whole queue, wherein the index value of each left child node is 2 times that of the parent node, and the index value of the right child node is 2 times +1 times that of the parent node. Sequentially constructing a priority queue composed of minimum binary heaps

And 5: and scanning idle resources in the virtual machine, sequentially popping up a first task in the binary heap, scanning a running task in the virtual machine, and backfilling the task.

Step 5-1: because the root node in the minimum binary heap is the task with the minimum BFV value, if idle computing resources exist in the current virtual machine, the first task of the priority queue is popped up, and then whether the task meets the resource constraint condition or not is judged according to a formula 3 if the task is backfilled. And if so, sequentially backfilling. Wherein, PE is the number of processor cores, and RAM and BW are the memory occupation and bandwidth usage of the task respectively.

Step 5-2: if the popped task is found not to meet the resource constraint condition in step 5-1, the task is reinserted into the tail of the queue of the binary heap, then the child nodes below the root node are popped up and tried to be refilled, and after the refilling is successful, the task is floated to a proper position according to the definition requirement of the binary heap because the task inserted from the tail of the queue is possibly smaller than the parent node of the task.

Step 5-3: and according to the two substeps, backfilling and scheduling the submitted tasks of all the virtual machines in sequence, and finally, after the tasks are executed, destroying the virtual machines and finishing the whole task scheduling process.

The technical means disclosed in the invention scheme are not limited to the technical means disclosed in the above embodiments, but also include the technical scheme formed by any combination of the above technical features.

Claims

1. A cloud task multidimensional constraint backfill scheduling method based on a priority queue is characterized by comprising the following steps,

and step 3: binding the tasks in the common task queue in the step 1 to the virtual machines in the index table in the step 2 in sequence, calculating the accumulated execution time of the current virtual machine and the next virtual machine while binding, and selecting the virtual machine with the shortest accumulated execution time to bind;

2. The method for multidimensional constraint backfill scheduling of cloud tasks based on priority queues according to claim 1, wherein the computing processing capacity of the virtual machine in the step 2 is characterized by MIPS.