CN104243617A - Task scheduling method and system facing mixed load in heterogeneous cluster - Google Patents

Abstract

The present invention relates to a mixed load-oriented task scheduling method and system in a heterogeneous cluster, comprising the following steps: a resource scheduler receives machine heartbeats and maintains machine attribute clusters; a job manager receives and parses jobs to obtain several tasks; The manager sets attribute clusters and constraint requirements for the task, and sends the task information to the resource manager; the resource scheduler matches the task with an optimal machine that meets the constraints, and returns the matching relationship between the task and the machine to the job manager; The manager sends the task to the executor on the matching machine to execute the task. The present invention expresses heterogeneous machine attributes and task requirements through an easily expandable constraint description method. On this basis, hard constraints are used as filtering criteria and soft constraints are used as selection criteria to allocate optimal machines for tasks, which is significantly The execution efficiency of tasks and the overall performance of the system are improved.

Description

A mixed load-oriented task scheduling method and system in a heterogeneous cluster

technical field

The invention relates to a mixed load-oriented task scheduling method and system in a heterogeneous cluster, belonging to the field of computer parallel computing.

Background technique

In recent years, cluster machines have shown increasingly significant heterogeneity. Modern clusters tend to be large, run for long periods of time, and may even be geographically distributed. Throughout its lifecycle, clusters often need to update machines. In addition, in the cluster integration scenario, the cluster administrator may integrate several small clusters of different batches into one large cluster. Given the above, it is likely that there are some differences in the hardware and software of the cluster.

On the other hand, with the continuous development of cloud computing, it has become a trend to run mixed loads on the same cluster, which has many benefits, such as improving resource utilization, sharing data, and reducing operation and maintenance costs. Specifically, many different types of tasks may be running on the cluster, including scientific computing, large-scale data analysis, long-running Internet services, and software development and testing.

Task scheduling refers to assigning resources to tasks, that is, placing tasks on machines. It can be said that tasks and machines are two important roles in task scheduling. In traditional application scenarios, tasks and machines are single and isomorphic, so scheduling only needs to consider basic resource requirements, such as slot-based resource allocation (a review of cloud data center virtual resource management research [J]. Computer Application Research, 2012 , 29(7):2411-2415.). However, with the continuous heterogeneity of clusters and loads, machine attributes and task requirements have changed greatly. Not all machines can meet the task constraints. Task scheduling must consider various constraints: for example, image processing tasks must run on On a machine with a GPU, some tasks can only run on a machine with a specific kernel version, and data analysis tasks should be run first on machines that store related data, etc. At present, task scheduling considering constraints is an important challenge in this field. How to describe a wide variety of constraints in an expandable way, and how to schedule tasks to the optimal machine according to the constraints, has become a task in heterogeneous environments. key issues of scheduling.

Some research results and open source software began to focus on task scheduling considering constraints. Hadoop YARN will consider the constraint of data locality when scheduling tasks, that is, it will preferentially schedule tasks to the machines that store the corresponding data (see Vavilapalli V K, Murthy A C, Douglas C, et al. Apache hadoop yarn: Yet another resource negotiator[C]//Proceedings of the 4th annual Symposium on Cloud Computing.ACM,2013:5.). The streaming computing framework Storm will prioritize frequent communication tasks to the same or similar machines (see Aniello L, Baldoni R, Querzoni L.Adaptive online scheduling in storm[C]//Proceedings of the 7th ACM international conference on Distributed event -based systems. ACM, 2013:207-218.). In the DAG computing scenario, Spark will try to schedule the subsequent tasks to the machine where the output data of the previous tasks are located (see The Apache Software Foundation.Spark Lightning-fast clustercomputing[EB/OL].(2012-1-10)[ 2014-9-5].https://spark.apache.org/.). In scenarios where data locality and fairness conflict, delay scheduling can be used to make a trade-off, which has achieved better results (see Zaharia M, Borthakur D, Sen Sarma J, et al. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling[C]//Proceedings of the 5th European conference on Computer systems. ACM,2010:265-278.). The above research considers some specific special cases of constraints, but the types of constraints are diverse, and cluster schedulers in actual work usually need to schedule different types of tasks and deal with various constraints. The above studies did not fully consider the heterogeneous characteristics of tasks and machines, and did not propose a scalable constraint scheduling mechanism.

The current general task scheduling method still assumes that tasks and machines are isomorphic. In the scheduling process, constraints are not considered, and only basic resource matching is considered. However, due to the heterogeneity of tasks and machines, task scheduling must consider various constraints. Existing methods cannot describe a wide variety of constraints, nor can they schedule tasks to the optimal machine according to the constraints, which will lead to abnormal execution of tasks or significantly longer running time, which seriously affects the execution efficiency of tasks and the performance of task scheduling systems. overall performance.

Contents of the invention

The technology of the present invention solves the problem: overcomes the deficiencies of the prior art, and provides a mixed-load-oriented task scheduling method and system in a heterogeneous cluster. Considering the task scheduling of constraints, tasks can be scheduled on the optimal machine according to the constraints, improving the Task execution efficiency and overall system performance.

Technical solution of the present invention: a task scheduling method for mixed loads in heterogeneous clusters, comprising the following steps:

Step 1, the resource scheduler receives the machine heartbeat and maintains the attribute cluster of the machine; the machine heartbeat is regularly sent to the resource scheduler by the executor, and the content of the heartbeat is the attribute cluster of the machine;

Step 2, the job manager receives and parses the job, and obtains several tasks;

Step 3, the job manager sets attribute clusters and constraint requirements for the task, and then sends the task information to the resource scheduler;

Step 4. After receiving the task information, the resource scheduler matches the task with the optimal machine that satisfies the constraints, and returns the matching relationship between the task and the machine to the job manager;

Step 5: After receiving the matching relationship between the task and the machine, the job manager sends the task to the executor on the matching machine to execute the task.

Further, the attribute cluster of the machine mentioned in step 1 includes a plurality of key-value (Key-Value) pairs, wherein the key represents the machine attribute, and the value represents the specific value of the attribute, wherein the attribute includes the machine host name, IP address, Machine type, machine architecture, operating system, total CPU, total memory, available CPU, available memory, constraint valuation, etc.

Further, the constraint requirements mentioned in step 3 include hard constraints and soft constraints. Hard constraints are necessary conditions for task execution and must be met during the scheduling process. The processing of hard constraints belongs to qualitative analysis. Soft constraints are preference conditions for task execution, which should be satisfied as much as possible to improve task execution efficiency, but if they cannot be satisfied, they can be ignored to avoid waste of resources and delays in task execution. The processing of soft constraints is a quantitative analysis.

Further, the step 3 specifically includes the following steps:

Step 3.1: Set an easy-to-extend attribute cluster for the task. The attribute cluster includes multiple key-value (Key-Value) pairs, where the key represents the attribute of the task, and the value represents the specific value of the attribute. The attribute cluster of the task includes task labeling, execution Commands, required CPU resources, required memory resources, etc.;

Step 3.2: Set hard constraint requirements for the task, and express the hard constraint requirements of the task through a Boolean expression. If there are multiple hard constraints, perform an "AND operation" on them, and still use a Boolean expression to represent multiple Hard constraint requirements;

Step 3.3: Set the soft constraint requirements for the task. Multiple soft constraint requirements of the task are represented by the soft constraint requirement linked list. The linked list contains several elements, and each element includes a Boolean expression and an estimate. The Boolean expression indicates the specific Soft constraint requirements, the valuation is used to quantify the improvement in execution efficiency brought about by meeting the soft constraint requirements;

Step 3.4: The job manager sends the task's attribute clusters and soft and hard constraint requirements to the resource scheduler, requesting machine allocation.

Further, the step 4 specifically includes the following steps:

Step 4.1: Record the received task as "task to be scheduled", initialize the machine set M, put all machines into M, and initialize the candidate machine list to be empty;

Step 4.2: Take a machine from the machine set M and record it as a "candidate machine". According to the information of the machine and the task, calculate the value of the hard constraint requirement of the task to be scheduled;

Step 4.3: Determine whether the hard constraint requirement of the task to be scheduled is true. If it is true, add the candidate machine to the list of candidate machines, and calculate the constraint of the candidate machine according to the machine information and the soft constraint list of the task Valuation;

Step 4.4: Remove the candidate machines from the machine set M, and judge whether the machine set M is empty, if not, go to step 4.2;

Step 4.5: Taking the constraint valuation as the standard, select the machine with the largest constraint valuation in the candidate machine list, and record it as the matching machine for the task to be scheduled.

Further, in the step 4.3, the calculation of the constraint estimate of the candidate machine further includes:

Initialize the constrained estimate of the candidate machine to 0;

Traversing the soft constraint linked list of tasks to be scheduled, for each element, calculate its soft constraint requirement, if the soft constraint requirement is true, add the current constraint estimate to the soft constraint element's estimate, and finally get the candidate machine Constrained valuation.

In order to solve the above technical problems, the present invention also proposes a mixed load-oriented task scheduling system in a heterogeneous cluster, including a job manager, a resource scheduler and an executor;

The job manager and resource scheduler are deployed on the master control node, and the job manager is used to manage jobs and tasks, set attribute clusters and soft and hard constraint requirements for tasks, and send task information to the resource scheduler, requesting tasks required machine;

The resource scheduler is used to receive the machine heartbeat sent regularly by the executor. On the basis of maintaining the heartbeat of the entire cluster machine, the resource scheduler can receive the task information sent by the job manager, and match the task with the optimal machine that meets the constraints;

The executor is deployed on all machines except the main control node, regularly reports the heartbeat of the machine to the resource scheduler, receives task instructions issued by the job manager, and is responsible for specific task execution.

The advantage of the present invention compared with prior art is:

(1) The task scheduling method and system proposed in the present invention represent heterogeneous machine attributes and task requirements through an easily expandable constraint description method. On this basis, hard constraints and soft constraints are treated differently, and hard constraints As a filtering criterion, the soft constraint is used as a selection criterion to match the task with an optimal machine that satisfies the hard constraint. The invention comprehensively considers various constraints in the task scheduling process, and significantly improves the task execution efficiency and the overall performance of the system.

(2) The task execution efficiency under the task scheduling that satisfies the constraints and the task scheduling strategy that ignores the constraints is tested, so as to verify the effectiveness of the task scheduling method that considers the constraints proposed by the present invention. Figure 11 records the task startup time in the virtual machine application scenario. The data shows that the task startup time that satisfies the constraints is significantly shorter than the task startup time that ignores the constraints. The specific speedup ratio is related to the size of the image. 24.18 varies. Figure 12 records the task completion time in the application scenario where tasks communicate with each other. The data shows that the task completion time that satisfies the constraints is also significantly shorter than the task that ignores the constraints. The specific speedup ratio is related to the data scale and network status. It is about 2.25 in this group of experiments. Generally speaking, the task scheduling method and system proposed by the present invention can handle various constraints and significantly improve task execution efficiency.

Description of drawings

FIG. 1 is a schematic diagram of the principle of a task scheduling method and system in an embodiment of the present invention;

Fig. 2 is the flow chart of task scheduling method in the embodiment of the present invention;

FIG. 3 is a schematic diagram of a machine attribute cluster in an embodiment of the present invention;

Fig. 4 is a flowchart of setting task attribute clusters and constraint requirements in an embodiment of the present invention;

5 is a schematic diagram of a task attribute cluster in an embodiment of the present invention;

FIG. 6 is a schematic diagram of task hard constraints in an embodiment of the present invention;

FIG. 7 is a schematic diagram of a linked list of task soft constraints in an embodiment of the present invention;

Fig. 8 is a flow chart of assigning an optimal machine for a task in an embodiment of the present invention;

FIG. 9 is a schematic diagram of hard constraints of computing tasks in an embodiment of the present invention;

Fig. 10 is a schematic diagram of computing machine constraint estimates in an embodiment of the present invention;

Fig. 11 is the task startup time in the virtual machine application scenario in the embodiment of the present invention;

FIG. 12 shows the task completion time in the application scenario of inter-task communication in the embodiment of the present invention.

Detailed ways

The principles and features of the present invention will be described below in conjunction with the accompanying drawings and embodiments, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

As shown in Figure 1, the embodiment of the present invention implements a mixed load-oriented task scheduling system running on a heterogeneous cluster, the system adopts a typical master-slave (Master-Slave) A core process Job Manager (Jobs Manager) and Resource Scheduler (Resource Scheduler), both deployed on the master physical node. The slave part (Slave) includes a core process executor (Executor), which is deployed on all other machines except the master physical node.

The job manager is responsible for managing jobs and tasks. A job includes several tasks, and a set of job IDs and task IDs can uniquely identify a task. The job manager receives and parses the job submitted by the user, sets attribute clusters and soft and hard constraints for the task according to the job parameters, and sends the task information to the resource scheduler to request the allocation of the required machines. After obtaining the matching machine, the job manager sends the task to the designated machine, monitors the execution status of the task and implements fault tolerance.

The resource scheduler is responsible for receiving the machine heartbeat (Heartbeat) information regularly sent by the executor, which contains the attribute clusters of the machine. On the basis of maintaining the attribute cluster of the entire cluster machine, the resource scheduler can receive the task information sent by the job manager, allocate the optimal machine that satisfies the constraints for the task, and return the matching relationship between the task and the machine as the result to the job scheduler device.

The executor is responsible for receiving the instructions from the job manager, starting the execution task, and reporting the task status to the job manager; on the other hand, the executor regularly reports the attribute cluster of the machine to the resource scheduler in the form of heartbeat. For each task, the executor first creates a virtualized environment, and then executes the task inside the virtualized environment.

As shown in Figure 2, in this embodiment, the task scheduling method may include the following steps:

Step 201, the resource scheduler receives the heartbeat of the machine, and maintains the attribute cluster of the machine;

Step 202, the job manager receives and parses the job to obtain several tasks;

Step 203, the job manager sets attribute clusters and constraint requirements for the task, and then sends the task information to the resource manager;

Step 204, after receiving the task information, the resource scheduler matches the task with an optimal machine that satisfies the constraints, and returns the matching relationship between the task and the machine to the job manager;

Step 205, after receiving the matching relationship between the task and the machine, the job manager sends the task to the executor on the matching machine to execute the task.

Fig. 3 is a schematic diagram of machine attribute clusters in an embodiment of the present invention. Figure 3 shows the attribute clusters and corresponding values of a certain machine. The attribute clusters of a machine are the specific content of the heartbeat of the machine, which is sent to the resource scheduler by the executor at regular intervals. The attribute cluster includes multiple key-value (Key-Value) pairs, where the key represents the machine attribute, and the value represents the specific value of the attribute, where the machine attribute includes the machine host name, IP address, machine type, machine architecture, operating system, and total number of CPUs , total memory, available CPU, available memory, constraint valuation, etc. Table 1 lists the attribute clusters of the machine. In the example shown in Figure 3, the host name of the machine is "Blade10", the IP address is "192.168.1.160", the machine type is "A", and the machine architecture is "X86_64". Attributes.

Table 1 The attribute cluster of the machine

attribute name illustrate type of data ATTR_MACHINE machine hostname string ATTR_IP IP address string ATTR_TYPE machine type string ATTR_ARCH machine architecture string ATTR_OS operating system string ATTR_TOTAL_CPU Total number of CPUs double ATTR_TOTAL_MEM total memory int ATTR_AVAIL_CPU Available CPU double ATTR_AVAIL_MEM Amount of memory available int ATTR_AVG_LOAD average load double CON_VALUE constrained valuation int

Fig. 4 is a flowchart of setting task attribute clusters and constraint requirements in an embodiment of the present invention. In particular, constraint requirements include hard constraints and soft constraints. Hard constraints are necessary conditions for task execution and must be met during the scheduling process. The processing of hard constraints belongs to qualitative analysis. Soft constraints are preference conditions for task execution, which should be satisfied as much as possible to improve task execution efficiency, but if they cannot be satisfied, they can be ignored to avoid waste of resources and delays in task execution. The processing of soft constraints is a quantitative analysis. As shown in Figure 4, the steps for the job manager to set task attribute clusters and constraint requirements are as follows:

Step 401: Set an easily expandable attribute cluster for the task. The attribute cluster includes multiple key-value pairs, where the key represents the attribute of the task, and the value represents the specific value of the attribute, specifically including the task label, execution command, required resources, etc.;

Step 402: setting hard constraint requirements for the task, expressing the hard constraint requirements of the task through a Boolean expression;

Furthermore, there may be multiple hard constraints in the task, and multiple hard constraints can be directly "ANDed", so that multiple hard constraint requirements can still be expressed through a Boolean expression;

Step 403: Set the soft constraint requirements for the task, and express the multiple soft constraint requirements of the task through the soft constraint requirement linked list. The linked list contains several elements, and each element includes a Boolean expression and an estimate, where the Boolean expression indicates For specific soft constraint requirements, the valuation is used to quantify the improvement in execution efficiency brought about by meeting the soft constraint requirements.

FIG. 5 is a schematic diagram of task attribute clusters in an embodiment of the present invention. Figure 5 shows the attribute cluster and the corresponding value of a certain task, and the attribute cluster of the task is set by the job manager. Similar to the machine attribute cluster, the task attribute cluster also includes multiple key-value (Key-Value) pairs, where the key represents the task attribute, and the value represents the specific value of the attribute, where the task attribute includes job ID, task ID, virtualization type , execution command, required CPU, required memory, etc. Table 2 lists the attribute clusters and constraint requirements of the task. In the example shown in Figure 5, the job ID of the task is 1, the task ID is 2, the virtualization type is "KVM", the execution command is "run.sh", the required CPU is 2, and the required memory is 2048 ( MB).

Table 2 Attribute clusters and constraint requirements of tasks

attribute/constraint name illustrate type of data ATTR_JOB_ID job id int ATTR_TASK_ID task ID int ATTR_VMTYPE virtualization type string ATTR_EXE_PATH Excuting an order string ATTR_NEED_CPU Required CPU double ATTR_NEED_MEM required memory int HARD_CONSTRAINT hard constraints boolean expression SOFT_CON_LIST soft constraint linked list linked list

Fig. 6 is a schematic diagram of task hard constraints in an embodiment of the present invention. Hard constraints are a necessary condition for task execution, and the processing results can only be satisfied or not satisfied, so dealing with hard constraints belongs to qualitative analysis. A task may have multiple hard constraints, and we can directly perform "AND operations" on them.

In the example shown in Figure 6, the task has four hard constraints, each of which can be represented by a Boolean expression. Among them, "ATTR_AVAIL_CPU>=ATTR_NEED_CPU" indicates that the currently available CPU of the machine should be greater than or equal to the CPU required by the task, "ATTR_AVAIL_MEM> = ATTR_NEED_MEM" indicates that the current available memory of the machine should be greater than or equal to the memory required by the task, "ATTR_ARCH == X86_64" Indicates that the architecture of the machine should be "X86_64", and "ATTR_OS==Centos 6.3" indicates that the operating system of the machine should be "Centos6.3". The first two of the four hard constraints belong to the constraint requirements at the resource level, ensuring that the machine contains the resources required for the task; the latter two belong to the constraint requirements at the non-resource level. Finally, we can directly AND the four hard constraints of the task to get HARD_CONSTRAINT=(ATTR_AVAIL_CPU>=ATTR_NEED_CPU)&&(ATTR_AVAIL_MEM>=ATTR_NEED_MEM)&&(ATTR_ARCH==X86_64)&&(ATTR_OS==Centos6.3), so still Multiple hard constraint requirements of a task can be represented by a Boolean expression.

FIG. 7 is a schematic diagram of a linked list of task soft constraints in an embodiment of the present invention. Soft constraints are preference conditions for task execution and should be met as much as possible, but they are not mandatory requirements. The processing of soft constraints is not limited to the two cases of satisfaction or dissatisfaction. The degree of satisfaction of multiple soft constraints and the performance improvement brought about by satisfying each soft constraint should be fully considered. Therefore, the processing of soft constraints is a quantitative analysis. The present invention represents a plurality of soft constraint requirements of a task through a soft constraint requirement linked list. The linked list contains several elements, and each element includes a Boolean expression and an estimate, wherein the Boolean expression indicates a specific soft constraint requirement, and the estimate It is used to quantify the improvement of execution efficiency brought about by meeting the soft constraint requirements.

In the example shown in Figure 7, the task has three soft constraints, the specific requirement of the first soft constraint is "ATTR_IP in (192.168.1.160,192.168.170,192.168.1.180)", indicating that the IP address of the machine is preferably the above three One of the IP addresses, the corresponding valuation is 50, indicating that satisfying this soft constraint can bring about a performance improvement of 50; the specific requirement of the second soft constraint is "ATTR_TYPE==A", indicating that the machine type is best type A, The corresponding estimate is 30, indicating that satisfying this soft constraint can bring a performance improvement of 30; the specific requirement of the third soft constraint is "ATTR_AVG_LOAD<=0.5", indicating that the average load of the machine should preferably be less than or equal to 0.5, and the corresponding estimate is 20 , showing that satisfying this soft constraint can lead to a performance improvement of 20.

Fig. 8 is a flow chart of assigning an optimal machine to a task in an embodiment of the present invention. As shown in FIG. 8, in this embodiment, assigning constraints and optimal machines to tasks includes the following steps:

Step 801: Record the received task as "task to be scheduled", initialize the machine set M, put all machines into M, and initialize the candidate machine list to be empty;

Step 802: Take a machine from the machine set M, record it as "candidate machine", and calculate the value of the hard constraint requirement of the task to be scheduled according to the information of the machine and the task;

Step 803: judge whether the task hard constraint is true, if true, go to step 804; if not true, go to step 805;

Step 804: Add the candidate machine to the candidate machine list, and calculate the constraint estimate of the candidate machine according to the machine information and the soft constraint linked list of the task;

Step 805: Take out candidate machines from the machine set M;

Step 806: Determine whether the machine set M is empty, if it is empty, go to step 807; if not, go to step 802;

Step 807: Using the constraint valuation as a standard, select the machine with the largest constraint valuation from the list of candidate machines, and record it as the matching machine for the task to be scheduled.

Fig. 9 is a schematic diagram of computing task hard constraints in an embodiment of the present invention. In the example shown in Figure 9, the attribute cluster of the machine records various attributes of the machine, including the host name of the machine "Blade10", the machine architecture of "x86_64", the operating system of "Centos6.3", and the available CPU of 13 cores, available memory is 23552MB, etc. At the same time, the task information records that its resource requirements are 2 CPU cores and 2GB of memory. The hard constraint requirements are that the available resources of the machine must be greater than the resources required by the task, the machine architecture must be "X86_64", and the operating system must be "Centos6.3". Hard constraint requirements can be written as HARD_CONSTRAINT=(ATTR_AVAIL_CPU>=ATTR_NEED_CPU)&&(ATTR_AVAIL_MEM>=ATTR_NEED_MEM)&&(ATTR_ARCH==X86_64)&&(ATTR_OS==Centos6.3). According to the cluster of machine and task attributes, it can be calculated that the Boolean return value of the hard constraint requirement of the task (HARD_CONSTRAINT) is true, which indicates that the machine meets the hard constraint requirement of the task.

Fig. 10 is a schematic diagram of calculating machine constraint estimates in an embodiment of the present invention. The present invention maintains a constraint valuation (CON_VALUE) for each machine to quantify the overall matching degree of the machine and the soft constraints of the task. The general steps for calculating the constraint valuation of the machine are: first initialize the constraint valuation of the machine to 0; then traverse The soft constraint linked list of the task, for each element, calculate its soft constraint requirement, if the soft constraint requirement is true, then add the constraint value to the corresponding value of the soft constraint; the final constraint value is the required value .

In the example shown in FIG. 10 , the IP address “192.168.1.160”, the machine type “B”, and the average load of 0.3 are recorded in the machine attribute cluster, and its constraint value is initially 0. Then traverse the soft constraint list of the task. For soft constraint 1, the soft constraint requirement: ATTR_IP in(192.168.1.160, 192.168.170, 192.168.1.180) is true, so the corresponding value 50 will be added to the constraint value; for soft constraint 2 , the soft constraint requirement (ATTR_TYPE==A) is not true; for soft constraint 3, the soft constraint requirement (ATTR_AVG_LOAD<=0.5) is true, so the constraint valuation will also add the corresponding valuation 20, and finally the constraint valuation for 70.

FIG. 11 shows the task startup time in the application scenario of the virtual machine in the embodiment of the present invention. In the example shown in Fig. 11, under the condition of different virtual machine image sizes, the task start times satisfying constraints and ignoring constraints are recorded. Among them, the solid line represents the task start time that satisfies the constraint, and the dashed line represents the task start time that ignores the constraint. The data shows that the start-up time of tasks satisfying the constraints is significantly shorter than that of ignoring the constraints, and the specific speedup is related to the size of the image. In this group of examples, the speedup ranges from 6.91 to 24.18.

FIG. 12 shows the task completion time in the application scenario of inter-task communication in the embodiment of the present invention. In the example shown in Fig. 12, the task completion time of satisfying constraints and ignoring constraints under different data scales is recorded. Among them, the solid line represents the task completion time satisfying the constraints, and the dashed line represents the task completion time ignoring the constraints. The data shows that the completion time of the task satisfying the constraints is significantly shorter than that of ignoring the constraints, and the specific speedup ratio is related to the data size and network status, which is about 2.25 in this embodiment. Generally speaking, the task scheduling method proposed in the present invention can handle various constraints and significantly improve task execution efficiency.

The above embodiments are provided only for the purpose of describing the present invention, not to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalent replacements and modifications made without departing from the spirit and principle of the present invention shall fall within the scope of the present invention.

Claims (7)

1. in isomeric group towards a method for scheduling task for mixed load, it is characterized in that performing step is as follows:

Step 1, Resource Scheduler receives machine heartbeat, the attribute bunch of machine maintenance; Described machine heartbeat is the attribute bunch of machine by actuator timed sending to Resource Scheduler, heartbeat content;

Step 2, job manager receives and resolves operation, obtains several tasks;

Step 3, job manager is that task to set a property bunch and constraint demand, then mission bit stream is sent to Resource Scheduler;

Step 4, after Resource Scheduler receives mission bit stream, for task coupling meets constraint and the machine of optimum, and returns to job manager by the matching relationship of task and machine;

Step 5, after job manager receives the matching relationship of task and machine, is issued to task on the actuator on coupling machine, executes the task.

2. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: in described step 1, it is right that the attribute bunch of described machine comprises multiple key assignments (Key-Value), wherein key table shows machine attribute, value then represents the occurrence of attribute, and wherein attribute comprises machine host name, IP address, Machine Type, machine architecture, operating system, CPU sum, memory amount, available CPU, free memory amount, constraint valuation.

3. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: the constraint demand in described step 3 comprises hard constraint and soft-constraint; Described hard constraint is the necessary condition of tasks carrying, must be met in scheduling process; Described soft-constraint is the preferences of tasks carrying, should meet to promote tasks carrying efficiency as far as possible, if but cannot meet, can ignore, in order to avoid cause the delay of the wasting of resources and tasks carrying.

4. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: described step 3 specific implementation step is as follows:

Step 3.1: for task arranges the attribute bunch easily expanded, it is right that described attribute bunch comprises multiple key assignments (Key-Value), wherein key table shows the attribute of task, value then represents the occurrence of attribute, and the attribute bunch of task comprises task sign, fill order, required cpu resource, required memory resource;

Step 3.2: for task arranges hard constraint demand, represents the hard constraint demand of task by a Boolean expression, if there is multiple hard constraint, then they done " with computing ", still can represent multiple hard constraint demand by a Boolean expression;

Step 3.3: for task arranges soft-constraint demand, multiple soft-constraint demands of task are represented by soft-constraint demand chained list, several elements are comprised in chained list, each element comprises a Boolean expression and a valuation, Boolean expression shows concrete soft-constraint demand, and valuation is in order to quantize the lifting meeting the execution efficiency that soft-constraint demand is brought;

Step 3.4: the attribute of task bunch is sent to Resource Scheduler, request dispatching machine with soft or hard constraint demand by job manager.

5. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: described step 4 specific implementation step is as follows:

Step 4.1: receiving of task is designated as " treating scheduler task ", initialization collection of machines M, puts into M by all machines, and the alternative machine list of initialization is empty;

Step 4.2: take out a machine from collection of machines M, be designated as " alternative machine ", according to the information of machine and task, calculates the value treating scheduler task hard constraint demand;

Step 4.3: judge whether the hard constraint demand treating scheduler task is true, if be true, is then joined in alternative machine list by alternative machine, and according to the soft-constraint chained list of machine information and task, calculate the constraint valuation of alternative machine;

Step 4.4: remove alternative machine from collection of machines M, judges whether collection of machines M is empty, does not then go to step 4.2 for sky;

Step 4.5: to retrain valuation for standard, selects the machine that constraint valuation is maximum, is designated as the coupling machine treating scheduler task in alternative machine list.

6. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: in described step 4.3, the constraint valuation calculating alternative machine comprises:

The constraint valuation of alternative machine is initialized as 0;

Traversal treats the soft-constraint chained list of scheduler task, and for each element, calculate its soft-constraint demand, if soft-constraint demand is true, then current constraint valuation adds the valuation of this soft-constraint element, finally obtains the constraint valuation of alternative machine.

7. in isomeric group towards a task scheduling system for mixed load, it is characterized in that comprising: job manager, Resource Scheduler and actuator; Described job manager and Resource Scheduler are deployed on main controlled node, wherein:

Job manager is used for management operations and task, for task to set a property bunch and soft or hard constraint demand, and mission bit stream is sent to Resource Scheduler, the machine needed for request task;

Resource Scheduler is used for the machine heartbeat of receiving actuator timed sending, and on the basis of safeguarding whole clustered machine heartbeat, Resource Scheduler can receive the mission bit stream that job manager sends, for task coupling meets constraint and the machine of optimum;

Described actuator is deployed on other all machines except main controlled node, and timing reports machine heartbeat to Resource Scheduler, and receives the assignment instructions that job manager issues, and is responsible for specifically executing the task.