CN104243617A - Task scheduling method and system facing mixed load in heterogeneous cluster - Google Patents
Task scheduling method and system facing mixed load in heterogeneous cluster Download PDFInfo
- Publication number
- CN104243617A CN104243617A CN201410543294.2A CN201410543294A CN104243617A CN 104243617 A CN104243617 A CN 104243617A CN 201410543294 A CN201410543294 A CN 201410543294A CN 104243617 A CN104243617 A CN 104243617A
- Authority
- CN
- China
- Prior art keywords
- task
- machine
- constraint
- soft
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a task scheduling method and a task scheduling system facing mixed load in a heterogeneous cluster. The method comprises the following steps: a resource scheduler is used for receivingthe beats of machines and maintaining the cluster attributes of the machines; a job manager is used for receiving and resolving operation to obtain a plurality of tasks; the job manager is used for setting the attribute cluster and restrain demand for the task, and transmitting the task information to a resource manager; the resource scheduler is an optimal machine which is used for meeting the restraint of task matching, and returning a matching relation between the task and the machine to the job manager; and the job manager is used for transmitting the task to an actuator on the matched machine for executing tasks. The attribute and task requirement of the heterogeneous machine can be indicated by an easily expandable description method, and the optimal machine is distributed for the task based on taking the hard constraint as a filtration standard and the soft constraint as a selection standard, therefore the executing efficiency of the task and the integral performance of the system can be obviously improved.
Description
Technical field
To the present invention relates in a kind of isomeric group towards the method for scheduling task of mixed load and system, belong to computer parallel computation field.
Background technology
In recent years, clustered machine shows day by day significant isomerization feature.Modern trunked is often larger, and the cycle of operation is longer, even may be distributed in different geographical position.In whole life cycle, cluster usually needs more new engine.In addition, under the scene that cluster is integrated, the small-sized cluster of several different batches may be integrated into a large-scale cluster by cluster administrator.Consider above-mentioned situation, probably there is some difference for the hardware and software of cluster.
On the other hand, along with the development of cloud computing, same cluster runs mixed load has also become a kind of trend, and this has many benefits, such as, improve resource utilization, share data, reduce O&M cost etc.Specifically, cluster may run the task of number of different types, comprise scientific algorithm, large-scale data analysis, long playing Internet service, and software development test etc.
Task scheduling refers to as task matching resource, is placed on machine by task.Can say that task and machine are two key players in task scheduling.In traditional application scenarios, task and machine are all single isomorphisms, therefore scheduling only need consider basic resource requirement, such as based on slot Resourse Distribute (cloud data center virtual resource management study summary [J]. computer application is studied, 2012,29 (7): 2411-2415.).But along with the continuous isomerization of cluster and load, machine attribute and mission requirements there occurs great changes, and the machine of not all can meet the constraint demand of task, task scheduling must consider various constraint: such as image processing tasks has to operate to be had on the machine of GPU, some tasks can only operate on the machine of particular core version, data analysis task should preferentially operate in store related data machine on etc.At present, consider that the task scheduling of constraint is the significant challenge in this field, how by the prolongable mode of one, miscellaneous constraint to be described, and how according to constraint by task scheduling on the machine of optimum, become the key issue of task scheduling in isomerous environment.
Some achievements in research and open source software start to pay close attention to the task scheduling considering constraint.Hadoop YARN can consider this constraint of data locality when carrying out task scheduling, namely preferential by task scheduling on the machine depositing corresponding data (see Vavilapalli V K, Murthy A C, Douglas C, et al.Apache hadoop yarn:Yet another resource negotiator [C] //Proceedings of the 4th annual Symposium on Cloud Computing.ACM, 2013:5.).Streaming Computational frame Storm can preferentially by the task scheduling that frequently communicates on identical or close machine (see Aniello L, Baldoni R, Querzoni L.Adaptive online scheduling in storm [C] //Proceedings of the7th ACM international conference on Distributed event-based systems.ACM, 2013:207-218.).Spark calculates in scene at DAG, rear sequence task can be dispatched to as far as possible front sequence task to export on the machine at data place (see The Apache Software Foundation.Spark Lightning-fast cluster computing [EB/OL]. (2012-1-10) [2014-9-5] .https: //spark.apache.org/.).Under there is the scene of conflict in data locality and fairness, delay dispatching can be used to weigh, this method achieves good effect (see Zaharia M, Borthakur D, Sen Sarma J, et al.Delay scheduling:a simple technique for achieving locality and fairness in cluster scheduling [C] //Proceedings of the 5th European conference on Computer systems.ACM, 2010:265-278.).Above-mentioned research considers some concrete constraint special cases, but constraint kind is diversified, and the colony dispatching device in real work needs to dispatch different types of task, process various constraint usually.Above-mentioned research does not take into full account the isomerization feature of task and machine, does not propose a kind of prolongable constrained dispatch mechanism.
Method for scheduling task general at present still gives tacit consent to task and machine is isomorphism, does not consider constraint in scheduling process, only considers basic resource matched.But due to the isomerization of task and machine, task scheduling must consider various constraint.Existing method can not describe miscellaneous constraint, can not according to retraining task scheduling on the machine of optimum, this task can be caused normally to perform or running time obviously elongated, had a strong impact on the execution efficiency of task and the overall performance of task scheduling system.
Summary of the invention
Technology of the present invention is dealt with problems: overcome the deficiencies in the prior art, there is provided in a kind of isomeric group towards the method for scheduling task of mixed load and system, consider the task scheduling of constraint, according to constraint by optimum for task scheduling machine, execution efficiency and the entire system performance of task can be improve.
The technology of the present invention solution: towards the method for scheduling task of mixed load in a kind of isomeric group, comprise the following steps:
Step 1, Resource Scheduler receives machine heartbeat, the attribute bunch of machine maintenance; Described machine heartbeat is the attribute bunch of machine by actuator timed sending to Resource Scheduler, heartbeat content;
Step 2, job manager receives and resolves operation, obtains several tasks;
Step 3, job manager is that task to set a property bunch and constraint demand, then mission bit stream is sent to Resource Scheduler;
Step 4, after Resource Scheduler receives mission bit stream, for task coupling meets constraint and the machine of optimum, and returns to job manager by the matching relationship of task and machine;
Step 5, after job manager receives the matching relationship of task and machine, is issued to task on the actuator on coupling machine, executes the task.
Further, it is right that the attribute bunch of the machine that described step 1 is mentioned comprises multiple key assignments (Key-Value), wherein key table shows machine attribute, value then represents the occurrence of attribute, and wherein attribute comprises machine host name, IP address, Machine Type, machine architecture, operating system, CPU sum, memory amount, available CPU, free memory amount, constraint valuation etc.
Further, the constraint demand that described step 3 is mentioned comprises hard constraint and soft-constraint.Hard constraint is the necessary condition of tasks carrying, must be met in scheduling process, and the process for hard constraint belongs to qualitative analysis.Soft-constraint is the preferences of tasks carrying, should meet to promote tasks carrying efficiency as far as possible, if but cannot meet, can ignore, in order to avoid cause the delay of the wasting of resources and tasks carrying, quantitative analysis be belonged to the process of soft-constraint.
Further, described step 3 specifically comprises the following steps:
Step 3.1: for task arranges the attribute bunch easily expanded, it is right that attribute bunch comprises multiple key assignments (Key-Value), wherein key table shows the attribute of task, value then represents the occurrence of attribute, and the attribute bunch of task comprises task sign, fill order, required cpu resource, required memory resource etc.;
Step 3.2: for task arranges hard constraint demand, represents the hard constraint demand of task by a Boolean expression, if there is multiple hard constraint, then they done " with computing ", still can represent multiple hard constraint demand by a Boolean expression;
Step 3.3: for task arranges soft-constraint demand, multiple soft-constraint demands of task are represented by soft-constraint demand chained list, several elements are comprised in chained list, each element comprises a Boolean expression and a valuation, Boolean expression shows concrete soft-constraint demand, and valuation is in order to quantize the lifting meeting the execution efficiency that soft-constraint demand is brought;
Step 3.4: the attribute of task bunch is sent to Resource Scheduler, request dispatching machine with soft or hard constraint demand by job manager.
Further, described step 4 specifically comprises the following steps:
Step 4.1: receiving of task is designated as " treating scheduler task ", initialization collection of machines M, puts into M by all machines, and the alternative machine list of initialization is empty;
Step 4.2: take out a machine from collection of machines M, be designated as " alternative machine ", according to the information of machine and task, calculates the value treating scheduler task hard constraint demand;
Step 4.3: judge whether the hard constraint demand treating scheduler task is true, if be true, is then joined in alternative machine list by alternative machine, and according to the soft-constraint chained list of machine information and task, calculate the constraint valuation of alternative machine;
Step 4.4: remove alternative machine from collection of machines M, judges whether collection of machines M is empty, does not then go to step 4.2 for sky;
Step 4.5: to retrain valuation for standard, selects the machine that constraint valuation is maximum, is designated as the coupling machine treating scheduler task in alternative machine list.
Further, in described step 4.3, comprise further when calculating the constraint valuation of alternative machine:
The constraint valuation of alternative machine is initialized as 0;
Traversal treats the soft-constraint chained list of scheduler task, and for each element, calculate its soft-constraint demand, if soft-constraint demand is true, then current constraint valuation adds the valuation of this soft-constraint element, finally obtains the constraint valuation of alternative machine.
For solving the problems of the technologies described above, the invention allows for the task scheduling system towards mixed load in a kind of isomeric group, comprising job manager, Resource Scheduler and actuator;
Described job manager and Resource Scheduler are deployed on main controlled node, and job manager is used for management operations and task, for task to set a property bunch and soft or hard constraint demand, and mission bit stream are sent to Resource Scheduler, the machine needed for request task;
Resource Scheduler is used for the machine heartbeat of receiving actuator timed sending, and on the basis of safeguarding whole clustered machine heartbeat, Resource Scheduler can receive the mission bit stream that job manager sends, for task coupling meets constraint and the machine of optimum;
Described actuator is deployed on other all machines except main controlled node, and timing reports machine heartbeat to Resource Scheduler, and receives the assignment instructions that job manager issues, and is responsible for specifically executing the task.
The present invention's advantage is compared with prior art:
(1) the present invention propose method for scheduling task and system, isomerized machine attribute and mission requirements are represented by a kind of constraint specification method of easy expansion, on this basis, treat hard constraint and soft-constraint with a certain discrimination, using hard constraint as filter criteria, using soft-constraint as choice criteria, for task coupling meets hard constraint and the machine of optimum.The present invention considers various constraint in task scheduling process, significantly improves the execution efficiency of task and the overall performance of system.
(2) task scheduling testing satisfied constraint and the tasks carrying efficiency under the task scheduling strategy ignoring constraint, verify the validity of the method for scheduling task of the consideration constraint that the present invention proposes with this.Figure 11 have recorded the task start time under virtual machine application scenarios, data show, the task start time meeting constraint is significantly shorter than the task start time ignoring constraint, and concrete speed-up ratio is relevant with mirror image size, is not 6.91 to 24.18 not etc. in this group is tested.Figure 12 have recorded the task completion time under the application scenarios of task intercommunication, data show, the task completion time meeting constraint is significantly shorter than the task completion time ignoring constraint equally, and concrete speed-up ratio and data scale, network state are relevant, in this group is tested, be about 2.25.In general, the method for scheduling task that the present invention proposes and system can process multiple restraint condition, and significantly improve tasks carrying efficiency.
Accompanying drawing explanation
Fig. 1 is the principle schematic of method for scheduling task and system in the embodiment of the present invention;
Fig. 2 is the flow chart of method for scheduling task in the embodiment of the present invention;
Fig. 3 is the schematic diagram of machine attribute bunch in the embodiment of the present invention;
Fig. 4 is the flow chart arranging task attribute bunch and constraint demand in the embodiment of the present invention;
Fig. 5 is the schematic diagram of task attribute bunch in the embodiment of the present invention;
Fig. 6 is the schematic diagram of task hard constraint in the embodiment of the present invention;
Fig. 7 is the schematic diagram of task soft-constraint chained list in the embodiment of the present invention;
Fig. 8 is the flow chart for the optimum machine of task matching in the embodiment of the present invention;
Fig. 9 is the schematic diagram of calculation task hard constraint in the embodiment of the present invention;
Figure 10 is the schematic diagram of embodiment of the present invention Computer device constraint valuation;
Figure 11 is the task start time in the embodiment of the present invention under virtual machine application scenarios;
Figure 12 is the task completion time in the embodiment of the present invention under task intercommunication application scenarios.
Embodiment
Be described principle of the present invention and feature below in conjunction with drawings and Examples, example, only for explaining the present invention, is not intended to limit scope of the present invention.
As shown in Figure 1, the embodiment of the present invention achieves a task scheduling system towards mixed load operated on isomeric group, this system adopts typical principal and subordinate (Master-Slave) framework, main control part (Master) comprises two kernel process job managers (Jobs Manager) and Resource Scheduler (Resource Scheduler), and the two is deployed on master control physical node.Comprise a kernel process actuator (Executor) from part (Slave), be deployed on other all machines beyond master control physical node.
Job manager is in charge of operation and task, and an operation comprises several tasks, and a group job ID and task ID uniquely can indicate a task.Job manager receives and resolves the operation that user submits to, is that task to set a property bunch and soft or hard constraint, and mission bit stream is sent to Resource Scheduler, the machine needed for request dispatching according to job parameter.After acquisition coupling machine, task is issued in specified machine by job manager again, and the executing state of monitor task is also carried out fault-tolerant.
Resource Scheduler is responsible for machine heartbeat (Heartbeat) information of receiving actuator timed sending, and these heartbeat messages contain the attribute bunch of machine.On the basis of safeguarding whole clustered machine attribute bunch, Resource Scheduler can receive the mission bit stream that job manager sends, and for task matching meets constraint and the machine of optimum, and by the matching relationship of task and machine as a result, returns to job scheduler.
Actuator is responsible for the instruction receiving job manager, and startup is executed the task, to job manager Report Tasks state; On the other hand, the attribute of this machine bunch is offered Resource Scheduler by heartbeat by actuator timing in form.For each task, first actuator creates a virtualized environment, then executes the task in virtualized environment inside.
As shown in Figure 2, in the present embodiment, method for scheduling task can comprise the steps:
Step 201, Resource Scheduler receives machine heartbeat, the attribute bunch of machine maintenance;
Step 202, job manager receives and resolves operation, obtains several tasks;
Step 203, job manager is that task to set a property bunch and constraint demand, then mission bit stream is sent to explorer;
Step 204, after Resource Scheduler receives mission bit stream, for task coupling meets constraint and the machine of optimum, and returns to job manager by the matching relationship of task and machine;
Step 205, after job manager receives the matching relationship of task and machine, is issued to task on the actuator on coupling machine, executes the task.
Fig. 3 is the schematic diagram of machine attribute bunch in the embodiment of the present invention.Fig. 3 indicates the attribute bunch of certain machine and is worth accordingly, and the attribute bunch of machine is the particular content of machine heartbeat, by actuator timed sending to Resource Scheduler.It is right that attribute bunch comprises multiple key assignments (Key-Value), wherein key table shows machine attribute, value then represents the occurrence of attribute, wherein machine attribute comprises machine host name, IP address, Machine Type, machine architecture, operating system, CPU sum, memory amount, available CPU, free memory amount, constraint valuation etc., and table 1 lists the attribute bunch of machine.In example shown in Fig. 3, the main frame " Blade10 " by name of this machine, IP address are " 192.168.1.160 ", Machine Type is " A ", and machine architecture is " X86_64 ", and Fig. 3 also demonstrates other attributes of machine in addition.
The attribute bunch of table 1 machine
| Attribute-name | Explanation | Data type |
| ATTR_MACHINE | Machine host name | string |
| ATTR_IP | IP address | string |
| ATTR_TYPE | Machine Type | string |
| ATTR_ARCH | Machine architecture | string |
| ATTR_OS | Operating system | string |
| ATTR_TOTAL_CPU | CPU sum | double |
| ATTR_TOTAL_MEM | Memory amount | int |
| ATTR_AVAIL_CPU | Available CPU | double |
| ATTR_AVAIL_MEM | Free memory amount | int |
| ATTR_AVG_LOAD | Average load | double |
| CON_VALUE | Constraint valuation | int |
Fig. 4 is the flow chart arranging task attribute bunch and constraint demand in the embodiment of the present invention.A bit, constraint demand comprises hard constraint and soft-constraint to special instruction.Hard constraint is the necessary condition of tasks carrying, must be met in scheduling process, and the process for hard constraint belongs to qualitative analysis.Soft-constraint is the preferences of tasks carrying, should meet to promote tasks carrying efficiency as far as possible, if but cannot meet, can ignore, in order to avoid cause the delay of the wasting of resources and tasks carrying, quantitative analysis be belonged to the process of soft-constraint.As shown in Figure 4, to arrange the step of task attribute bunch and constraint demand as follows for job manager:
Step 401: for task arranges the attribute bunch easily expanded, attribute bunch comprises multiple key-value pair, and wherein key table shows the attribute of task, and value then represents the occurrence of attribute, specifically comprises task sign, fill order, resource requirement etc.;
Step 402: for task arranges hard constraint demand, represents the hard constraint demand of task by a Boolean expression;
Further, may there is multiple hard constraint in task, can directly be done " with computing " by multiple hard constraint, so still can represent multiple hard constraint demand by a Boolean expression;
Step 403: for task arranges soft-constraint demand, multiple soft-constraint demands of task are represented by soft-constraint demand chained list, several elements are comprised in chained list, each element comprises a Boolean expression and a valuation, wherein Boolean expression shows concrete soft-constraint demand, and valuation is in order to quantize the lifting meeting the execution efficiency that soft-constraint demand is brought.
Fig. 5 is the schematic diagram of task attribute bunch in the embodiment of the present invention.Fig. 5 indicates the attribute bunch of certain task and is worth accordingly, and the attribute bunch of task is responsible for setting by job manager.Bunch similar with machine attribute, it is right that the attribute bunch of task also comprises multiple key assignments (Key-Value), wherein key table shows task attribute, value then represents the occurrence of attribute, wherein task attribute comprises operation ID, task ID, virtual type, fill order, required CPU, required memory etc., and table 2 lists attribute bunch and the constraint demand of task.In the example embodiment shown in fig. 5, the operation ID of this task is 1, task ID is 2, and virtual type is " KVM ", fill order is " run.sh ", required CPU is 2, required memory is 2048 (MB).
The attribute of table 2 task bunch and constraint demand
| Attributes/constraints name | Explanation | Data type |
| ATTR_JOB_ID | Operation ID | int |
| ATTR_TASK_ID | Task ID | int |
| ATTR_VMTYPE | Virtual type | string |
| ATTR_EXE_PATH | Fill order | string |
| ATTR_NEED_CPU | Required CPU | double |
| ATTR_NEED_MEM | Required memory | int |
| HARD_CONSTRAINT | Hard constraint demand | Bool expression formula |
| SOFT_CON_LIST | Soft-constraint chained list | Chained list |
Fig. 6 is the schematic diagram of task hard constraint in the embodiment of the present invention.Hard constraint is the necessary condition of tasks carrying, and its result can only be meet or do not meet two kinds of situations, therefore processes hard constraint and belongs to qualitative analysis.May be there is multiple hard constraint in a task, we can directly do " with computing " it.
In example shown in Fig. 6, task has four hard constraints, and each hard constraint can be represented by a Boolean expression.Wherein, " ATTR_AVAIL_CPU>=ATTR_NEED_CPU " represents that the current available CPU of machine should be more than or equal to the CPU of required by task, " ATTR_AVAIL_MEM>=ATTR_NEED_MEM " represents that the current available internal memory of machine should be more than or equal to the internal memory of required by task, " ATTR_ARCH==X86_64 " represents that the framework of machine should be " X86_64 ", and " ATTR_OS==Centos 6.3 " represents that the operating system of machine should be " Centos6.3 ".The first two in four hard constraints belongs to the constraint demand of resource aspect, ensures the resource comprising required by task in machine; Latter two then belongs to the constraint demand of non-resource aspect.Finally, we can by four of a task hard constraint directly with, obtain HARD_CONSTRAINT=(ATTR_AVAIL_CPU>=ATTR_NEED_CPU) & & (ATTR_AVAIL_MEM>=ATTR_NEED_MEM) & & (ATTR_ARCH==X86_64) & & (ATTR_OS==Centos6.3), so still, multiple hard constraint demands of task can be represented by a Boolean expression.
Fig. 7 is the schematic diagram of task soft-constraint chained list in the embodiment of the present invention.Soft-constraint is the preferences of tasks carrying, should be met as far as possible, but is not mandatory demand.Process for soft-constraint not only has and meets or do not meet two kinds of situations, and should take into full account the satisfaction degree of multiple soft-constraint, and meet the performance boost that each soft-constraint brings tasks carrying, the process therefore for soft-constraint belongs to quantitative analysis.The present invention represents multiple soft-constraint demands of task by soft-constraint demand chained list, several elements are comprised in chained list, each element comprises a Boolean expression and a valuation, wherein Boolean expression shows concrete soft-constraint demand, and valuation is in order to quantize the lifting meeting the execution efficiency that soft-constraint demand is brought.
In example shown in Fig. 7, task has three soft-constraints, the real needs of first soft-constraint are " ATTR_IP in (192.168.1.160; 192.168.170; 192.168.1.180) ", show preferably one of above-mentioned three IP addresses, IP address of machine, corresponding valuation is 50, shows to meet the performance boost that this soft-constraint can bring 50; The real needs of second soft-constraint are " ATTR_TYPE==A ", and show Machine Type preferably A type, corresponding estimation is 30, show to meet the performance boost that this soft-constraint can bring 30; The real needs of the 3rd soft-constraint are " ATTR_AVG_LOAD<=0.5 ", and show that the average load of machine is preferably less than or equal to 0.5, corresponding valuation is 20, show to meet the performance boost that this soft-constraint can bring 20.
Fig. 8 is the flow chart for the optimum machine of task matching in the embodiment of the present invention.As shown in Figure 8, in the present embodiment, for task matching constraint and the machine of optimum comprise the steps:
Step 801: receiving of task is designated as " treating scheduler task ", initialization collection of machines M, puts into M by all machines, and the alternative machine list of initialization is empty;
Step 802: take out a machine from collection of machines M, be designated as " alternative machine ", according to the information of machine and task, calculates the value treating scheduler task hard constraint demand;
Step 803: judge whether task hard constraint is true, if be true, then goes to step 804; If be not true, then go to step 805;
Step 804: alternative machine is joined in alternative machine list, and according to the soft-constraint chained list of machine information and task, calculate the constraint valuation of alternative machine;
Step 805: take out alternative machine from collection of machines M;
Step 806: judge whether collection of machines M is empty, if be empty, then goes to step 807; If be not empty, then go to step 802;
Step 807: to retrain valuation for standard, selects the machine that constraint valuation is maximum, is designated as the coupling machine treating scheduler task in alternative machine list.
Fig. 9 is the schematic diagram of calculation task hard constraint in the embodiment of the present invention.In the example shown in Fig. 9, the attribute bunch of machine records each attribute of this machine, comprises machine main frame by name " Blade10 ", machine architecture is " x86_64 ", operating system is " Centos6.3 ", available CPU is 13 cores, free memory is 23552MB etc.Simultaneously, its resource requirement of mission bit stream record is 2 CPU cores, 2GB internal memory, hard constraint demand is that machine available resources must be greater than required by task resource, machine architecture must be " X86_64 ", operating system must be " Centos6.3 ", its hard constraint demand can write HARD_CONSTRAINT=(ATTR_AVAIL_CPU>=ATTR_NEED_CPU) & & (ATTR_AVAIL_MEM>=ATTR_NEED_MEM) & & (ATTR_ARCH==X86_64) & & (ATTR_OS==Centos6.3).According to machine and task attribute bunch, the Boolean return that can calculate task hard constraint demand (HARD_CONSTRAINT) is true, and this shows that this machine meets the hard constraint demand of task.
Figure 10 is the schematic diagram of embodiment of the present invention Computer device constraint valuation.The present invention is each machine maintenance one constraint valuation (CON_VALUE), and in order to quantize the overall matching degree of machine and task soft-constraint, the general steps of computing machine constraint valuation is: the constraint valuation of first initialization machine is 0; Then travel through the soft-constraint chained list of task, for each element, calculate its soft-constraint demand, if soft-constraint demand is true, then constraint valuation is added the corresponding valuation of this soft-constraint; Required by the constraint valuation finally obtained is.
In the example shown in Figure 10, record that its IP address is " 192.168.1.160 ", Machine Type is " B ", average load is 0.3 in machine attribute bunch, its constraint valuation is 0 at first.Then travel through the soft-constraint chained list of task, for soft-constraint 1, soft-constraint demand: ATTR_IP in (192.168.1.160,192.168.170,192.168.1.180) is true, so constraint valuation can add corresponding valuation 50; For soft-constraint 2, soft-constraint demand (ATTR_TYPE==A) is not true; For soft-constraint 3, soft-constraint demand (ATTR_AVG_LOAD<=0.5) is true, so constraint valuation also can add corresponding valuation 20, finally obtaining constraint valuation is 70.
Figure 11 is the task start time in the embodiment of the present invention under virtual machine application scenarios.In the example shown in Figure 11, have recorded when different virtual machine mirror image size, meet the task start time retraining and ignore constraint.Wherein, solid line representative meets the task start time of constraint, and represented by dotted arrows ignores the task start time of constraint.Data show, and the task start time meeting constraint is significantly shorter than the task start time ignoring constraint, and concrete speed-up ratio is relevant with mirror image size, and in this group embodiment, speed-up ratio is not 6.91 to 24.18 not etc.
Figure 12 is the task completion time in the embodiment of the present invention under task intercommunication application scenarios.In the example shown in Figure 12, have recorded when different pieces of information scale, meet the task completion time retraining and ignore constraint.Wherein, solid line representative meets the task completion time of constraint, and represented by dotted arrows ignores the task completion time of constraint.Data show, and the task completion time meeting constraint is significantly shorter than the task completion time ignoring constraint, and concrete speed-up ratio and data scale, network state are relevant, are about 2.25 in the present embodiment.In general, the method for scheduling task that the present invention proposes can process multiple restraint condition, and significantly improves tasks carrying efficiency.
There is provided above embodiment to be only used to describe object of the present invention, and do not really want to limit the scope of the invention.Scope of the present invention is defined by the following claims.Do not depart from spirit of the present invention and principle and the various equivalent substitutions and modifications made, all should contain within the scope of the present invention.
Claims (7)
1. in isomeric group towards a method for scheduling task for mixed load, it is characterized in that performing step is as follows:
Step 1, Resource Scheduler receives machine heartbeat, the attribute bunch of machine maintenance; Described machine heartbeat is the attribute bunch of machine by actuator timed sending to Resource Scheduler, heartbeat content;
Step 2, job manager receives and resolves operation, obtains several tasks;
Step 3, job manager is that task to set a property bunch and constraint demand, then mission bit stream is sent to Resource Scheduler;
Step 4, after Resource Scheduler receives mission bit stream, for task coupling meets constraint and the machine of optimum, and returns to job manager by the matching relationship of task and machine;
Step 5, after job manager receives the matching relationship of task and machine, is issued to task on the actuator on coupling machine, executes the task.
2. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: in described step 1, it is right that the attribute bunch of described machine comprises multiple key assignments (Key-Value), wherein key table shows machine attribute, value then represents the occurrence of attribute, and wherein attribute comprises machine host name, IP address, Machine Type, machine architecture, operating system, CPU sum, memory amount, available CPU, free memory amount, constraint valuation.
3. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: the constraint demand in described step 3 comprises hard constraint and soft-constraint; Described hard constraint is the necessary condition of tasks carrying, must be met in scheduling process; Described soft-constraint is the preferences of tasks carrying, should meet to promote tasks carrying efficiency as far as possible, if but cannot meet, can ignore, in order to avoid cause the delay of the wasting of resources and tasks carrying.
4. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: described step 3 specific implementation step is as follows:
Step 3.1: for task arranges the attribute bunch easily expanded, it is right that described attribute bunch comprises multiple key assignments (Key-Value), wherein key table shows the attribute of task, value then represents the occurrence of attribute, and the attribute bunch of task comprises task sign, fill order, required cpu resource, required memory resource;
Step 3.2: for task arranges hard constraint demand, represents the hard constraint demand of task by a Boolean expression, if there is multiple hard constraint, then they done " with computing ", still can represent multiple hard constraint demand by a Boolean expression;
Step 3.3: for task arranges soft-constraint demand, multiple soft-constraint demands of task are represented by soft-constraint demand chained list, several elements are comprised in chained list, each element comprises a Boolean expression and a valuation, Boolean expression shows concrete soft-constraint demand, and valuation is in order to quantize the lifting meeting the execution efficiency that soft-constraint demand is brought;
Step 3.4: the attribute of task bunch is sent to Resource Scheduler, request dispatching machine with soft or hard constraint demand by job manager.
5. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: described step 4 specific implementation step is as follows:
Step 4.1: receiving of task is designated as " treating scheduler task ", initialization collection of machines M, puts into M by all machines, and the alternative machine list of initialization is empty;
Step 4.2: take out a machine from collection of machines M, be designated as " alternative machine ", according to the information of machine and task, calculates the value treating scheduler task hard constraint demand;
Step 4.3: judge whether the hard constraint demand treating scheduler task is true, if be true, is then joined in alternative machine list by alternative machine, and according to the soft-constraint chained list of machine information and task, calculate the constraint valuation of alternative machine;
Step 4.4: remove alternative machine from collection of machines M, judges whether collection of machines M is empty, does not then go to step 4.2 for sky;
Step 4.5: to retrain valuation for standard, selects the machine that constraint valuation is maximum, is designated as the coupling machine treating scheduler task in alternative machine list.
6. in isomeric group according to claim 1 towards the method for scheduling task of mixed load, it is characterized in that: in described step 4.3, the constraint valuation calculating alternative machine comprises:
The constraint valuation of alternative machine is initialized as 0;
Traversal treats the soft-constraint chained list of scheduler task, and for each element, calculate its soft-constraint demand, if soft-constraint demand is true, then current constraint valuation adds the valuation of this soft-constraint element, finally obtains the constraint valuation of alternative machine.
7. in isomeric group towards a task scheduling system for mixed load, it is characterized in that comprising: job manager, Resource Scheduler and actuator; Described job manager and Resource Scheduler are deployed on main controlled node, wherein:
Job manager is used for management operations and task, for task to set a property bunch and soft or hard constraint demand, and mission bit stream is sent to Resource Scheduler, the machine needed for request task;
Resource Scheduler is used for the machine heartbeat of receiving actuator timed sending, and on the basis of safeguarding whole clustered machine heartbeat, Resource Scheduler can receive the mission bit stream that job manager sends, for task coupling meets constraint and the machine of optimum;
Described actuator is deployed on other all machines except main controlled node, and timing reports machine heartbeat to Resource Scheduler, and receives the assignment instructions that job manager issues, and is responsible for specifically executing the task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410543294.2A CN104243617B (en) | 2014-10-14 | 2014-10-14 | Towards the method for scheduling task and system of mixed load in a kind of isomeric group |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410543294.2A CN104243617B (en) | 2014-10-14 | 2014-10-14 | Towards the method for scheduling task and system of mixed load in a kind of isomeric group |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104243617A true CN104243617A (en) | 2014-12-24 |
| CN104243617B CN104243617B (en) | 2017-10-27 |
Family
ID=52230945
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410543294.2A Active CN104243617B (en) | 2014-10-14 | 2014-10-14 | Towards the method for scheduling task and system of mixed load in a kind of isomeric group |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104243617B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105022670A (en) * | 2015-07-17 | 2015-11-04 | 中国海洋大学 | Heterogeneous distributed task processing system and processing method in cloud computing platform |
| CN105302643A (en) * | 2015-10-14 | 2016-02-03 | 浪潮集团有限公司 | Job scheduling method and self-learning scheduling machine |
| CN105589745A (en) * | 2015-12-18 | 2016-05-18 | 中国科学院软件研究所 | Unbalanced task allocation supported dynamic vulnerability discovery system and method |
| CN107025141A (en) * | 2017-05-18 | 2017-08-08 | 成都海天数联科技有限公司 | A kind of dispatching method based on big data mixture operation model |
| CN107357661A (en) * | 2017-07-12 | 2017-11-17 | 北京航空航天大学 | A kind of fine granularity GPU resource management method for mixed load |
| CN107515784A (en) * | 2016-06-16 | 2017-12-26 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of computing resource in a distributed system |
| CN107678752A (en) * | 2017-08-31 | 2018-02-09 | 北京百度网讯科技有限公司 | A kind of task processing method and device towards isomeric group |
| CN109101339A (en) * | 2018-08-15 | 2018-12-28 | 北京邮电大学 | Video task parallel method, device and Heterogeneous Cluster Environment in isomeric group |
| CN110012062A (en) * | 2019-02-22 | 2019-07-12 | 北京奇艺世纪科技有限公司 | A kind of multimachine room method for scheduling task, device and storage medium |
| CN111147546A (en) * | 2019-11-29 | 2020-05-12 | 中科院计算技术研究所大数据研究院 | Method and system for processing edge cluster resources |
| CN114168283A (en) * | 2021-12-02 | 2022-03-11 | 北京千帆阅文科技有限公司 | Distributed timed task scheduling method and system |
| CN114787830A (en) * | 2019-12-20 | 2022-07-22 | 惠普发展公司，有限责任合伙企业 | Machine learning workload orchestration in heterogeneous clusters |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101271405A (en) * | 2008-05-13 | 2008-09-24 | 武汉理工大学 | Bidirectional grade gridding resource scheduling method based on QoS restriction |
| CN102495758A (en) * | 2011-12-05 | 2012-06-13 | 中南大学 | Scheduling method of real-time tasks in distributing type high performance calculation environment |
| US20140068049A1 (en) * | 2012-09-03 | 2014-03-06 | Bull Sas | Method and device for processing commands in a set of components of a computer system |
| CN103631870A (en) * | 2013-11-06 | 2014-03-12 | 广东电子工业研究院有限公司 | System and method used for large-scale distributed data processing |
-
2014
- 2014-10-14 CN CN201410543294.2A patent/CN104243617B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101271405A (en) * | 2008-05-13 | 2008-09-24 | 武汉理工大学 | Bidirectional grade gridding resource scheduling method based on QoS restriction |
| CN102495758A (en) * | 2011-12-05 | 2012-06-13 | 中南大学 | Scheduling method of real-time tasks in distributing type high performance calculation environment |
| US20140068049A1 (en) * | 2012-09-03 | 2014-03-06 | Bull Sas | Method and device for processing commands in a set of components of a computer system |
| CN103631870A (en) * | 2013-11-06 | 2014-03-12 | 广东电子工业研究院有限公司 | System and method used for large-scale distributed data processing |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105022670B (en) * | 2015-07-17 | 2018-03-13 | 中国海洋大学 | Heterogeneous distributed task processing system and its processing method in a kind of cloud computing platform |
| CN105022670A (en) * | 2015-07-17 | 2015-11-04 | 中国海洋大学 | Heterogeneous distributed task processing system and processing method in cloud computing platform |
| CN105302643A (en) * | 2015-10-14 | 2016-02-03 | 浪潮集团有限公司 | Job scheduling method and self-learning scheduling machine |
| CN105302643B (en) * | 2015-10-14 | 2018-08-24 | 浪潮集团有限公司 | A kind of method and self study scheduler of job scheduling |
| CN105589745A (en) * | 2015-12-18 | 2016-05-18 | 中国科学院软件研究所 | Unbalanced task allocation supported dynamic vulnerability discovery system and method |
| CN107515784A (en) * | 2016-06-16 | 2017-12-26 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of computing resource in a distributed system |
| CN107515784B (en) * | 2016-06-16 | 2021-07-06 | 阿里巴巴集团控股有限公司 | Method and equipment for calculating resources in distributed system |
| CN107025141A (en) * | 2017-05-18 | 2017-08-08 | 成都海天数联科技有限公司 | A kind of dispatching method based on big data mixture operation model |
| CN107025141B (en) * | 2017-05-18 | 2020-09-01 | 成都海天数联科技有限公司 | Scheduling method based on big data mixed operation model |
| CN107357661B (en) * | 2017-07-12 | 2020-07-10 | 北京航空航天大学 | Fine-grained GPU resource management method for mixed load |
| CN107357661A (en) * | 2017-07-12 | 2017-11-17 | 北京航空航天大学 | A kind of fine granularity GPU resource management method for mixed load |
| US10977076B2 (en) | 2017-08-31 | 2021-04-13 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Method and apparatus for processing a heterogeneous cluster-oriented task |
| CN107678752A (en) * | 2017-08-31 | 2018-02-09 | 北京百度网讯科技有限公司 | A kind of task processing method and device towards isomeric group |
| CN109101339B (en) * | 2018-08-15 | 2019-05-31 | 北京邮电大学 | Video task parallel method, device and Heterogeneous Cluster Environment in isomeric group |
| CN109101339A (en) * | 2018-08-15 | 2018-12-28 | 北京邮电大学 | Video task parallel method, device and Heterogeneous Cluster Environment in isomeric group |
| CN110012062A (en) * | 2019-02-22 | 2019-07-12 | 北京奇艺世纪科技有限公司 | A kind of multimachine room method for scheduling task, device and storage medium |
| CN110012062B (en) * | 2019-02-22 | 2022-02-08 | 北京奇艺世纪科技有限公司 | Multi-computer-room task scheduling method and device and storage medium |
| CN111147546A (en) * | 2019-11-29 | 2020-05-12 | 中科院计算技术研究所大数据研究院 | Method and system for processing edge cluster resources |
| CN114787830A (en) * | 2019-12-20 | 2022-07-22 | 惠普发展公司，有限责任合伙企业 | Machine learning workload orchestration in heterogeneous clusters |
| CN114168283A (en) * | 2021-12-02 | 2022-03-11 | 北京千帆阅文科技有限公司 | Distributed timed task scheduling method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104243617B (en) | 2017-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104243617A (en) | Task scheduling method and system facing mixed load in heterogeneous cluster | |
| Liu et al. | Adaptive asynchronous federated learning in resource-constrained edge computing | |
| CN106776005B (en) | Resource management system and method for containerized application | |
| Jalaparti et al. | Network-aware scheduling for data-parallel jobs: Plan when you can | |
| Warneke et al. | Nephele: efficient parallel data processing in the cloud | |
| CN104050042B (en) | The resource allocation methods and device of ETL operations | |
| CN103593242B (en) | Resource sharing control system based on Yarn frameworks | |
| US20240036937A1 (en) | Workload placement for virtual gpu enabled systems | |
| CN103425536B (en) | A kind of test resource management method of Based on Distributed system performance testing | |
| CN110661842B (en) | Resource scheduling management method, electronic equipment and storage medium | |
| Zhang et al. | MrHeter: improving MapReduce performance in heterogeneous environments | |
| CN108920153A (en) | A kind of Docker container dynamic dispatching method based on load estimation | |
| CN104050043A (en) | Share cache perception-based virtual machine scheduling method and device | |
| CN112416585A (en) | GPU resource management and intelligent scheduling method for deep learning | |
| US20210390405A1 (en) | Microservice-based training systems in heterogeneous graphic processor unit (gpu) cluster and operating method thereof | |
| Patni et al. | Load balancing strategies for grid computing | |
| CN111190691A (en) | Automatic migration method, system, device and storage medium suitable for virtual machine | |
| CN104503832A (en) | Virtual machine scheduling system and virtual machine scheduling method with balanced equity and efficiency | |
| Kijsipongse et al. | A hybrid GPU cluster and volunteer computing platform for scalable deep learning | |
| CN104112049A (en) | P2P (peer-to-peer) architecture based cross-data-center MapReduce task scheduling system and P2P architecture based cross-data-center MapReduce task scheduling method | |
| Di Sanzo et al. | A flexible framework for accurate simulation of cloud in-memory data stores | |
| Sánchez et al. | Agent-based platform to support the execution of parallel tasks | |
| Liu et al. | Optimizing shuffle in wide-area data analytics | |
| Zhang et al. | A spark scheduling strategy for heterogeneous cluster | |
| CN107528871A (en) | Data analysis in storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |