CN104243617B - Towards the method for scheduling task and system of mixed load in a kind of isomeric group - Google Patents
Towards the method for scheduling task and system of mixed load in a kind of isomeric group Download PDFInfo
- Publication number
- CN104243617B CN104243617B CN201410543294.2A CN201410543294A CN104243617B CN 104243617 B CN104243617 B CN 104243617B CN 201410543294 A CN201410543294 A CN 201410543294A CN 104243617 B CN104243617 B CN 104243617B
- Authority
- CN
- China
- Prior art keywords
- task
- machine
- constraint
- soft
- demand
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to the method for scheduling task and system in a kind of isomeric group towards mixed load, comprise the following steps：Resource Scheduler receives machine heartbeat, the attribute cluster of machine maintenance；Job manager receives and parses through operation, obtains several tasks；Job manager is that task sets a property cluster and constraint demand, and mission bit stream is sent into explorer；Resource Scheduler is that task matching meets constraint and optimal machine, and the matching relationship of task and machine is returned into job manager；Job manager performs task by the actuator in mission dispatching to matching machine.The present invention represents the machine attribute and mission requirements of isomerization by a kind of constraint specification method easily expanded, on this basis, using hard constraint as filter criteria, by soft-constraint alternatively standard, for task optimal scheme machine, the execution efficiency of task and the overall performance of system are significantly improved.
Description
Technical field
The present invention relates to the method for scheduling task and system in a kind of isomeric group towards mixed load, belong to computer simultaneously
Row calculating field.
Background technology
In recent years, clustered machine shows increasingly significant isomerization feature.Modern trunked is often larger, operation week
Phase is longer, in some instances it may even be possible to be distributed in different geographical position.In whole life cycle, cluster usually needs more new engine.This
Outside, under the scene that cluster is integrated, the small-sized cluster of several different batches may be integrated into one large-scale by cluster administrator
Cluster.In view of above-mentioned situation, the hardware and software of cluster is likely that there are different.
On the other hand, continuing to develop with cloud computing, operation mixed load also has become one kind on same cluster
Trend, this has many benefits, such as improving resource utilization, shared data, reduction O＆M cost.Specifically, cluster
On may run a variety of different types of tasks, including scientific algorithm, large-scale data analysis, the interconnection of long-play
Net service, and software development test etc..
Task scheduling refers to distribute resource for task, i.e., task is placed on machine.It may be said that task and machine are to appoint
Two key players in business scheduling.In traditional application scenarios, task and machine are all single isomorphisms, therefore scheduling need to only be examined
Consider basic resource requirement, such as resource allocation (cloud data center virtual resource management study summary [J] meters based on slot
Calculation machine application study, 2012,29 (7):2411-2415.).But with cluster and load continuous isomerization, machine attribute and
Mission requirements there occurs great changes, and the machine of not all can meet the constraint demand of task, and task scheduling must take into consideration
Various constraints：Such as image processing tasks are had to operate on the machine with GPU, and some tasks can only operate in particular core
On the machine of version, data analysis task should preferentially operate in storage related data machine on etc..At present, it is considered to constraint
Task scheduling is the significant challenge in the field, and miscellaneous constraint how is described by a kind of prolongable mode, and
How according to constraining in task scheduling to optimal machine, the key issue of task scheduling in isomerous environment is had become.
Some achievements in research and open source software begin to focus on the task scheduling for considering constraint.Hadoop YARN are being appointed
Can consider this constraint of data locality during business scheduling, i.e., it is preferential by task scheduling to the machine for depositing corresponding data (referring to
Vavilapalli V K,Murthy A C,Douglas C,et al.Apache hadoop yarn:Yet another
resource negotiator[C]//Proceedings of the 4th annual Symposium on Cloud
Computing.ACM,2013:5.).Streaming computing framework Storm can be preferential by the task scheduling frequently communicated to identical or phase
(referring to Aniello L, Baldoni R, Querzoni L.Adaptive online scheduling near machine
storm[C]//Proceedings of the7th ACM international conference on Distributed
event-based systems.ACM,2013:207-218.).Spark is calculated in DAG in scene, and rear sequence task can be tried one's best
(referring to The Apache Software Foundation.Spark on machine before being dispatched to where sequence task output data
Lightning-fast cluster computing[EB/OL].(2012-1-10)[2014-9-5].https://
spark.apache.org/.).Exist in data locality and fairness under the scene of conflict, delay dispatching can be used to carry out
Balance, this method achieves preferable effect (referring to Zaharia M, Borthakur D, Sen Sarma J, et
al.Delay scheduling:a simple technique for achieving locality and fairness in
cluster scheduling[C]//Proceedings of the 5th European conference on Computer
systems.ACM,2010:265-278.).The studies above considers some specific constraint special cases, but constraint species is many
Sample, the colony dispatching device in real work usually requires to dispatch different types of task, handles various constraints.On
State research and do not take into full account task and the isomerization feature of machine, do not propose a kind of prolongable constrained dispatch mechanism.
Method for scheduling task still default task and machine general at present is isomorphism, is not considered about in scheduling process
Beam, only considers basic resource matched.But, due to the isomerization of task and machine, task scheduling must take into consideration various constraints.
Existing method can not describe miscellaneous constraint, can not be according to constraint by task scheduling to optimal machine, and this can lead
Cause task can not normally be performed or run time is substantially elongated, has had a strong impact on the execution efficiency and task scheduling system of task
Overall performance.
The content of the invention
The technology of the present invention solves problem：Overcoming the deficiencies in the prior art, there is provided negative towards mixing in a kind of isomeric group
The method for scheduling task and system of load, it is considered to the task scheduling of constraint, can be carried according to constraint by the optimal machine of task scheduling
The execution efficiency and systematic entirety energy of high task.
The technology of the present invention solution：Towards the method for scheduling task of mixed load in a kind of isomeric group, including it is following
Step：
Step 1, Resource Scheduler receives machine heartbeat, the attribute cluster of machine maintenance；The machine heartbeat is by actuator
Attribute cluster that timing is sent to Resource Scheduler, that heartbeat content is machine；
Step 2, job manager receives and parses through operation, obtains several tasks；
Step 3, job manager is that task sets a property cluster and constraint demand, and mission bit stream then is sent into resource adjusts
Spend device；
Step 4, Resource Scheduler is received after mission bit stream, is that task matching meets constraint and optimal machine, and will appoint
Business and the matching relationship of machine return to job manager；
Step 5, job manager is received after the matching relationship of task and machine, by mission dispatching to matching machine
On actuator, task is performed.
Further, the attribute cluster for the machine that the step 1 is mentioned is right including multiple key assignments (Key-Value), wherein key
Machine attribute is represented, value then represents the occurrence of attribute, and wherein attribute includes machine host name, IP address, machine type, machine
Framework, operating system, CPU sums, memory amount, CPU, free memory amount, constraint valuation etc. can be used.
Further, the constraint demand that the step 3 is mentioned includes hard constraint and soft-constraint.Hard constraint is tasks carrying
Necessary condition, must be satisfied in scheduling process, and the processing for hard constraint belongs to qualitative analysis.Soft-constraint is that task is held
Capable preferences, the satisfaction that should try one's best can be ignored with lifting tasks carrying efficiency if it can not meet, in order to avoid cause
The wasting of resources and the delay of tasks carrying, the processing to soft-constraint belong to quantitative analysis.
Further, the step 3 specifically includes following steps：
Step 3.1：The attribute cluster easily expanded is set for task, attribute cluster is right including multiple key assignments (Key-Value), wherein
Key represents the attribute of task, and value then represents the occurrence of attribute, the attribute cluster of task include task sign, perform order, it is required
Cpu resource, required memory source etc.；
Step 3.2：Hard constraint demand is set for task, the hard constraint of task is represented by a Boolean expression to be needed
Ask, if there is multiple hard constraints, then they are done into " with computing ", still can represent multiple by a Boolean expression
Hard constraint demand；
Step 3.3：Soft-constraint demand is set for task, multiple soft-constraints of task are represented by soft-constraint demand chained list
Several elements are included in demand, chained list, each element includes a Boolean expression and a valuation, and Boolean expression shows
Specific soft-constraint demand, lifting of the valuation to quantify to meet the execution efficiency that soft-constraint demand is brought；
Step 3.4：The attribute cluster of task and soft or hard constraint demand are sent to Resource Scheduler, request point by job manager
With machine.
Further, the step 4 specifically includes following steps：
Step 4.1：Received task is designated as " treating scheduler task ", collection of machines M is initialized, all machines is put into
In M, it is sky to initialize alternative machine list；
Step 4.2：A machine is taken out from collection of machines M, " alternative machine " is designated as, according to machine and the letter of task
Breath, calculates the value for obtaining treating scheduler task hard constraint demand；
Step 4.3：Whether the hard constraint demand for judging to treat scheduler task is true, if true, is then added alternative machine
Into alternative machine list, and according to machine information and the soft-constraint chained list of task, calculate the constraint valuation for obtaining alternative machine；
Step 4.4：Alternative machine is removed from collection of machines M, whether be empty, do not go to step then for sky if judging collection of machines M
Rapid 4.2；
Step 4.5：To constrain valuation as standard, the maximum machine of selection constraint valuation, is designated as in alternative machine list
Treat the matching machine of scheduler task.
Further, in the step 4.3, further comprise during the constraint valuation for calculating alternative machine：
The constraint valuation of alternative machine is initialized as 0；
Traversal treats the soft-constraint chained list of scheduler task, for each element, and calculating obtains its soft-constraint demand, if it is soft about
Beam demand is true, then current constraint valuation adds the valuation of the soft-constraint element, finally obtains the constraint valuation of alternative machine.
In order to solve the above technical problems, the invention also provides the task scheduling in a kind of isomeric group towards mixed load
System, including job manager, Resource Scheduler and actuator；
The job manager and Resource Scheduler are deployed on main controlled node, and job manager is used to manage operation and appointed
Business, is that task sets a property cluster and soft or hard constraint demand, and mission bit stream is sent to needed for Resource Scheduler, request task
Machine；
Resource Scheduler is used for the machine heartbeat that receiving actuator timing is sent, and is safeguarding the base of whole clustered machine heartbeat
On plinth, Resource Scheduler can receive the mission bit stream of job manager transmission, be that task matching meets constraint and optimal machine
Device；
The actuator is deployed on other all machines in addition to main controlled node, regularly reports machine to Resource Scheduler
Heartbeat, and the assignment instructions that job manager is issued are received, it is responsible for specific execution task.
The advantage of the present invention compared with prior art is：
(1) method for scheduling task proposed by the present invention and system, are represented by a kind of constraint specification method easily expanded
The machine attribute and mission requirements of isomerization, on this basis, treat hard constraint and soft-constraint with a certain discrimination, regard hard constraint as filtering
Standard, is that task matching meets hard constraint and optimal machine by soft-constraint alternatively standard.The present invention is in task scheduling mistake
Various constraints are considered in journey, the execution efficiency of task and the overall performance of system is significantly improved.
(2) the tasks carrying efficiency under the task scheduling for meeting constraint and the task scheduling strategy for ignoring constraint is tested,
The validity of the method for scheduling task proposed by the present invention for considering constraint is verified with this.Figure 11 have recorded virtual machine application scenarios
Under the task start time, data display meets the task start time of constraint when being significantly shorter than the task start for ignoring constraint
Between, specific speed-up ratio is relevant with mirror image size, is 6.91 to 24.18 in the experiment of this group.Figure 12 have recorded between task
Task completion time under the application scenarios being in communication with each other, data display, the task completion time for meeting constraint is same substantially short
In the task completion time for ignoring constraint, specific speed-up ratio is relevant with data scale, network state, in the experiment of this group about
2.25.In general, method for scheduling task proposed by the present invention and system can handle a variety of restraint conditions, and significantly improve and appoint
Business execution efficiency.
Brief description of the drawings
Fig. 1 is the principle schematic of method for scheduling task and system in the embodiment of the present invention；
Fig. 2 is the flow chart of method for scheduling task in the embodiment of the present invention；
Fig. 3 is the schematic diagram of machine attribute cluster in the embodiment of the present invention；
Fig. 4 is setting task attribute cluster and the flow chart of constraint demand in the embodiment of the present invention；
Fig. 5 is the schematic diagram of task attribute cluster in the embodiment of the present invention；
Fig. 6 is the schematic diagram of task hard constraint in the embodiment of the present invention；
Fig. 7 is the schematic diagram of task soft-constraint chained list in the embodiment of the present invention；
The flow chart that it is task optimal scheme machine in the embodiment of the present invention that Fig. 8, which is,；
Fig. 9 is the schematic diagram of calculating task hard constraint in the embodiment of the present invention；
Figure 10 is the schematic diagram that Computer of embodiment of the present invention device constrains valuation；
Figure 11 is the task start time under virtual machine application scenarios in the embodiment of the present invention；
Figure 12 is the task completion time under task intercommunication application scenarios in the embodiment of the present invention.
Embodiment
The principle and feature of the present invention are described below in conjunction with drawings and examples, example is served only for explaining this
Invention, is not intended to limit the scope of the present invention.
As shown in figure 1, the embodiment of the present invention realizes a task towards mixed load operated on isomeric group
Scheduling system, the system uses typical principal and subordinate (Master-Slave) framework, and main control part (Master) includes two cores
Process job manager (Jobs Manager) and Resource Scheduler (Resource Scheduler), the two is deployed in master control thing
Manage on node.Include a kernel process actuator (Executor) from part (Slave), be deployed in beyond master control physical node
Other all machines on.
Job manager is responsible for operation and task, and an operation includes several tasks, a group job ID and task
ID can uniquely indicate a task.Job manager receives and parses through the operation of user's submission, is task according to job parameter
Set a property cluster and soft or hard constraint, and mission bit stream is sent into Resource Scheduler, asks the machine needed for distribution.In acquisition
After machine, job manager is again by mission dispatching to specified machine, and the execution state of monitor task is simultaneously carried out fault-tolerant.
Resource Scheduler is responsible for machine heartbeat (Heartbeat) information that receiving actuator timing is sent, these heartbeats letter
Breath includes the attribute cluster of machine.On the basis of whole clustered machine attribute cluster is safeguarded, Resource Scheduler can receive operation
The mission bit stream that manager is sent, is that task distribution meets constraint and optimal machine, and by task and the matching relationship of machine
As a result, returning to job scheduler.
Actuator is responsible for receiving the instruction of job manager, starts execution task, to job manager Report Tasks state；
On the other hand, the attribute cluster of the machine is reported Resource Scheduler by actuator timing by heartbeat form.For each task,
Actuator creates a virtualized environment first, and task is then performed inside virtualized environment.
As shown in Fig. 2 in the present embodiment, method for scheduling task may include steps of：
Step 201, Resource Scheduler receives machine heartbeat, the attribute cluster of machine maintenance；
Step 202, job manager receives and parses through operation, obtains several tasks；
Step 203, job manager is that task sets a property cluster and constraint demand, and mission bit stream then is sent into resource
Manager；
Step 204, Resource Scheduler is received after mission bit stream, is that task matching meets constraint and optimal machine, and will
The matching relationship of task and machine returns to job manager；
Step 205, job manager is received after the matching relationship of task and machine, by mission dispatching to matching machine
Actuator on, perform task.
Fig. 3 is the schematic diagram of machine attribute cluster in the embodiment of the present invention.Fig. 3 indicates the attribute cluster of certain machine and corresponding
Value, the attribute cluster of machine is the particular content of machine heartbeat, by actuator timing be sent to Resource Scheduler.Attribute cluster includes
Multiple key assignments (Key-Value) are right, and wherein key represents machine attribute, and value then represents the occurrence of attribute, wherein machine attribute bag
Include machine host name, IP address, machine type, machine architecture, operating system, CPU sums, memory amount, CPU, available can be used
Amount of ram, constraint valuation etc., table 1 lists the attribute cluster of machine.In example shown in Fig. 3, the main frame of the machine is entitled
" Blade10 ", IP address are that " 192.168.1.160 ", machine type are " A ", and machine architecture is " X86_64 ", and Fig. 3 is also in addition
Indicate other attributes of machine.
The attribute cluster of the machine of table 1
| Attribute-name | Explanation | Data type |
| ATTR_MACHINE | Machine host name | string |
| ATTR_IP | IP address | string |
| ATTR_TYPE | Machine type | string |
| ATTR_ARCH | Machine architecture | string |
| ATTR_OS | Operating system | string |
| ATTR_TOTAL_CPU | CPU sums | double |
| ATTR_TOTAL_MEM | Memory amount | int |
| ATTR_AVAIL_CPU | CPU can be used | double |
| ATTR_AVAIL_MEM | Free memory amount | int |
| ATTR_AVG_LOAD | Average load | double |
| CON_VALUE | Constrain valuation | int |
Fig. 4 is setting task attribute cluster and the flow chart of constraint demand in the embodiment of the present invention.Special instruction a bit, is constrained
Demand includes hard constraint and soft-constraint.Hard constraint is the necessary condition of tasks carrying, be must be satisfied in scheduling process, right
Belong to qualitative analysis in the processing of hard constraint.Soft-constraint is the preferences of tasks carrying, and the satisfaction that should try one's best is held with lifting task
Line efficiency, but can ignore if it can not meet, in order to avoid the wasting of resources and the delay of tasks carrying are caused, to soft-constraint
Processing belongs to quantitative analysis.As shown in figure 4, job manager is as follows the step of setting task attribute cluster and constraint demand：
Step 401：The attribute cluster easily expanded is set for task, attribute cluster includes multiple key-value pairs, and wherein key represents task
Attribute, value then represent attribute occurrence, specifically include task sign, perform order, required resource etc.；
Step 402：Hard constraint demand is set for task, the hard constraint of task is represented by a Boolean expression to be needed
Ask；
Further, task there may be multiple hard constraints, multiple hard constraints directly can be done into " with computing ", so still
Multiple hard constraint demands can be so represented by a Boolean expression；
Step 403：Soft-constraint demand is set for task, multiple soft-constraints of task are represented by soft-constraint demand chained list
Several elements are included in demand, chained list, each element includes a Boolean expression and a valuation, wherein Boolean expression
Show specific soft-constraint demand, lifting of the valuation to quantify to meet the execution efficiency that soft-constraint demand is brought.
Fig. 5 is the schematic diagram of task attribute cluster in the embodiment of the present invention.Fig. 5 indicates the attribute cluster of certain task and corresponding
Value, the attribute cluster of task is responsible for setting by job manager.Similar with machine attribute cluster, the attribute cluster of task also includes multiple keys
Value (Key-Value) is right, and wherein key represents task attribute, and value then represents the occurrence of attribute, and wherein task attribute includes operation
ID, task ID, virtualization type, perform order, required CPU, required internal memory etc., table 2 lists attribute cluster and the constraint of task
Demand.In the example embodiment shown in fig. 5, the operation ID of the task is that 1, task ID is 2, and virtualization type is " KVM ", execution order is
" run.sh ", required CPU for 2, needed in save as 2048 (MB).
The attribute cluster and constraint demand of the task of table 2
| Attributes/constraints name | Explanation | Data type |
| ATTR_JOB_ID | Operation ID | int |
| ATTR_TASK_ID | Task ID | int |
| ATTR_VMTYPE | Virtualize type | string |
| ATTR_EXE_PATH | Perform order | string |
| ATTR_NEED_CPU | Required CPU | double |
| ATTR_NEED_MEM | Required internal memory | int |
| HARD_CONSTRAINT | Hard constraint demand | Bool expression formulas |
| SOFT_CON_LIST | Soft-constraint chained list | Chained list |
Fig. 6 is the schematic diagram of task hard constraint in the embodiment of the present invention.Hard constraint is the necessary condition of tasks carrying, at it
Reason result can only meet or be unsatisfactory for two kinds of situations, therefore processing hard constraint belongs to qualitative analysis.One task may be deposited
In multiple hard constraints, we directly can do " with computing " to it.
In example shown in Fig. 6, task has four hard constraints, and each hard constraint can pass through a Boolean expression
To represent.Wherein, " ATTR_AVAIL_CPU>=ATTR_NEED_CPU " represents that the currently available CPU of machine should be more than or equal to
The CPU of required by task, " ATTR_AVAIL_MEM>=ATTR_NEED_MEM " represents that the currently available internal memory of machine should be more than
Equal to the internal memory of required by task, " ATTR_ARCH==X86_64 " represents that the framework of machine should be " X86_64 ", " ATTR_OS
==Centos 6.3 " represents that the operating system of machine should be " Centos6.3 ".The first two in four hard constraints belongs to money
The constraint demand in active layer face, it is ensured that the resource of required by task is included in machine；Latter two constraint for then belonging to non-resource aspect is needed
Ask.Finally, we can be by four hard constraints of task directly with obtaining HARD_CONSTRAINT=(ATTR_AVAIL_
CPU>=ATTR_NEED_CPU) ＆＆ (ATTR_AVAIL_MEM>=ATTR_NEED_MEM) ＆＆ (ATTR_ARCH==X86_
64) ＆＆ (ATTR_OS==Centos6.3), still can so represent the multiple hard of task by a Boolean expression
Constraint demand.
Fig. 7 is the schematic diagram of task soft-constraint chained list in the embodiment of the present invention.Soft-constraint is the preferences of tasks carrying,
It should as far as possible be met, but not be mandatory demand.Processing for soft-constraint not only meets or is unsatisfactory for two kinds of feelings
Condition, should take into full account the satisfaction degree of multiple soft-constraints, and meets each soft-constraint the performance that tasks carrying is brought is carried
Rise, therefore belong to quantitative analysis for the processing of soft-constraint.The present invention represents the multiple of task by soft-constraint demand chained list
Several elements are included in soft-constraint demand, chained list, each element includes a Boolean expression and a valuation, wherein boolean
Expression formula shows specific soft-constraint demand, lifting of the valuation to quantify to meet the execution efficiency that soft-constraint demand is brought.
In example shown in Fig. 7, task has three soft-constraints, and the real needs of first soft-constraint are " ATTR_IP
In (192.168.1.160,192.168.170,192.168.1.180) ", the IP address for showing machine is preferably above three IP
One of address, corresponding valuation is 50, shows that 50 performance boost can be brought by meeting this soft-constraint；The tool of second soft-constraint
Body demand is " ATTR_TYPE==A ", and it is preferably A types to show machine type, and corresponding estimation is 30, shows to meet this soft about
Beam can bring 30 performance boost；The real needs of 3rd soft-constraint are " ATTR_AVG_LOAD<=0.5 ", show machine
Average load more preferably less than be equal to 0.5, corresponding valuation be 20, show that meeting this soft-constraint can bring 20 performance to carry
Rise.
The flow chart that it is task optimal scheme machine in the embodiment of the present invention that Fig. 8, which is,.As shown in figure 8, in the present embodiment, being
Task assignment constraints and optimal machine comprises the following steps：
Step 801：Received task is designated as " treating scheduler task ", collection of machines M is initialized, all machines is put into
In M, it is sky to initialize alternative machine list；
Step 802：A machine is taken out from collection of machines M, " alternative machine " is designated as, according to machine and the letter of task
Breath, calculates the value for obtaining treating scheduler task hard constraint demand；
Step 803：Whether be true, if true, then go to step 804 if judging task hard constraint；If not being true, turn
To step 805；
Step 804：Alternative machine is added in alternative machine list, and according to machine information and the soft-constraint chain of task
Table, calculates the constraint valuation for obtaining alternative machine；
Step 805：Alternative machine is taken out from collection of machines M；
Step 806：Whether be empty, if sky, then go to step 807 if judging collection of machines M；If being not sky, turn
To step 802；
Step 807：To constrain valuation as standard, the maximum machine of selection constraint valuation, is designated as in alternative machine list
Treat the matching machine of scheduler task.
Fig. 9 is the schematic diagram of calculating task hard constraint in the embodiment of the present invention.In the example shown in Fig. 9, the category of machine
Property cluster record each attribute of the machine, including machine main frame entitled " Blade10 ", machine architecture are " x86_64 ", operation
It is 23552MB etc. that system, which is " Centos6.3 ", with CPU can be 13 cores, free memory.Meanwhile, mission bit stream records its resource
Demand is 2 CPU cores, 2GB internal memories, and hard constraint demand is that machine available resources have to be larger than required by task resource, machine architecture
Must be that " X86_64 ", operating system must be " Centos6.3 ", its hard constraint demand can write HARD_CONSTRAINT=
(ATTR_AVAIL_CPU>=ATTR_NEED_CPU) ＆＆ (ATTR_AVAIL_MEM>=ATTR_NEED_MEM) ＆＆ (ATTR_
ARCH==X86_64) ＆＆ (ATTR_OS==Centos6.3).According to machine and task attribute cluster, task can be calculated
The Boolean return of hard constraint demand (HARD_CONSTRAINT) is true, and this shows that the machine meets the hard constraint demand of task.
Figure 10 is the schematic diagram that Computer of embodiment of the present invention device constrains valuation.The present invention is each machine maintenance one
Valuation (CON_VALUE) is constrained, the overall matching degree to quantify machine and task soft-constraint, computing machine constrains valuation
General steps are：First the constraint valuation of initialization machine is 0；Then the soft-constraint chained list of task is traveled through, for each element, meter
Calculation obtains its soft-constraint demand, if soft-constraint demand is true, and constraint valuation is added into the corresponding valuation of the soft-constraint；Finally
Obtained constraint valuation is required.
In the example shown in Figure 10, its IP address is record in machine attribute cluster for " 192.168.1.160 ", machine type
Type is that " B ", average load are 0.3, and it is 0 that initially it, which constrains valuation,.Then the soft-constraint chained list of task is traveled through, for soft-constraint 1,
Soft-constraint demand：ATTR_IP in (192.168.1.160,192.168.170,192.168.1.180) are true, so constraint is estimated
Value can add corresponding valuation 50；For soft-constraint 2, soft-constraint demand (ATTR_TYPE==A) is not true；For soft-constraint
3, soft-constraint demand (ATTR_AVG_LOAD<=0.5) it is true, so constraint valuation can also add corresponding valuation 20, final
It is 70 to constraint valuation.
Figure 11 is the task start time under virtual machine application scenarios in the embodiment of the present invention.In the example shown in Figure 11
In, it have recorded in the case of different virtual machine mirror image size, meet constraint and ignore the task start time of constraint.Wherein,
Solid line represents the task start time for meeting constraint, and dotted line represents the task start time for ignoring constraint.Data display, meets about
The task start time of beam is significantly shorter than the task start time for ignoring constraint, and specific speed-up ratio is relevant with mirror image size,
Speed-up ratio is 6.91 to 24.18 in this group of embodiment.
Figure 12 is the task completion time under task intercommunication application scenarios in the embodiment of the present invention.Shown in Figure 12
Example in, have recorded in the case of different pieces of information scale, meet constraint and ignore the task completion time of constraint.Wherein,
Solid line represents the task completion time for meeting constraint, and dotted line represents the task completion time for ignoring constraint.Data display, meets about
The task completion time of beam is significantly shorter than the task completion time for ignoring constraint, specific speed-up ratio and data scale, network-like
State is relevant, in this embodiment about 2.25.In general, method for scheduling task proposed by the present invention can handle a variety of constraints
Situation, and significantly improve tasks carrying efficiency.
Above example is provided just for the sake of the description purpose of the present invention, and is not intended to limit the scope of the present invention.This
The scope of invention is defined by the following claims.The various equivalent substitutions that do not depart from spirit and principles of the present invention and make and repair
Change, all should cover within the scope of the present invention.
Claims (5)
1. towards the method for scheduling task of mixed load in a kind of isomeric group, it is characterised in that realize that step is as follows：
Step 1, Resource Scheduler receives machine heartbeat, the attribute cluster of machine maintenance；The machine heartbeat is by actuator timing
Attribute cluster that be sent to Resource Scheduler, that heartbeat content is machine；
Step 2, job manager receives and parses through operation, obtains several tasks；
Step 3, job manager is that task sets a property cluster and constraint demand, and mission bit stream then is sent into scheduling of resource
Device；
Step 4, Resource Scheduler is received after mission bit stream, be task matching meet constraint and optimal machine, and by task with
The matching relationship of machine returns to job manager；
Step 5, job manager is received after the matching relationship of task and machine, by the execution in mission dispatching to matching machine
On device, task is performed；
It is as follows that the step 3 implements step：
Step 3.1：The attribute cluster easily expanded is set for task, the attribute cluster is right including multiple key assignments (Key-Value), wherein
Key represents the attribute of task, and value then represents the occurrence of attribute, the attribute cluster of task include task sign, perform order, it is required
Cpu resource, required memory source；
Step 3.2：Hard constraint demand is set for task, the hard constraint demand of task is represented by a Boolean expression, such as
There are multiple hard constraints in fruit, then they are done into " with computing ", remain able to represent by a Boolean expression it is multiple it is hard about
Beam demand；
Step 3.3：Soft-constraint demand is set for task, multiple soft-constraints of task are represented by soft-constraint demand chained list to be needed
Ask, several elements are included in chained list, each element includes a Boolean expression and a valuation, and Boolean expression shows tool
The soft-constraint demand of body, lifting of the valuation to quantify to meet the execution efficiency that soft-constraint demand is brought；
Step 3.4：The attribute cluster of task and soft or hard constraint demand are sent to Resource Scheduler by job manager, ask dispenser
Device；
It is as follows that the step 4 implements step：
Step 4.1：Received task is designated as " treating scheduler task ", collection of machines M is initialized, all machines is put into M,
It is sky to initialize alternative machine list；
Step 4.2：A machine is taken out from collection of machines M, " alternative machine " is designated as, according to machine and the information of task, meter
Calculate the value for obtaining treating scheduler task hard constraint demand；
Step 4.3：Whether the hard constraint demand for judging to treat scheduler task is true, if true, is then added to alternative machine standby
Select in machine list, and according to machine information and the soft-constraint chained list of task, calculate the constraint valuation for obtaining alternative machine；
Step 4.4：Alternative machine is removed from collection of machines M, whether be empty, do not go to step then for sky if judging collection of machines M
4.2；
Step 4.5：To constrain valuation as standard, the maximum machine of selection constraint valuation, is designated as waiting to adjust in alternative machine list
The matching machine of degree task.
2. towards the method for scheduling task of mixed load in isomeric group according to claim 1, it is characterised in that：It is described
In step 1, the attribute cluster of the machine is right including multiple key assignments (Key-Value), and wherein key represents machine attribute, and value is then represented
It is total that the occurrence of attribute, wherein attribute include machine host name, IP address, machine type, machine architecture, operating system, CPU
Number, memory amount, can with CPU, free memory amount, constraint valuation.
3. towards the method for scheduling task of mixed load in isomeric group according to claim 1, it is characterised in that：It is described
Constraint demand in step 3 includes hard constraint and soft-constraint；The hard constraint is the necessary condition of tasks carrying, in scheduling process
In must be satisfied for；The soft-constraint is the preferences of tasks carrying, and the satisfaction that should try one's best is to lift tasks carrying efficiency, no
Crossing can ignore if it can not meet, in order to avoid cause the wasting of resources and the delay of tasks carrying.
4. towards the method for scheduling task of mixed load in isomeric group according to claim 1, it is characterised in that：It is described
In step 4.3, calculating the constraint valuation of alternative machine includes：
The constraint valuation of alternative machine is initialized as 0；
Traversal treats the soft-constraint chained list of scheduler task, for each element, and calculating obtains its soft-constraint demand, if soft-constraint is needed
It is true to ask, then current constraint valuation adds the valuation of the soft-constraint element, finally obtains the constraint valuation of alternative machine.
5. a kind of task scheduling system realized in one of any isomeric groups of claim 1-4 towards mixed load, its
It is characterised by including：Job manager, Resource Scheduler and actuator；The job manager and Resource Scheduler are deployed in master
Control on node, wherein：
Job manager is used to manage operation and task, is that task sets a property cluster and soft or hard constraint demand, and by mission bit stream
It is sent to the machine needed for Resource Scheduler, request task；
Resource Scheduler is used for the machine heartbeat that receiving actuator timing is sent, and is safeguarding the basis of whole clustered machine heartbeat
On, Resource Scheduler can receive the mission bit stream of job manager transmission, be that task matching meets constraint and optimal machine；
The actuator is deployed on other all machines in addition to main controlled node, regularly reports the machine heart to Resource Scheduler
Jump, and receive the assignment instructions that job manager is issued, be responsible for specific execution task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410543294.2A CN104243617B (en) | 2014-10-14 | 2014-10-14 | Towards the method for scheduling task and system of mixed load in a kind of isomeric group |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410543294.2A CN104243617B (en) | 2014-10-14 | 2014-10-14 | Towards the method for scheduling task and system of mixed load in a kind of isomeric group |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104243617A CN104243617A (en) | 2014-12-24 |
| CN104243617B true CN104243617B (en) | 2017-10-27 |
Family
ID=52230945
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410543294.2A Active CN104243617B (en) | 2014-10-14 | 2014-10-14 | Towards the method for scheduling task and system of mixed load in a kind of isomeric group |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104243617B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105022670B (en) * | 2015-07-17 | 2018-03-13 | 中国海洋大学 | Heterogeneous distributed task processing system and its processing method in a kind of cloud computing platform |
| CN105302643B (en) * | 2015-10-14 | 2018-08-24 | 浪潮集团有限公司 | A kind of method and self study scheduler of job scheduling |
| CN105589745A (en) * | 2015-12-18 | 2016-05-18 | 中国科学院软件研究所 | Unbalanced task allocation supported dynamic vulnerability discovery system and method |
| CN107515784B (en) * | 2016-06-16 | 2021-07-06 | 阿里巴巴集团控股有限公司 | Method and equipment for calculating resources in distributed system |
| CN107025141B (en) * | 2017-05-18 | 2020-09-01 | 成都海天数联科技有限公司 | Scheduling method based on big data mixed operation model |
| CN107357661B (en) * | 2017-07-12 | 2020-07-10 | 北京航空航天大学 | Fine-grained GPU resource management method for mixed load |
| CN107678752B (en) * | 2017-08-31 | 2021-09-21 | 北京百度网讯科技有限公司 | Task processing method and device for heterogeneous cluster |
| CN109101339B (en) * | 2018-08-15 | 2019-05-31 | 北京邮电大学 | Video task parallel method, device and Heterogeneous Cluster Environment in isomeric group |
| CN110012062B (en) * | 2019-02-22 | 2022-02-08 | 北京奇艺世纪科技有限公司 | Multi-computer-room task scheduling method and device and storage medium |
| CN111147546B (en) * | 2019-11-29 | 2021-05-14 | 中科院计算技术研究所大数据研究院 | Method and system for processing edge cluster resources |
| WO2021126272A1 (en) * | 2019-12-20 | 2021-06-24 | Hewlett-Packard Development Company, L.P. | Machine learning workload orchestration in heterogeneous clusters |
| CN114168283A (en) * | 2021-12-02 | 2022-03-11 | 北京千帆阅文科技有限公司 | Distributed timed task scheduling method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101271405A (en) * | 2008-05-13 | 2008-09-24 | 武汉理工大学 | Bidirectional grade gridding resource scheduling method based on QoS restriction |
| CN102495758A (en) * | 2011-12-05 | 2012-06-13 | 中南大学 | Scheduling method of real-time tasks in distributing type high performance calculation environment |
| CN103631870A (en) * | 2013-11-06 | 2014-03-12 | 广东电子工业研究院有限公司 | System and method used for large-scale distributed data processing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2995106B1 (en) * | 2012-09-03 | 2015-08-14 | Bull Sas | METHOD AND DEVICE FOR PROCESSING CONTROLS IN A SET OF COMPUTER ELEMENTS |
-
2014
- 2014-10-14 CN CN201410543294.2A patent/CN104243617B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101271405A (en) * | 2008-05-13 | 2008-09-24 | 武汉理工大学 | Bidirectional grade gridding resource scheduling method based on QoS restriction |
| CN102495758A (en) * | 2011-12-05 | 2012-06-13 | 中南大学 | Scheduling method of real-time tasks in distributing type high performance calculation environment |
| CN103631870A (en) * | 2013-11-06 | 2014-03-12 | 广东电子工业研究院有限公司 | System and method used for large-scale distributed data processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104243617A (en) | 2014-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104243617B (en) | Towards the method for scheduling task and system of mixed load in a kind of isomeric group | |
| Hung et al. | Wide-area analytics with multiple resources | |
| Guo et al. | Cloud resource scheduling with deep reinforcement learning and imitation learning | |
| CN103870314B (en) | Method and system for simultaneously operating different types of virtual machines by single node | |
| Lin et al. | Multi-resource scheduling and power simulation for cloud computing | |
| CN106776005B (en) | Resource management system and method for containerized application | |
| US8205208B2 (en) | Scheduling grid jobs using dynamic grid scheduling policy | |
| CN104636197B (en) | A kind of evaluation method of data center's virtual machine (vm) migration scheduling strategy | |
| CN102360309B (en) | Scheduling system and scheduling execution method of multi-core heterogeneous system on chip | |
| CN104050042B (en) | The resource allocation methods and device of ETL operations | |
| CN101697141B (en) | Prediction method of operational performance based on historical data modeling in grid | |
| CN105373432B (en) | A kind of cloud computing resource scheduling method based on virtual resource status predication | |
| US9239734B2 (en) | Scheduling method and system, computing grid, and corresponding computer-program product | |
| US11816509B2 (en) | Workload placement for virtual GPU enabled systems | |
| CN103425536B (en) | A kind of test resource management method of Based on Distributed system performance testing | |
| Malhotra et al. | Study and comparison of CloudSim simulators in the cloud computing | |
| WO2012033909A2 (en) | Method and system of simulating a data center | |
| CN109669774A (en) | Quantization method, method of combination, device and the network equipment of hardware resource | |
| CN108920153A (en) | A kind of Docker container dynamic dispatching method based on load estimation | |
| Singh et al. | Energy based efficient resource scheduling: a step towards green computing | |
| CN110389816A (en) | Method, apparatus and computer program product for scheduling of resource | |
| CN105975340A (en) | Deployment algorithm of application distribution of virtual machine | |
| CN102707995A (en) | Service scheduling method and device based on cloud computing environments | |
| CN109828790A (en) | A kind of data processing method and system based on Shen prestige isomery many-core processor | |
| Aksakalli et al. | Systematic approach for generation of feasible deployment alternatives for microservices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |