CN102609303A - Slow-task dispatching method and slow-task dispatching device of Map Reduce system - Google Patents
Slow-task dispatching method and slow-task dispatching device of Map Reduce system Download PDFInfo
- Publication number
- CN102609303A CN102609303A CN2012100161432A CN201210016143A CN102609303A CN 102609303 A CN102609303 A CN 102609303A CN 2012100161432 A CN2012100161432 A CN 2012100161432A CN 201210016143 A CN201210016143 A CN 201210016143A CN 102609303 A CN102609303 A CN 102609303A
- Authority
- CN
- China
- Prior art keywords
- computing node
- slow
- node
- task
- computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention provides a slow-task dispatching method and a slow-task dispatching device of a Map Reduce system. The slow-task dispatching method includes: respectively acquiring computing power values of computing nodes in the Map Reduce system, arraying the computing nodes into a slow-node array according to the computing power values, selecting last M computing nodes in the slow-node array as target computing nodes, acquiring pre-estimated computing power values after the target computing nodes downloads slow tasks to be dispatched, arraying the target computing nodes and other computing nodes to be a new slow-node array according to the sequence from large to small of the computing power values, presetting the Nth computing node counted from the end to the start of the slow-node array to be an evaluation reference node, and dispatching the slow tasks to be dispatched to the target computing nodes when the pre-estimated computing power values of the target computing nodes are larger than the computing power value of the evaluation referent node, wherein the N is a natural number. By the aid of the slow-task dispatching method, fluctuation during slow-task dispatch is suppressed effectively.
Description
Technical field
The present invention relates to computing technique, relate in particular to the slow method for scheduling task and the device of a kind of MapReduce system.
Background technology
MapReduce is as a kind of basic calculating framework, is widely used in the internet, applications for example cloud computing environment.Development along with cloud computing environment gradually adopts Intel Virtualization Technology, possibly have a plurality of virtual machines on the physical host; Because the computing power of different physical hosts exists than big-difference; And also exist bigger performance difference between each virtual machine on the same physical host; Cause the node isomerism of MapReduce system very outstanding, the processing speed that this node isomerism refers to the various computing node there are differences (virtual machine is equivalent to a computing node).Calculation task is dispatched to different computing nodes; The response time that obtains result is different; Main controlled node in the MapReduce system finds that certain computing node is too slow to the execution speed of calculation task, and when confirming that this calculation task is slow task, is to improve processing speed; Then should slow task scheduling to another computing node, carry out simultaneously, this is called the scheduling of slow task.
Concrete; Wherein a kind of realization mechanism Hadoop of MapReduce system, the scheme of its slow task scheduling is: all computing nodes in the Hadoop supposing the system have defined a process index between 0 and 1 with identical velocity process calculation task to calculation task; And set a fixing threshold value; As long as the process index satisfies imposing a condition of said threshold value, confirm that then this calculation task is slow task, and carry out slow task scheduling.Hadoop carries out slow task scheduling according to the neighbor node dispatching principle, is about to slow task scheduling to the nearest neighbor node of physical transfer.The shortcoming of such scheme is: have too many calculation task to be confirmed as slow task, too much slow task scheduling has taken more system resources; And closing on dispatching principle, also possibly to appear at the neighbor node operation of scheduling back slower, and this slow task also can further be scheduled, and causes slow task by repeatedly scheduling, the shake of system call promptly occurred.
Other a kind of realization mechanism LATE of MapReduce system optimizes above-mentioned Hadoop; It has stipulated a slow task scheduling ratio; For example 10%, have only 10% slow task to be scheduled, to avoid too much slow task to be scheduled and to take more system resources.And LATE also accomplishes according to calculation task and has defined slow node queue required excess time, and the node of stipulating this formation last 25% is slow node, can be with slow task scheduling to described slow node, to avoid the node operation slower.But practice is found; LATE still can not resolution system the jitter problem of scheduling; Even select a computing node that is positioned at outside said last 25% to carry out slow task scheduling, it is too many and slack-off to make still that also this node computing power after this slow task of loading descends, and the slow task of being dispatched remains slow task; To be caused occurring the thrashing phenomenon that slow task is repeatedly dispatched by scheduling once more.
Summary of the invention
First aspect of the present invention provides the slow method for scheduling task of a kind of MapReduce system, the generation of jitter phenomenon during with the slow task scheduling of effective inhibition MapReduce system.
Another aspect of the present invention provides the slow task scheduling apparatus of a kind of MapReduce system, the generation of jitter phenomenon during with the slow task scheduling of effective inhibition MapReduce system.
The slow method for scheduling task of MapReduce provided by the invention system comprises:
Obtain the computing power value of each computing node in the MapReduce system respectively, the computing power value of said computing node
Said v
1... v
mRepresent the processing speed of each calculation task on the said computing node, said
Represent the average treatment speed of the work under said each calculation task respectively, said m representes the total quantity of the calculation task on the said computing node;
According to said computing power value order from big to small said each computing node is arranged as slow node queue, and chooses preceding M computing node in the said slow node queue, each said computing node of choosing is as the target computing node, and said M is a natural number;
Obtain the computing power value that M said target computing node estimated respectively after loading slow task to be scheduled, the said computing power of estimating
Said v
iTreating on the expression target computing node dispatched slow task handling speed, and be said
Treating on the expression target computing node dispatched the average treatment speed of the work under the slow task;
According to the computing power value of estimating of said target computing node and the computing power value of each computing node outside the said target computing node; With the slow node queue of each computing node outside said target computing node and the said target computing node according to computing power value series arrangement Cheng Xin from big to small; And N the computing node that begins forward from tail of the queue in the preset said slow node queue be the metewand node, and said N is a natural number;
The computing power value of estimating at said target computing node is during greater than the computing power value of said metewand node, with said slow task scheduling to be scheduled to said target computing node.
The slow task scheduling apparatus of MapReduce provided by the invention system, comprising: parameter acquiring unit, ability are estimated unit, queue column unit and scheduling processing unit;
Said parameter acquiring unit is used for obtaining respectively the computing power value of each computing node of MapReduce system, the computing power value of said computing node
Said v
1... v
mRepresent the processing speed of each calculation task on the said computing node, said
Represent the average treatment speed of the work under said each calculation task respectively, said m representes the total quantity of the calculation task on the said computing node;
Said ability is estimated the unit, is used for choosing preceding M computing node of the slow node queue that said queue column unit produces, and each said computing node of choosing is as the target computing node, and said M is a natural number; And obtain the computing power value that M said target computing node is estimated, the said computing power of estimating respectively after loading slow task to be scheduled
Said v
iTreating on the expression target computing node dispatched slow task handling speed, and be said
Treating on the expression target computing node dispatched the average treatment speed of the work under the slow task;
Said queue column unit is used for according to said computing power value order from big to small said each computing node being arranged as slow node queue; And; According to the computing power value of estimating of said target computing node and the computing power value of each computing node outside the said target computing node, with the slow node queue of each computing node outside said target computing node and the said target computing node according to computing power value series arrangement Cheng Xin from big to small;
Said scheduling processing unit, being used for preset said slow node queue is the metewand node from N the computing node that tail of the queue begins forward, said N is a natural number; And the computing power value of estimating at said target computing node is during greater than the computing power value of said metewand node, with said slow task scheduling to be scheduled to said target computing node.
The technique effect of the slow method for scheduling task of MapReduce of the present invention system is: through before with slow task scheduling to target computing node; Estimate this target computing node and load slow task to be scheduled computing power value afterwards; And in this computing power value of estimating during greater than the computing power value of metewand node; Just slow task scheduling to be scheduled is arrived said target computing node, it is too many to guarantee that this target computing node can not make that after loading slow task computing power descends, thereby eliminates slow task; Prevented the repeatedly scheduling of slow task, the generation of jitter phenomenon when effectively having suppressed slow task scheduling.
The technique effect of the slow task scheduling apparatus of MapReduce of the present invention system is: through before with slow task scheduling to target computing node; Estimate this target computing node and load slow task to be scheduled computing power value afterwards; And in this computing power value of estimating during greater than the computing power value of metewand node; Just slow task scheduling to be scheduled is arrived said target computing node, it is too many to guarantee that this target computing node can not make that after loading slow task computing power descends, thereby eliminates slow task; Prevented the repeatedly scheduling of slow task, the generation of jitter phenomenon when effectively having suppressed slow task scheduling.
Description of drawings
Fig. 1 is the schematic flow sheet of the slow method for scheduling task embodiment of MapReduce of the present invention system;
Fig. 2 is the structural representation of the slow task scheduling apparatus embodiment of MapReduce of the present invention system.
Embodiment
Clearer for explanation to the slow method for scheduling task of the MapReduce system of the embodiment of the invention, at first the structure and the principle of work of MapReduce system are done simple declaration:
The MapReduce system generally includes a main controlled node (master) and a plurality of computing node (slave); Main controlled node is in charge of computing node.Main controlled node receives the data computation request of client, and the data computation of this request can be called a job (Job), and work can have polytype, and for example, data query work, data are on average worked etc.Main controlled node can be split as a plurality of calculation tasks (task) with work, and calculation task is distributed to each computing node, is specifically carried out the processing of calculation task by each computing node.The processing that the MapReduce system will work is divided into two stages: Map stage and Reduce stage; Be that calculation task comprises two types; The calculation task of the calculation task of Map type and Reduce type; The Map stage mainly is each calculation task that splits to be distributed to each computing node handle, and the Reduce stage then mainly is that the result of calculation with each computing node gathers; When the calculation task of all computing nodes was all finished, main controlled node gathered result of calculation and reports to client.Wherein, in processing procedure, exist heartbeat between computing node and the main controlled node, computing node can be carried at the progress situation of its calculation task and notify main controlled node in the heartbeat message; And when handling calculation task and be in the free time, computing node can be initiatively to main controlled node request Distribution Calculation task.
Embodiments of the invention are followed following assumed condition: suppose that the first, each computing node in the MapReduce system is an isomery; The second, suppose that handled each work of MapReduce system is isomery: obvious difference between the dissimilar work, the data volume of generation is different.
On the basis of above introduction, slow method for scheduling task and the device in the face of the embodiment of the invention describes down:
Fig. 1 need to prove for the schematic flow sheet of the slow method for scheduling task embodiment of MapReduce of the present invention system, and following 101～103 just to the enumerating of each performed in this method action, and the execution sequence between it does not done strict restriction.As shown in Figure 1, this method can comprise:
101, obtain the computing power value of each computing node in the MapReduce system respectively;
In the present embodiment, the main controlled node of MapReduce system can obtain the computing power value of each computing node.Optional, main controlled node can obtain the computing power of each computing node according to following obtain manner:
Each computing node can calculate the processing speed of the calculation task of its processing, and for example, computing node A goes up and handles calculation task a, calculation task b; Computing node B goes up and handles calculation task c, calculation task d; Then computing node A can calculate the processing speed of calculation task a, calculation task b respectively, and computing node B can calculate the processing speed of calculation task c, calculation task d respectively.
Dissimilar calculation tasks, its processing speed has different account forms.For example, suppose that computing node A goes up handled calculation task a, calculation task b is the calculation task of Map type, computing node B goes up handled calculation task c, calculation task d is the calculation task of Reduce type.Then, for the calculation task of Map type, its processing speed v=p/t, wherein, p is the data volume of the current calculation task of finishing dealing with of computing node, said t is that the processing of calculation task of said data volume is consuming time; For the calculation task of Reduce type, its processing speed v=p/t, wherein, because the Reduce operation generally is divided into three phases: copy, ordering and stipulations; If calculation task is in the copy stage; Then
is if calculation task is in phase sorting; Then
if calculation task is in reduction stages, and then
t is that the processing of calculation task is consuming time.
Each computing node can through and main controlled node between heartbeat message, the processing speed of carrying the calculation task that aforementioned calculation obtains reports to main controlled node.Main controlled node can calculate the average treatment speed of the affiliated work of this calculation task according to described processing speed.
For example; Above-mentioned calculation task a, calculation task c are split into by work (job) G1, that is, main controlled node is behind the work G1 that receives the client-requested processing; This G1 is split as calculation task a, calculation task c, and is distributed to described computing node A and computing node B processing; These two computing nodes feed back to main controlled node with the processing speed of the calculation task of handling on it, and main controlled node just can obtain the average treatment speed of work G1 according to this processing speed.The said v of said average treatment speed
representes the processing speed of each calculation task of the split one-tenth of said work, and said n representes total number of the computing node that each calculation task of the split one-tenth of work belongs to; The processing speed of supposing the calculation task a of computing node A feedback is v1; The processing speed of the calculation task c of computing node B feedback is v2; Then work G1 average treatment speed
in like manner; Suppose that calculation task b and calculation task d are split by work G2; The processing speed of calculation task b is v3; The processing speed of calculation task d is v4, the G2 average treatment of then working speed
On the basis of each average treatment speed of working of aforementioned calculation, the computing power of each computing node can be calculated according to following formula:
Wherein, v
1... v
mThe processing speed of each calculation task on the expression target computing node,
Represent the average treatment speed of the work under each calculation task respectively, said m representes the quantity of each calculation task on the said computing node.For example, the computing power of computing node A
Optional, main controlled node can be sent to each computing node with average treatment speed of each work that calculates, obtains oneself computing power value by each computing node, again this computing power is reported to main controlled node; Perhaps, average treatment speed of the processing speed that also can be reported according to computing node by main controlled node and each work of oneself calculating obtains the computing power value of each computing node.
Present embodiment is through above-mentioned technical scheme; The evaluation method and the evaluation index of the computing power of computing node are provided, and, in the calculating of this computing power; The isomerism of dissimilar work and the isomerism of computing node have been considered; That is, considered the processing speed of various computing node, and; Because the work of each type does not have comparability, do the normalization processing so adopted
.Aforesaid way makes the computing power of computing node more reasonably obtain reflection.
102, according to computing power value order from big to small said each computing node is arranged as slow node queue, and chooses preceding M computing node in the said slow node queue, each said computing node of choosing is as the target computing node; And obtain the computing power value that the target computing node is estimated after loading slow task to be scheduled;
Optional, during application calculation task message that main controlled node can certain computing node in receiving the MapReduce system sends, the scheduling that begins to carry out slow task is handled.
Main controlled node can be safeguarded two formations when the computing power of processing speed that gets access to calculation task and computing node, slow task queue and slow node queue.Wherein, slow task queue is that main controlled node is ranked to the calculation task of handling in the MapReduce system according to the processing speed of calculation task; Slow node queue is that main controlled node is ranked to each computing node according to the computing power of computing node, can be according to computing power value order from big to small each computing node to be arranged as slow node queue.And in the present embodiment, main controlled node can begin from the tail of the queue of slow task queue, and getting last 10% calculation task is slow task; In the practical implementation, this ratio value can change, and for example, can measuring task accomplishes the variance of speed, if variance is less, can suitably reduce this ratio.
In the present embodiment, can choose preceding M computing node in the said slow node queue, each said computing node of choosing is as the target computing node, and said M is a natural number; For example, in the practical implementation, can choose preceding two nodes or first three node etc. in the slow node queue, quantity can independently be set, and still selects several nodes forward in the formation as far as possible.Need slow task be dispatched to an above-mentioned M computing node simultaneously, above-mentioned M target computing node is the node that plan loads slow task.Before with slow task scheduling to above-mentioned target computing node, will load slow task computing power afterwards to the target computing node and carry out Pre-Evaluation.
Concrete, the computing power of target computing node
Wherein, said v
iTreating on the expression target computing node dispatched slow task handling speed, and be said
Treating on the expression target computing node dispatched the average treatment speed of the work under the slow task, said v
1... v
mSaid processing speed of waiting to dispatch each calculation task outside the slow task on the expression target computing node, said
Represent said average treatment speed of waiting to dispatch each calculation task work under respectively outside the slow task, said m representes the quantity of waiting to dispatch the calculation task outside the slow task on the said target computing node.
For example; Suppose that plan is loaded into calculation task c on the computing node A, then the computing power of pre-estimation operator node A
103, with the slow node queue of each computing node outside said target computing node and the said target computing node according to computing power value series arrangement Cheng Xin from big to small;
Owing in 102, the computing power of the target computing node in the slow node queue has been carried out estimating again; So in this step; Will be according to the computing power after the estimating of target computing node; And the computing power value of each computing node outside the said target computing node, slow node queue is requeued.
104, the computing power value of estimating at said target computing node is during greater than the computing power value of said metewand node, with said slow task scheduling to be scheduled to said target computing node.
In the present embodiment, presetting N the computing node that begins forward from tail of the queue in the said slow node queue that requeues in 103 is the metewand node, and said N is a natural number; For example, can select the ratio of the sum of the computing node in said N and the said MapReduce system is 10%.That is, supposing has 100 computing nodes in the MapReduce system, and in the slow node queue that these 100 computing nodes are formed, a part of node computing power of formation back is lower, and processing speed is slower, can be called slow node; Then be the metewand node wherein from the 10th computing node of tail of the queue inverse forward.
If the computing power value that said target computing node is estimated is greater than the computing power value of metewand node; Then show if after slow task to be scheduled is loaded into the target computing node; But can descending, the computing power of target computing node can not drop to back 10% of slow node queue; Also show accordingly, because the computing power of target computing node is stronger, therefore; The slow task that is loaded can not become new slow task again on the target computing node, then said slow task scheduling to be scheduled is arrived said target computing node.Otherwise, show if slow task to be scheduled is loaded into the target computing node, with making the computing power of target computing node seriously descend, will inevitably produce slow task, then can be with slow task scheduling to be scheduled to the target computing node.
Through before loading slow task; Computing power to the target computing node that loads is estimated, and can foresee the ability drop situation of this target computing node after loading slow task in advance, can obtain whether still can producing slow task after the loading; And prevention in time can produce the scheduling of slow task; Prevent the phenomenon that slow task is repeatedly dispatched, thereby effectively suppressed the generation of jitter phenomenon, improved the performance of MapReduce system; And,, also reduced the resource occupation amount that slow task scheduling caused owing in time prevented irrational slow task scheduling; In addition, this method through with slow node queue as scheduled basis, the computing power of estimating that loads after the slow task according to node requeues, and also can control the ratio that slow task is dispatched again.
Below through one group of experimental data, the effect of slow method for scheduling task of the MapReduce system of present embodiment is described:
Wherein, present embodiment has carried out emulation experiment, the relatively effect of the slow task scheduling scheme of Hadoop, LATE and present embodiment difference.Simulated environment is set to: there are a main controlled node and 50 computing nodes in the MapReduce system; According to the Hadoop model, each computing node can be handled 10 computing powers simultaneously; The computing power of computing node is chosen according to the distribution of following table 1.Two technical indicators are mainly investigated in this emulation: the number of times (promptly adding up the number through the calculation task of twice above scheduling) of shake takes place in response time that the MapReduce system finishes the work and MapReduce system.
The computing power of table 1 computing node distributes
| The physical machine type | The platform number |
| 1 | VMs/host | 202 |
| 2 | VMs/host | 264 |
| 3 | VMs/host | 201 |
| 4 | VMs/host | 140 |
| 5 | VMs/host | 45 |
| 6 | VMs/host | 12 |
| 7 | VMs/host | 7 |
Following table 2 and table 3 are all shaken the number of times form for the response time contrast form peace of above-mentioned three kinds of models in this emulation experiment, and wherein, Berkeley representes the LATE scheme, and Patent representes the present embodiment scheme;
Table 2 response time (unit: tick) contrast form
| Number of tasks | Hadoop | Berkeley | Patent |
| 1000 | 21749.4 | 19990.2 | 16814.6 |
| 2000 | 38381.2 | 38223.6 | 33608.8 |
| 3000 | 57929.4 | 55130.6 | 51802.3 |
| 4000 | 81571.7 | 73063.4 | 69270.5 |
| 5000 | 99111.3 | 94638.1 | 86405.8 |
Table 3 average jitter number of times form
| Number of tasks | Hadoop | Berkeley | Patent |
| 1000 | 71.8 | 45.7 | 14.1 |
| 2000 | 77.9 | 47.5 | 24 |
| 3000 | 79.8 | 50.4 | 42.6 |
| 4000 | 90.3 | 57.7 | 45.1 |
| 5000 | 94.5 | 62.2 | 49.3 |
Can recognize that from table 2 and table 3 scheme of present embodiment all is being better than Hadoop and LATE aspect deadline and the shake.
The slow method for scheduling task of the MapReduce system of present embodiment; Through before with slow task scheduling to target computing node; Estimate this target computing node and load the computing power value after the slow task to be scheduled, and during greater than the computing power value of metewand node, just slow task scheduling to be scheduled is arrived said target computing node in this computing power value of estimating; It is too many to guarantee that this target computing node can not make that after loading slow task computing power descends; Thereby avoid the generation of new slow task, prevented the repeatedly scheduling of slow task, the generation of jitter phenomenon when effectively having suppressed slow task scheduling.
Fig. 2 is the structural representation of the slow task scheduling apparatus embodiment of MapReduce of the present invention system; This device can be carried out the slow method for scheduling task of any embodiment of the present invention; Present embodiment is only done simple declaration to the structure of this device, and concrete principle of work can combine referring to method embodiment said.
As shown in Figure 2, the slow task scheduling apparatus of present embodiment can comprise: parameter acquiring unit 21, ability are estimated unit 22, queue column unit 23, scheduling processing unit 24; Wherein,
Ability is estimated unit 22, is used for choosing preceding M computing node of the slow node queue that said queue column unit produces, and each said computing node of choosing is as the target computing node, and said M is a natural number; And obtain the computing power value that M said target computing node is estimated, the said computing power of estimating respectively after loading slow task to be scheduled
Said v
iTreating on the expression target computing node dispatched slow task handling speed, and be said
Treating on the expression target computing node dispatched the average treatment speed of the work under the slow task;
Optional, parameter acquiring unit 21 can comprise: speed reception subelement 211, average treatment subelement 212, ability are obtained subelement 213; Wherein,
Speed receives subelement 211, is used for receiving the processing speed of the calculation task that each computing node of said MapReduce system reports, the calculation task of said calculation task on computing node, handling;
Ability is obtained subelement 213, is used for the average treatment speed of said work is sent to said each computing node respectively, and receives the computing power value of said each computing node that the average treatment speed according to said work that said each computing node reports obtains; Perhaps, be used for obtaining the computing power value of said each computing node according to the average treatment speed of said work.
Optional; Speed receives subelement 211; Specifically be used for when the type of calculation task is the Map type, receive the processing speed of the calculation task of the corresponding said Map type that said each computing node calculates, said processing speed v=p/t; Said p is the data volume of the current calculation task of finishing dealing with of said computing node, and said t is that the processing of calculation task of said data volume is consuming time.
Optional; The device of present embodiment can also comprise: scheduling trigger element 25; Be used for before the target computing node that obtains the MapReduce system is loading the computing power value of estimating after the slow task to be scheduled, receive the application calculation task message of the computing node transmission in the said MapReduce system.
The slow task scheduling apparatus of the MapReduce system of present embodiment; Through before with slow task scheduling to target computing node; Estimate this target computing node and load the computing power value after the slow task to be scheduled, and during greater than the computing power value of metewand node, just slow task scheduling to be scheduled is arrived said target computing node in this computing power value of estimating; It is too many to guarantee that this target computing node can not make that after loading slow task computing power descends; Thereby avoid the generation of new slow task, prevented the repeatedly scheduling of slow task, the generation of jitter phenomenon when effectively having suppressed slow task scheduling.
One of ordinary skill in the art will appreciate that: all or part of step that realizes above-mentioned each method embodiment can be accomplished through the relevant hardware of programmed instruction.Aforesaid program can be stored in the computer read/write memory medium.This program the step that comprises above-mentioned each method embodiment when carrying out; And aforesaid storage medium comprises: various media that can be program code stored such as ROM, RAM, magnetic disc or CD.
What should explain at last is: above each embodiment is only in order to explaining technical scheme of the present invention, but not to its restriction; Although the present invention has been carried out detailed explanation with reference to aforementioned each embodiment; Those of ordinary skill in the art is to be understood that: it still can be made amendment to the technical scheme that aforementioned each embodiment put down in writing, perhaps to wherein part or all technical characteristic are equal to replacement; And these are revised or replacement, do not make the scope of the essence disengaging various embodiments of the present invention technical scheme of relevant art scheme.
Claims (8)
1. the slow method for scheduling task of a MapReduce system is characterized in that, comprising:
Obtain the computing power value of each computing node in the MapReduce system respectively, the computing power value of said computing node
Said v
1... v
mRepresent the processing speed of each calculation task on the said computing node, said
Represent the average treatment speed of the work under said each calculation task respectively, said m representes the total quantity of the calculation task on the said computing node;
According to said computing power value order from big to small said each computing node is arranged as slow node queue, and chooses preceding M computing node in the said slow node queue, each said computing node of choosing is as the target computing node, and said M is a natural number;
Obtain the computing power value that M said target computing node estimated respectively after loading slow task to be scheduled, the said computing power of estimating
Said v
iTreating on the expression target computing node dispatched slow task handling speed, and be said
Treating on the expression target computing node dispatched the average treatment speed of the work under the slow task;
According to the computing power value of estimating of said target computing node and the computing power value of each computing node outside the said target computing node; With the slow node queue of each computing node outside said target computing node and the said target computing node according to computing power value series arrangement Cheng Xin from big to small; And N the computing node that begins forward from tail of the queue in the preset said slow node queue be the metewand node, and said N is a natural number;
The computing power value of estimating at said target computing node is during greater than the computing power value of said metewand node, with said slow task scheduling to be scheduled to said target computing node.
2. the slow method for scheduling task of MapReduce according to claim 1 system is characterized in that the said computing power value of obtaining each computing node in the MapReduce system comprises:
Receive the processing speed of the calculation task that each the said computing node in the said MapReduce system reports, the calculation task of said calculation task on said computing node, handling;
Obtain the average treatment speed of the work under the said calculation task according to said processing speed; The said v of said average treatment speed
representes the said processing speed of each calculation task that said work splits into, and said n representes total number of the computing node that each calculation task of the split one-tenth of said work belongs to;
The average treatment speed of said work is sent to said each computing node respectively, and receives the computing power value of said each computing node that the average treatment speed according to said work that said each computing node reports obtains.
3. the slow method for scheduling task of MapReduce according to claim 1 system is characterized in that the said computing power value of obtaining each computing node in the MapReduce system comprises:
Receive the processing speed of the calculation task that each the said computing node in the said MapReduce system reports, the calculation task of said calculation task on said computing node, handling;
Obtain the average treatment speed of the work under the said calculation task according to said processing speed; The said v of said average treatment speed
representes the said processing speed of each said calculation task of the split one-tenth of said work, and said n representes total number of the computing node that each calculation task of the split one-tenth of said work belongs to;
Obtain the computing power value of said each computing node according to the average treatment speed of said work.
4. according to the slow method for scheduling task of the arbitrary described MapReduce of claim 1-3 system, it is characterized in that, obtain before the target computing node loading the computing power value of estimating after the slow task to be scheduled, also comprise said:
Receive the application calculation task message that the computing node in the said MapReduce system sends.
5. according to the slow method for scheduling task of the arbitrary described MapReduce of claim 1-4 system, it is characterized in that the ratio of the sum of the computing node in said N and the said MapReduce system is 10%.
6. the slow task scheduling apparatus of a MapReduce system is characterized in that, comprising: parameter acquiring unit, ability are estimated unit, queue column unit and scheduling processing unit;
Said parameter acquiring unit is used for obtaining respectively the computing power value of each computing node of MapReduce system, the computing power value of said computing node
Said v
1... v
mRepresent the processing speed of each calculation task on the said computing node, said
Represent the average treatment speed of the work under said each calculation task respectively, said m representes the total quantity of the calculation task on the said computing node;
Said ability is estimated the unit, is used for choosing preceding M computing node of the slow node queue that said queue column unit produces, and each said computing node of choosing is as the target computing node, and said M is a natural number; And obtain the computing power value that M said target computing node is estimated respectively after loading slow task to be scheduled; The said vi of the said computing power of estimating
representes that treating on the target computing node dispatch slow task handling speed, and treating on said
expression target computing node dispatched the average treatment speed of the work under the slow task;
Said queue column unit is used for according to said computing power value order from big to small said each computing node being arranged as slow node queue; And; According to the computing power value of estimating of said target computing node and the computing power value of each computing node outside the said target computing node, with the slow node queue of each computing node outside said target computing node and the said target computing node according to computing power value series arrangement Cheng Xin from big to small;
Said scheduling processing unit, being used for preset said slow node queue is the metewand node from N the computing node that tail of the queue begins forward, said N is a natural number; And the computing power value of estimating at said target computing node is during greater than the computing power value of said metewand node, with said slow task scheduling to be scheduled to said target computing node.
7. the slow task scheduling apparatus of MapReduce according to claim 6 system is characterized in that said parameter acquiring unit comprises:
Speed receives subelement, is used for receiving the processing speed of the calculation task that each said computing node of said MapReduce system reports, the calculation task of said calculation task on computing node, handling;
The average treatment subelement; The average treatment speed that is used for the work under the said calculation task that obtains according to said processing speed; The said v of said average treatment speed
representes the said processing speed of each said calculation task of the split one-tenth of said work, and said n representes total number of the computing node that each calculation task of the split one-tenth of said work belongs to;
Ability is obtained subelement, is used for the average treatment speed of said work is sent to said each computing node respectively, and receives the computing power value of said each computing node that the average treatment speed according to said work that said each computing node reports obtains; Perhaps, be used for obtaining the computing power value of said each computing node according to the average treatment speed of said work.
8. according to the slow task scheduling apparatus of claim 6 or 7 described MapReduce systems, it is characterized in that, also comprise:
The scheduling trigger element was used for before the target computing node of the said MapReduce of obtaining system is loading the computing power value of estimating after the slow task to be scheduled, received the application calculation task message of the computing node transmission in the said MapReduce system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210016143.2A CN102609303B (en) | 2012-01-18 | 2012-01-18 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210016143.2A CN102609303B (en) | 2012-01-18 | 2012-01-18 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102609303A true CN102609303A (en) | 2012-07-25 |
| CN102609303B CN102609303B (en) | 2014-12-17 |
Family
ID=46526697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210016143.2A Active CN102609303B (en) | 2012-01-18 | 2012-01-18 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102609303B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309738A (en) * | 2013-05-31 | 2013-09-18 | 中国联合网络通信集团有限公司 | User job scheduling method and device |
| CN103699433A (en) * | 2013-12-18 | 2014-04-02 | 中国科学院计算技术研究所 | Method and system for performing dynamic adjustment on number of tasks in Hadoop platform |
| CN103761146A (en) * | 2014-01-06 | 2014-04-30 | 浪潮电子信息产业股份有限公司 | Method for dynamically setting quantities of slots for MapReduce |
| CN104102794A (en) * | 2014-08-06 | 2014-10-15 | 浪潮电子信息产业股份有限公司 | Optimizing method of HADOOP scheduling method |
| CN104243531A (en) * | 2013-06-21 | 2014-12-24 | 华为技术有限公司 | Data processing method, device and system |
| CN104765648A (en) * | 2015-04-30 | 2015-07-08 | 北京奇艺世纪科技有限公司 | Problem node detection method and device based on real-time computing system |
| CN105808346A (en) * | 2014-12-30 | 2016-07-27 | 华为技术有限公司 | Task scheduling method and device |
| CN106357813A (en) * | 2016-11-02 | 2017-01-25 | 龙存科技（北京）股份有限公司 | Task rescheduling method applied in shared-file system |
| WO2017041674A1 (en) * | 2015-09-10 | 2017-03-16 | 阿里巴巴集团控股有限公司 | Method, device and electronic equipment for starting backup task |
| CN111835854A (en) * | 2020-07-17 | 2020-10-27 | 北京航空航天大学 | Slow task prediction method based on grey prediction algorithm |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101005689A (en) * | 2006-06-23 | 2007-07-25 | 华为技术有限公司 | Setting method for dispatching priority node of customer terminal and base station |
| CN101883113A (en) * | 2010-06-25 | 2010-11-10 | 中兴通讯股份有限公司 | Method and physical nodes for realizing overlay network load balance |
| CN102004670A (en) * | 2009-12-17 | 2011-04-06 | 华中科技大学 | Self-adaptive job scheduling method based on MapReduce |
| US20110093852A1 (en) * | 2009-10-21 | 2011-04-21 | Sap Ag | Calibration of resource allocation during parallel processing |
-
2012
- 2012-01-18 CN CN201210016143.2A patent/CN102609303B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101005689A (en) * | 2006-06-23 | 2007-07-25 | 华为技术有限公司 | Setting method for dispatching priority node of customer terminal and base station |
| US20110093852A1 (en) * | 2009-10-21 | 2011-04-21 | Sap Ag | Calibration of resource allocation during parallel processing |
| CN102004670A (en) * | 2009-12-17 | 2011-04-06 | 华中科技大学 | Self-adaptive job scheduling method based on MapReduce |
| CN101883113A (en) * | 2010-06-25 | 2010-11-10 | 中兴通讯股份有限公司 | Method and physical nodes for realizing overlay network load balance |
Non-Patent Citations (1)
| Title |
|---|
| 陈全: "异构环境下Map-Reduce调度算法的研究", 《中国优秀硕士学位论文全文数据库信息科技辑I138-6》 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309738B (en) * | 2013-05-31 | 2016-12-28 | 中国联合网络通信集团有限公司 | User job dispatching method and device |
| CN103309738A (en) * | 2013-05-31 | 2013-09-18 | 中国联合网络通信集团有限公司 | User job scheduling method and device |
| CN104243531A (en) * | 2013-06-21 | 2014-12-24 | 华为技术有限公司 | Data processing method, device and system |
| CN103699433A (en) * | 2013-12-18 | 2014-04-02 | 中国科学院计算技术研究所 | Method and system for performing dynamic adjustment on number of tasks in Hadoop platform |
| CN103761146A (en) * | 2014-01-06 | 2014-04-30 | 浪潮电子信息产业股份有限公司 | Method for dynamically setting quantities of slots for MapReduce |
| CN103761146B (en) * | 2014-01-06 | 2017-10-31 | 浪潮电子信息产业股份有限公司 | A kind of method that MapReduce dynamically sets slots quantity |
| CN104102794A (en) * | 2014-08-06 | 2014-10-15 | 浪潮电子信息产业股份有限公司 | Optimizing method of HADOOP scheduling method |
| CN105808346A (en) * | 2014-12-30 | 2016-07-27 | 华为技术有限公司 | Task scheduling method and device |
| CN104765648A (en) * | 2015-04-30 | 2015-07-08 | 北京奇艺世纪科技有限公司 | Problem node detection method and device based on real-time computing system |
| CN104765648B (en) * | 2015-04-30 | 2017-12-08 | 北京奇艺世纪科技有限公司 | The problem of one kind is based on real time computation system nodal test method and device |
| WO2017041674A1 (en) * | 2015-09-10 | 2017-03-16 | 阿里巴巴集团控股有限公司 | Method, device and electronic equipment for starting backup task |
| CN106357813A (en) * | 2016-11-02 | 2017-01-25 | 龙存科技（北京）股份有限公司 | Task rescheduling method applied in shared-file system |
| CN106357813B (en) * | 2016-11-02 | 2019-08-06 | 龙存科技（北京）股份有限公司 | A kind of task applied to shared-file system dispatching method again |
| CN111835854A (en) * | 2020-07-17 | 2020-10-27 | 北京航空航天大学 | Slow task prediction method based on grey prediction algorithm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102609303B (en) | 2014-12-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102609303B (en) | Slow-task dispatching method and slow-task dispatching device of Map Reduce system | |
| JP6898496B2 (en) | Computation graph processing | |
| AlEbrahim et al. | Task scheduling for heterogeneous computing systems | |
| Taheri et al. | A bee colony based optimization approach for simultaneous job scheduling and data replication in grid environments | |
| US20140019987A1 (en) | Scheduling map and reduce tasks for jobs execution according to performance goals | |
| WO2012154177A1 (en) | Varying a characteristic of a job profile relating to map and reduce tasks according to a data size | |
| JP2020004370A (en) | Systems and methods for scheduling set of non-preemptive tasks in multi-robot environment | |
| Wang et al. | Design and implementation of an analytical framework for interference aware job scheduling on apache spark platform | |
| Wang et al. | Task scheduling for MapReduce in heterogeneous networks | |
| US20220300323A1 (en) | Job Scheduling Method and Job Scheduling Apparatus | |
| CN109032769B (en) | Container-based continuous integrated CI (CI) task processing method and device | |
| US20210390405A1 (en) | Microservice-based training systems in heterogeneous graphic processor unit (gpu) cluster and operating method thereof | |
| Shah et al. | Dynamic multilevel hybrid scheduling algorithms for grid computing | |
| CN103970612A (en) | Load balancing method and device based on pre-division of virtual machine | |
| Fotohi et al. | A cluster based job scheduling algorithm for grid computing | |
| Djebbar et al. | Optimization of tasks scheduling by an efficacy data placement and replication in cloud computing | |
| CN108108242B (en) | Storage layer intelligent distribution control method based on big data | |
| EP3200083B1 (en) | Resource scheduling method and related apparatus | |
| Hu et al. | Requirement-aware strategies with arbitrary processor release times for scheduling multiple divisible loads | |
| CN110073321A (en) | A kind of storage control and I/O request processing method | |
| Geetha et al. | Hadoop scheduler with deadline constraint | |
| CN116501828B (en) | Non-perception vector query method and system for server based on unstructured data set | |
| US10402514B2 (en) | Modeling and simulation of distributed computing frameworks | |
| Singh | Hybrid metaheuristic based scheduling with job duplication for cloud data centers | |
| Kopanski et al. | Plan-based job scheduling for supercomputers with shared burst buffers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |