CN102609303B - Slow-task dispatching method and slow-task dispatching device of Map Reduce system - Google Patents
Slow-task dispatching method and slow-task dispatching device of Map Reduce system Download PDFInfo
- Publication number
- CN102609303B CN102609303B CN201210016143.2A CN201210016143A CN102609303B CN 102609303 B CN102609303 B CN 102609303B CN 201210016143 A CN201210016143 A CN 201210016143A CN 102609303 B CN102609303 B CN 102609303B
- Authority
- CN
- China
- Prior art keywords
- computing
- node
- slow
- task
- power value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention provides a slow-task dispatching method and a slow-task dispatching device of a Map Reduce system. The slow-task dispatching method includes: respectively acquiring computing power values of computing nodes in the Map Reduce system, arraying the computing nodes into a slow-node array according to the computing power values, selecting last M computing nodes in the slow-node array as target computing nodes, acquiring pre-estimated computing power values after the target computing nodes downloads slow tasks to be dispatched, arraying the target computing nodes and other computing nodes to be a new slow-node array according to the sequence from large to small of the computing power values, presetting the Nth computing node counted from the end to the start of the slow-node array to be an evaluation reference node, and dispatching the slow tasks to be dispatched to the target computing nodes when the pre-estimated computing power values of the target computing nodes are larger than the computing power value of the evaluation referent node, wherein the N is a natural number. By the aid of the slow-task dispatching method, fluctuation during slow-task dispatch is suppressed effectively.
Description
Technical field
The present invention relates to computing technique, particularly relate to a kind of slow method for scheduling task and device of Map Reduce system.
Background technology
MapReduce, as a kind of basic calculating framework, is widely used in such as cloud computing environment in internet, applications.Along with the development of cloud computing environment, gradually adopt Intel Virtualization Technology, a physical host may exist multiple virtual machine; Because the computing power of different physical host exists larger difference, and also there is larger performance difference between each virtual machine on same physical host, cause the node isomerism of Map Reduce system very outstanding, the processing speed that this node isomerism refers to different computing node there are differences (virtual machine is equivalent to a computing node).Calculation task is dispatched to different computing nodes, the response time obtaining result is different, when the main controlled node in Map Reduce system finds that certain computing node is too slow to the execution speed of calculation task, and when determining that this calculation task is slow task, for improving processing speed, then this slow task scheduling performed to another computing node, this is called the scheduling of slow task simultaneously.
Concrete, wherein a kind of realization mechanism Hadoop of Map Reduce system, the scheme of its slow task scheduling is: all computing nodes in Hadoop supposing the system are with identical velocity process calculation task, a process index between 0 and 1 is defined to calculation task, and set a fixing threshold value, as long as process index meets imposing a condition of described threshold value, then determine that this calculation task is slow task, and perform slow task scheduling.Hadoop carries out slow task scheduling according to neighbor node dispatching principle, by slow task scheduling on the nearest neighbor node of physical transfer.The shortcoming of such scheme is: have too many calculation task to be confirmed as slow task, and too much slow task scheduling occupies more system resource; Further, close on dispatching principle and also may appear at the rear neighbor node operation of scheduling more slowly, this slow task also can be scheduled further, causes slow task repeatedly to be dispatched, and has namely occurred the shake of system call.
Another realization mechanism LATE of Map Reduce system is optimized above-mentioned Hadoop, which specify a slow task scheduling ratio, such as 10%, only have the slow task of 10% to be scheduled, take more system resource to avoid too much slow task to be scheduled.Further, LATE also completes according to calculation task and defines slow node queue required excess time, specifies that the node of this queue last 25% is slow node, can not by slow task scheduling to described slow node, slower to avoid node to run.But practice finds, LATE still can not resolution system scheduling jitter problem, even if select a computing node be positioned at outside described last 25% to carry out slow task scheduling, also this node computing power after this slow task of loading is still likely made to decline too many and slack-off, the slow task dispatched remains slow task, again will be dispatched, caused the thrashing phenomenon occurring that slow task is repeatedly dispatched.
Summary of the invention
First aspect of the present invention is to provide a kind of slow method for scheduling task of Map Reduce system, the generation of jitter phenomenon during effectively to suppress Map Reduce system slow task scheduling.
Another aspect of the present invention is to provide a kind of slow task scheduling apparatus of Map Reduce system, the generation of jitter phenomenon during effectively to suppress Map Reduce system slow task scheduling.
The slow method for scheduling task of Map Reduce system provided by the invention, comprising:
Obtain the computing power value of each computing node in Map Reduce system respectively, the computing power value of described computing node
described v
1... v
mrepresent the processing speed of each calculation task on described computing node, described in
represent the average treatment speed of the work belonging to described each calculation task respectively, described m represents the total quantity of the calculation task on described computing node;
According to described computing power value order from big to small, described each computing node is arranged as slow node queue, and chooses front M computing node in described slow node queue, each described computing node chosen is as target computing nodes, and described M is natural number;
Obtain the computing power value that M described target computing nodes is estimated after loading slow task to be scheduled respectively, described in the computing power estimated
described v
irepresent and target computing nodes waits the processing speed of dispatching slow task, described in
represent and target computing nodes waits the average treatment speed of dispatching the work belonging to slow task;
According to the computing power value of each computing node outside the computing power value estimated of described target computing nodes and described target computing nodes, each computing node outside described target computing nodes and described target computing nodes is arranged in new slow node queue according to computing power value order from big to small, and the N number of computing node preset in described slow node queue from tail of the queue is forward metewand node, described N is natural number;
When the computing power value that described target computing nodes is estimated is greater than the computing power value of described metewand node, by described slow task scheduling to be scheduled to described target computing nodes.
The slow task scheduling apparatus of Map Reduce system provided by the invention, comprising: parameter acquiring unit, ability estimate unit, queue column unit and scheduling processing unit;
Described parameter acquiring unit, for obtaining the computing power value of each computing node in Map Reduce system respectively, the computing power value of described computing node
described v
1... v
mrepresent the processing speed of each calculation task on described computing node, described in
represent the average treatment speed of the work belonging to described each calculation task respectively, described m represents the total quantity of the calculation task on described computing node;
Described ability estimates unit, for choose described queue column unit produce slow node queue in before M computing node, each described computing node chosen is as target computing nodes, and described M is natural number; And obtain the computing power value that M described target computing nodes estimates after loading slow task to be scheduled respectively, described in the computing power estimated
described v
irepresent and target computing nodes waits the processing speed of dispatching slow task, described in
represent and target computing nodes waits the average treatment speed of dispatching the work belonging to slow task;
Described queue column unit, for being arranged as slow node queue according to described computing power value order from big to small by described each computing node; And, according to the computing power value of each computing node outside the computing power value estimated of described target computing nodes and described target computing nodes, each computing node outside described target computing nodes and described target computing nodes is arranged in new slow node queue according to computing power value order from big to small;
Described scheduling processing unit, be metewand node for the N number of computing node preset in described slow node queue from tail of the queue forward, described N is natural number; And when the computing power value that described target computing nodes is estimated is greater than the computing power value of described metewand node, by described slow task scheduling to be scheduled to described target computing nodes.
The technique effect of the slow method for scheduling task of Map Reduce system of the present invention is: by before by slow task scheduling to target computing nodes, estimate the computing power value after this target computing nodes loading slow task to be scheduled, and when the computing power value that this is estimated is greater than the computing power value of metewand node, just by slow task scheduling to be scheduled to described target computing nodes, can ensure that this target computing nodes can not make computing power decline after the slow task of loading too many, thus eliminate slow task, prevent the repeatedly scheduling of slow task, effectively inhibit the generation of jitter phenomenon during slow task scheduling.
The technique effect of the slow task scheduling apparatus of Map Reduce system of the present invention is: by before by slow task scheduling to target computing nodes, estimate the computing power value after this target computing nodes loading slow task to be scheduled, and when the computing power value that this is estimated is greater than the computing power value of metewand node, just by slow task scheduling to be scheduled to described target computing nodes, can ensure that this target computing nodes can not make computing power decline after the slow task of loading too many, thus eliminate slow task, prevent the repeatedly scheduling of slow task, effectively inhibit the generation of jitter phenomenon during slow task scheduling.
Accompanying drawing explanation
Fig. 1 is the schematic flow sheet of the slow method for scheduling task embodiment of Map Reduce system of the present invention;
Fig. 2 is the structural representation of the slow task scheduling apparatus embodiment of Map Reduce system of the present invention.
Embodiment
In order to the slow method for scheduling task of the Map Reduce system to the embodiment of the present invention explanation clearly, first simple declaration is done to the structure of Map Reduce system and principle of work:
Map Reduce system generally includes a main controlled node (master) and multiple computing node (slave); Main controlled node is in charge of computing node.Main controlled node receives the data computation requests of client, and the data of this request calculate and can be called a job (Job), and work can have polytype, such as, and data query work, data average operation etc.Work can be split as multiple calculation task (task) by main controlled node, and calculation task is distributed to each computing node, is specifically performed the process of calculation task by each computing node.The process of work is divided into two stages by Map Reduce system: Map stage and Reduce stage, namely calculation task comprises two types, the calculation task of Map type and the calculation task of Reduce type, the each calculation task split mainly is distributed to each computing node process by the Map stage, and the result of calculation of each computing node then mainly gathers by the Reduce stage; When the calculation task of all computing nodes is all finished, result of calculation gathers and is reported to client by main controlled node.Wherein, in processing procedure, there is heartbeat between computing node and main controlled node, the progress situation of its calculation task can be carried in heartbeat message and notify main controlled node by computing node; And when processing calculation task and being in the free time, computing node can initiatively to main controlled node request dispatching calculation task.
Embodiments of the invention follow following assumed condition: the first, suppose that each computing node in Map Reduce system is isomery; The second, suppose that each work handled by Map Reduce system is isomery: between dissimilar work, difference is obvious, the data volume of generation is different.
On the above basis introduced, below the slow method for scheduling task of the embodiment of the present invention and device are described:
Fig. 1 is the schematic flow sheet of the slow method for scheduling task embodiment of Map Reduce system of the present invention, it should be noted that, following 101 ~ 103 enumerating just to each action performed in the method, do not do strict restriction to the execution sequence between it.As shown in Figure 1, the method can comprise:
101, the computing power value of each computing node in Map Reduce system is obtained respectively;
In the present embodiment, the main controlled node of Map Reduce system can obtain the computing power value of each computing node.Optionally, main controlled node can obtain the computing power of each computing node according to following obtain manner:
Each computing node can calculate the processing speed of the calculation task of its process, such as, computing node A processes calculation task a, calculation task b; Computing node B processes calculation task c, calculation task d; Then computing node A can calculate the processing speed of calculation task a, calculation task b respectively, and computing node B can calculate the processing speed of calculation task c, calculation task d respectively.
Dissimilar calculation task, its processing speed has different account forms.Such as, suppose that calculation task a, calculation task b handled on computing node A are the calculation tasks of Map type, calculation task c, calculation task d handled on computing node B are the calculation tasks of Reduce type.Then, for the calculation task of Map type, its processing speed v=p/t, wherein, p is the data volume of the calculation task that the current process of computing node completes, and described t is that the process of the calculation task of described data volume is consuming time; For the calculation task of Reduce type, its processing speed v=p/t, wherein, because Reduce operation is generally divided into three phases: copy, sequence and stipulations; If calculation task is in the copy stage, then
if calculation task is in phase sorting, then
if calculation task is in reduction stages, then
t is that the process of calculation task is consuming time.
Each computing node can by the heartbeat message between main controlled node, and the processing speed of carrying the above-mentioned calculation task calculated reports to main controlled node.Main controlled node according to described processing speed, can calculate the average treatment speed of the work belonging to this calculation task.
Such as, above-mentioned calculation task a, calculation task c are split into by work (job) G1, namely, main controlled node is after receiving the work G1 of client-requested process, this G1 is split as calculation task a, calculation task c, and is distributed to described computing node A and computing node B process; The processing speed of the calculation task that it processes is fed back to main controlled node by these two computing nodes, and main controlled node just can obtain the average treatment speed of work G1 according to this processing speed.Described average treatment speed
described v represents the processing speed of each calculation task of the split one-tenth of described work, and described n represents total number of the computing node at each calculation task place of the split one-tenth of work; The processing speed supposing the calculation task a that computing node A feeds back is v1, and the processing speed of the calculation task c that computing node B feeds back is v2, then the average treatment speed of the G1 that works
in like manner, suppose that calculation task b and calculation task d is split by work G2, the processing speed of calculation task b is v3, and the processing speed of calculation task d is v4, then the G2 average treatment that works speed
On the basis of the average treatment speed of each work of above-mentioned calculating, the computing power of each computing node can according to following formulae discovery:
wherein, v
1... v
mrepresent the processing speed of each calculation task on target computing nodes,
represent the average treatment speed of the work belonging to each calculation task respectively, described m represents the quantity of each calculation task on described computing node.Such as, the computing power of computing node A
Optionally, the average treatment speed of each work calculated can be sent to each computing node by main controlled node, is obtained the computing power value of oneself, then this computing power is reported to main controlled node by each computing node; Or the processing speed that also can be reported according to computing node by main controlled node and the average treatment speed of each work oneself calculated, obtain the computing power value of each computing node.
The present embodiment is by above-mentioned technical scheme, provide evaluation method and the evaluation index of the computing power of computing node, and, in the calculating of this computing power, consider the isomerism of dissimilar work and the isomerism of computing node, namely, consider the processing speed of different computing node, and, because the work of each type does not have comparability, so have employed
do normalized.Aforesaid way makes the computing power of computing node more reasonably be reflected.
102, according to computing power value order from big to small, described each computing node is arranged as slow node queue, and chooses front M computing node in described slow node queue, each described computing node chosen is as target computing nodes; And obtain target computing nodes loading the computing power value estimated after slow task to be scheduled;
Optionally, during the application calculation task message that main controlled node can send at certain computing node received in Map Reduce system, the dispatch deal carrying out slow task is started.
Main controlled node, when getting the computing power of the processing speed of calculation task and computing node, can safeguard two queues, slow task queue and slow node queue.Wherein, slow task queue is that main controlled node is ranked to the calculation task processed in Map Reduce system according to the processing speed of calculation task; Slow node queue is that main controlled node is ranked to each computing node according to the computing power of computing node, can be, according to computing power value order from big to small, each computing node is arranged as slow node queue.Further, in the present embodiment, main controlled node can from the tail of the queue of slow task queue, and the calculation task getting last 10% is slow task; In concrete enforcement, this ratio value can change, and such as, task of can measuring completes the variance of speed, if variance is less, can suitably reduce this ratio.
In the present embodiment, can choose front M computing node in described slow node queue, each described computing node chosen is as target computing nodes, and described M is natural number; Such as, concrete to implement, can choose the first two node in slow node queue or first three node etc., quantity can independently set, but as far as possible forward in selection queue several nodes.Need slow task to be dispatched to an above-mentioned M computing node, M above-mentioned target computing nodes is the node that plan loads slow task simultaneously.Before by slow task scheduling to above-mentioned target computing nodes, the computing power loaded target computing nodes after slow task is carried out Pre-Evaluation.
Concrete, the computing power of target computing nodes
wherein, described v
irepresent and target computing nodes waits the processing speed of dispatching slow task, described in
represent and target computing nodes waits the average treatment speed of dispatching the work belonging to slow task, described v
1... v
mrepresent the processing speed waiting each calculation task dispatched outside slow task described on target computing nodes, described in
wait the average treatment speed of the work belonging to each calculation task difference of dispatching outside slow task described in expression, described m represents that treating on described target computing nodes dispatches the quantity of the calculation task outside slow task.
Such as, suppose that calculation task c is loaded on computing node A by plan, then the computing power of pre-estimation operator node A
103, each computing node outside described target computing nodes and described target computing nodes is arranged in new slow node queue according to computing power value order from big to small;
Owing to having carried out again estimating to the computing power of the target computing nodes in slow node queue in 102, so in this step, by according to the computing power after the estimating of target computing nodes, and the computing power value of each computing node outside described target computing nodes, slow node queue is requeued.
104, when the computing power value that described target computing nodes is estimated is greater than the computing power value of described metewand node, by described slow task scheduling to be scheduled to described target computing nodes.
In the present embodiment, the N number of computing node preset in 103 in the described slow node queue requeued from tail of the queue is forward metewand node, and described N is natural number; Such as, the ratio of the sum of the computing node in described N and described Map Reduce system can be selected to be 10%.That is, suppose there are 100 computing nodes in Map Reduce system, in the slow node queue that these 100 computing nodes form, a part of node calculate ability of queue back is lower, and processing speed is comparatively slow, can be called slow node; Wherein from tail of the queue inverse the 10th computing node be forward then metewand node.
If the computing power value that described target computing nodes is estimated is greater than the computing power value of metewand node, then show if after slow task to be scheduled is loaded into target computing nodes, but the computing power of target computing nodes can decline can not drop to rear 10% of slow node queue, also show accordingly, because the computing power of target computing nodes is stronger, therefore, the slow task loaded can not become new slow task again on target computing nodes, then by described slow task scheduling to be scheduled to described target computing nodes.Otherwise, show, if slow task to be scheduled is loaded into target computing nodes, will the computing power degradation of target computing nodes be made, slow task will inevitably be produced, then can not by slow task scheduling to be scheduled to target computing nodes.
By before the slow task of loading, the computing power of the target computing nodes loaded is estimated, the ability decline situation of this target computing nodes after the slow task of loading can be predicted in advance, whether still slow task can be produced after can obtaining loading, and prevention in time can produce the scheduling of slow task, prevent the phenomenon that slow task is repeatedly dispatched, thus effectively inhibit the generation of jitter phenomenon, improve the performance of Map Reduce system; Further, owing to having prevented irrational slow task scheduling in time, the resource occupation amount that slow task scheduling causes has been also reduced; In addition, the method by using slow node queue as scheduled basis, requeue according to the computing power of estimating after node loads slow task, also can control the ratio that slow task is dispatched again.
Below by one group of experimental data, the effect of the slow method for scheduling task of the Map Reduce system of the present embodiment is described:
Wherein, the present embodiment has carried out emulation experiment, compares the effect difference of the slow task scheduling approach of Hadoop, LATE and the present embodiment.Simulated environment is set to: Map Reduce system has a main controlled node and 50 computing nodes; According to Hadoop model, each computing node can process 10 computing powers simultaneously; The computing power of computing node is chosen according to the distribution of following table 1.This emulation paper examines two technical indicators: the number of times (namely adding up the number of the calculation task through more than twice scheduling) of shake occurs for the response time that Map Reduce system is finished the work and Map Reduce system.
The computing power distribution of table 1 computing node
| Physical machine type | Number of units |
| 1 | VMs/host | 202 |
| 2 | VMs/host | 264 |
| 3 | VMs/host | 201 |
| 4 | VMs/host | 140 |
| 5 | VMs/host | 45 |
| 6 | VMs/host | 12 |
| 7 | VMs/host | 7 |
Following table 2 and table 3 are that the response time contrast form peace of above-mentioned three kinds of models in this emulation experiment all shakes number of times form, and wherein, Berkeley represents LATE scheme, and Patent represents the present embodiment scheme;
Table 2 response time, (unit: tick) contrasted form
| Number of tasks | Hadoop | Berkeley | Patent |
| 1000 | 21749.4 | 19990.2 | 16814.6 |
| 2000 | 38381.2 | 38223.6 | 33608.8 |
| 3000 | 57929.4 | 55130.6 | 51802.3 |
| 4000 | 81571.7 | 73063.4 | 69270.5 |
| 5000 | 99111.3 | 94638.1 | 86405.8 |
Table 3 average jitter number of times form
| Number of tasks | Hadoop | Berkeley | Patent |
| 1000 | 71.8 | 45.7 | 14.1 |
| 2000 | 77.9 | 47.5 | 24 |
| 3000 | 79.8 | 50.4 | 42.6 |
| 4000 | 90.3 | 57.7 | 45.1 |
| 5000 | 94.5 | 62.2 | 49.3 |
Can recognize from table 2 and table 3, the scheme of the present embodiment is all better than Hadoop and LATE in deadline and shake.
The slow method for scheduling task of the Map Reduce system of the present embodiment, by before by slow task scheduling to target computing nodes, estimate the computing power value after this target computing nodes loading slow task to be scheduled, and when the computing power value that this is estimated is greater than the computing power value of metewand node, just by slow task scheduling to be scheduled to described target computing nodes, can ensure that this target computing nodes can not make computing power decline after the slow task of loading too many, thus avoid the generation of new slow task, prevent the repeatedly scheduling of slow task, effectively inhibit the generation of jitter phenomenon during slow task scheduling.
Fig. 2 is the structural representation of the slow task scheduling apparatus embodiment of Map Reduce system of the present invention, this device can perform the slow method for scheduling task of any embodiment of the present invention, the present embodiment only does simple declaration to the structure of this device, and concrete principle of work can in conjunction with see described in embodiment of the method.
As shown in Figure 2, the slow task scheduling apparatus of the present embodiment can comprise: parameter acquiring unit 21, ability estimate unit 22, queue column unit 23, scheduling processing unit 24; Wherein,
Parameter acquiring unit 21, for obtaining the computing power value of each computing node in Map Reduce system respectively, the computing power value of described computing node
described v
1... v
mrepresent the processing speed of each calculation task on described computing node, described in
represent the average treatment speed of the work belonging to described each calculation task respectively, described m represents the total quantity of the calculation task on described computing node;
Ability estimates unit 22, for choose described queue column unit produce slow node queue in before M computing node, each described computing node chosen is as target computing nodes, and described M is natural number; And obtain the computing power value that M described target computing nodes estimates after loading slow task to be scheduled respectively, described in the computing power estimated
described v
irepresent and target computing nodes waits the processing speed of dispatching slow task, described in
represent and target computing nodes waits the average treatment speed of dispatching the work belonging to slow task;
Queue column unit 23, for being arranged as slow node queue according to described computing power value order from big to small by described each computing node; And, according to the computing power value of each computing node outside the computing power value estimated of described target computing nodes and described target computing nodes, each computing node outside described target computing nodes and described target computing nodes is arranged in new slow node queue according to computing power value order from big to small;
Scheduling processing unit 24, be metewand node for the N number of computing node preset in described slow node queue from tail of the queue forward, described N is natural number; And when the computing power value that described target computing nodes is estimated is greater than the computing power value of described metewand node, by described slow task scheduling to be scheduled to described target computing nodes.
Optionally, parameter acquiring unit 21 can comprise: speed receives subelement 211, average treatment subelement 212, ability acquisition subelement 213; Wherein,
Speed receives subelement 211, the processing speed of the calculation task that each computing node for receiving in described Map Reduce system reports, and described calculation task is the calculation task processed on computing node;
Average treatment subelement 212, for obtaining the average treatment speed of the work belonging to described calculation task, described average treatment speed according to described processing speed
described v represents the described processing speed of each described calculation task of the split one-tenth of described work, and described n represents total number of the computing node at each calculation task place of the split one-tenth of described work;
Ability obtains subelement 213, for the average treatment speed of described work is sent to described each computing node respectively, and the computing power value of the described each computing node obtained according to the average treatment speed of described work receiving that described each computing node reports; Or, for obtaining the computing power value of described each computing node according to the average treatment speed of described work.
Optionally, speed receives subelement 211, specifically for when the type of calculation task is Map type, receive the processing speed of the calculation task of the described Map type of correspondence that described each computing node calculates, described processing speed v=p/t, described p is the data volume of the calculation task that the current process of described computing node completes, and described t is that the process of the calculation task of described data volume is consuming time.
Optionally, the device of the present embodiment can also comprise: scheduling trigger element 25, for before the computing power value that the target computing nodes in acquisition Map Reduce system is estimated after loading slow task to be scheduled, receive the application calculation task message that the computing node in described Map Reduce system sends.
The slow task scheduling apparatus of the Map Reduce system of the present embodiment, by before by slow task scheduling to target computing nodes, estimate the computing power value after this target computing nodes loading slow task to be scheduled, and when the computing power value that this is estimated is greater than the computing power value of metewand node, just by slow task scheduling to be scheduled to described target computing nodes, can ensure that this target computing nodes can not make computing power decline after the slow task of loading too many, thus avoid the generation of new slow task, prevent the repeatedly scheduling of slow task, effectively inhibit the generation of jitter phenomenon during slow task scheduling.
One of ordinary skill in the art will appreciate that: all or part of step realizing above-mentioned each embodiment of the method can have been come by the hardware that programmed instruction is relevant.Aforesaid program can be stored in a computer read/write memory medium.This program, when performing, performs the step comprising above-mentioned each embodiment of the method; And aforesaid storage medium comprises: ROM, RAM, magnetic disc or CD etc. various can be program code stored medium.
Last it is noted that above each embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to foregoing embodiments to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein some or all of technical characteristic; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the scope of various embodiments of the present invention technical scheme.
Claims (8)
1. a slow method for scheduling task for Map Reduce system, is characterized in that, comprising:
Obtain the computing power value of each computing node in Map Reduce system respectively, the computing power value of described each computing node
described v
1... v
mrepresent the processing speed of each calculation task on described each computing node, described in
represent the average treatment speed of the work belonging to described each calculation task respectively, described m represents the total quantity of the calculation task on described each computing node;
According to described computing power value order from big to small, described each computing node is arranged as slow node queue, and chooses front M computing node in described slow node queue, each described computing node chosen is as target computing nodes, and described M is natural number;
Obtain the computing power value that M described target computing nodes is estimated after loading slow task to be scheduled respectively, described in the computing power value estimated
Described v
irepresent and target computing nodes waits the processing speed of dispatching slow task, described in
represent and target computing nodes waits the average treatment speed of dispatching the work belonging to slow task;
According to the computing power value of each computing node outside the computing power value estimated of described target computing nodes and described target computing nodes, each computing node outside described target computing nodes and described target computing nodes is arranged in new slow node queue according to computing power value order from big to small, and the N number of computing node preset in described slow node queue from tail of the queue is forward metewand node, described N is natural number;
When the computing power value that described target computing nodes is estimated is greater than the computing power value of described metewand node, by described slow task scheduling to be scheduled to described target computing nodes.
2. the slow method for scheduling task of Map Reduce system according to claim 1, is characterized in that, the computing power value of each computing node in described acquisition Map Reduce system, comprising:
Receive the processing speed of the calculation task that each described computing node in described Map Reduce system reports, described calculation task is the calculation task processed on described computing node;
The average treatment speed of the work belonging to described calculation task is obtained, described average treatment speed according to described processing speed
described v represents the described processing speed of each calculation task that described work splits into, and described n represents total number of the computing node at each calculation task place of the split one-tenth of described work;
The average treatment speed of described work is sent to described each computing node respectively, and the computing power value of the described each computing node obtained according to the average treatment speed of described work receiving that described each computing node reports.
3. the slow method for scheduling task of Map Reduce system according to claim 1, is characterized in that, the computing power value of each computing node in described acquisition Map Reduce system, comprising:
Receive the processing speed of the calculation task that each described computing node in described Map Reduce system reports, described calculation task is the calculation task processed on described computing node;
The average treatment speed of the work belonging to described calculation task is obtained, described average treatment speed according to described processing speed
described v represents the described processing speed of each described calculation task of the split one-tenth of described work, and described n represents total number of the computing node at each calculation task place of the split one-tenth of described work;
The computing power value of described each computing node is obtained according to the average treatment speed of described work.
4. according to the slow method for scheduling task of the arbitrary described Map Reduce system of claim 1-3, it is characterized in that, before the computing power value that described acquisition target computing nodes is estimated after loading slow task to be scheduled, also comprise:
Receive the application calculation task message that the computing node in described Map Reduce system sends.
5. the slow method for scheduling task of Map Reduce system according to claim 1, is characterized in that, the ratio of the sum of the computing node in described N and described Map Reduce system is 10%.
6. a slow task scheduling apparatus for Map Reduce system, is characterized in that, comprising: parameter acquiring unit, ability estimate unit, queue column unit and scheduling processing unit;
Described parameter acquiring unit, for obtaining the computing power value of each computing node in Map Reduce system respectively, the computing power value of described each computing node
described v
1... v
mrepresent the processing speed of each calculation task on described each computing node, described in
represent the average treatment speed of the work belonging to described each calculation task respectively, described m represents the total quantity of the calculation task on described each computing node;
Described ability estimates unit, for choose described queue column unit produce slow node queue in before M computing node, each described computing node chosen is as target computing nodes, and described M is natural number; And obtain the computing power value that M described target computing nodes estimates after loading slow task to be scheduled respectively, described in the computing power value estimated
Described v
irepresent and target computing nodes waits the processing speed of dispatching slow task, described in
represent and target computing nodes waits the average treatment speed of dispatching the work belonging to slow task;
Described queue column unit, for being arranged as slow node queue according to described computing power value order from big to small by described each computing node; And, according to the computing power value of each computing node outside the computing power value estimated of described target computing nodes and described target computing nodes, each computing node outside described target computing nodes and described target computing nodes is arranged in new slow node queue according to computing power value order from big to small;
Described scheduling processing unit, be metewand node for the N number of computing node preset in described slow node queue from tail of the queue forward, described N is natural number; And when the computing power value that described target computing nodes is estimated is greater than the computing power value of described metewand node, by described slow task scheduling to be scheduled to described target computing nodes.
7. the slow task scheduling apparatus of Map Reduce system according to claim 6, is characterized in that, described parameter acquiring unit comprises:
Speed receives subelement, the processing speed of the calculation task that each described computing node for receiving in described Map Reduce system reports, and described calculation task is the calculation task processed on computing node;
Average treatment subelement, for obtaining the average treatment speed of the work belonging to described calculation task, described average treatment speed according to described processing speed
described v represents the described processing speed of each described calculation task of the split one-tenth of described work, and described n represents total number of the computing node at each calculation task place of the split one-tenth of described work;
Ability obtains subelement, for the average treatment speed of described work is sent to described each computing node respectively, and the computing power value of the described each computing node obtained according to the average treatment speed of described work receiving that described each computing node reports; Or, for obtaining the computing power value of described each computing node according to the average treatment speed of described work.
8. the slow task scheduling apparatus of the Map Reduce system according to claim 6 or 7, is characterized in that, also comprise:
Scheduling trigger element, for before the computing power value that the target computing nodes in described acquisition Map Reduce system is estimated after loading slow task to be scheduled, receives the application calculation task message that the computing node in described Map Reduce system sends.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210016143.2A CN102609303B (en) | 2012-01-18 | 2012-01-18 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210016143.2A CN102609303B (en) | 2012-01-18 | 2012-01-18 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102609303A CN102609303A (en) | 2012-07-25 |
| CN102609303B true CN102609303B (en) | 2014-12-17 |
Family
ID=46526697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210016143.2A Active CN102609303B (en) | 2012-01-18 | 2012-01-18 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102609303B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309738B (en) * | 2013-05-31 | 2016-12-28 | 中国联合网络通信集团有限公司 | User job dispatching method and device |
| CN104243531A (en) * | 2013-06-21 | 2014-12-24 | 华为技术有限公司 | Data processing method, device and system |
| CN103699433B (en) * | 2013-12-18 | 2017-07-14 | 中国科学院计算技术研究所 | One kind dynamically adjusts number of tasks purpose method and system in Hadoop platform |
| CN103761146B (en) * | 2014-01-06 | 2017-10-31 | 浪潮电子信息产业股份有限公司 | A kind of method that MapReduce dynamically sets slots quantity |
| CN104102794A (en) * | 2014-08-06 | 2014-10-15 | 浪潮电子信息产业股份有限公司 | Optimizing method of HADOOP scheduling method |
| CN105808346B (en) * | 2014-12-30 | 2019-09-20 | 华为技术有限公司 | A kind of method for scheduling task and device |
| CN104765648B (en) * | 2015-04-30 | 2017-12-08 | 北京奇艺世纪科技有限公司 | The problem of one kind is based on real time computation system nodal test method and device |
| CN106528189B (en) * | 2015-09-10 | 2019-05-28 | 阿里巴巴集团控股有限公司 | A kind of method, apparatus and electronic equipment starting backup tasks |
| CN106357813B (en) * | 2016-11-02 | 2019-08-06 | 龙存科技（北京）股份有限公司 | A kind of task applied to shared-file system dispatching method again |
| CN111835854B (en) * | 2020-07-17 | 2021-08-10 | 北京航空航天大学 | Slow task prediction method based on grey prediction algorithm |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101005689A (en) * | 2006-06-23 | 2007-07-25 | 华为技术有限公司 | Setting method for dispatching priority node of customer terminal and base station |
| CN101883113A (en) * | 2010-06-25 | 2010-11-10 | 中兴通讯股份有限公司 | Method and physical nodes for realizing overlay network load balance |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043673B (en) * | 2009-10-21 | 2015-06-03 | Sap欧洲公司 | Calibration of resource allocation during parallel processing |
| CN102004670B (en) * | 2009-12-17 | 2012-12-05 | 华中科技大学 | Self-adaptive job scheduling method based on MapReduce |
-
2012
- 2012-01-18 CN CN201210016143.2A patent/CN102609303B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101005689A (en) * | 2006-06-23 | 2007-07-25 | 华为技术有限公司 | Setting method for dispatching priority node of customer terminal and base station |
| CN101883113A (en) * | 2010-06-25 | 2010-11-10 | 中兴通讯股份有限公司 | Method and physical nodes for realizing overlay network load balance |
Non-Patent Citations (1)
| Title |
|---|
| 异构环境下Map-Reduce调度算法的研究;陈全;《中国优秀硕士学位论文全文数据库信息科技辑I138-6》;20100731;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102609303A (en) | 2012-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102609303B (en) | Slow-task dispatching method and slow-task dispatching device of Map Reduce system | |
| JP6898496B2 (en) | Computation graph processing | |
| Taheri et al. | A bee colony based optimization approach for simultaneous job scheduling and data replication in grid environments | |
| US11025561B2 (en) | Systems and methods for computing infrastructure resource allocation | |
| Liu et al. | Scheduling parallel jobs with tentative runs and consolidation in the cloud | |
| WO2012154177A1 (en) | Varying a characteristic of a job profile relating to map and reduce tasks according to a data size | |
| CN108205469B (en) | MapReduce-based resource allocation method and server | |
| US9298500B2 (en) | Information processing system and control method of information processing system for managing jobs in a distributed multi-node environment | |
| Gómez-Martín et al. | Fattened backfilling: An improved strategy for job scheduling in parallel systems | |
| CN108509280A (en) | A kind of Distributed Calculation cluster locality dispatching method based on push model | |
| Wang et al. | Design and implementation of an analytical framework for interference aware job scheduling on apache spark platform | |
| CN109032769B (en) | Container-based continuous integrated CI (CI) task processing method and device | |
| Shah et al. | Dynamic multilevel hybrid scheduling algorithms for grid computing | |
| US20210390405A1 (en) | Microservice-based training systems in heterogeneous graphic processor unit (gpu) cluster and operating method thereof | |
| Zikos et al. | A clairvoyant site allocation policy based on service demands of jobs in a computational grid | |
| Goswami et al. | ALVEC: auto-scaling by Lotka Volterra Elastic Cloud: a QoS aware non linear dynamical allocation model | |
| US20220413941A1 (en) | Computing clusters | |
| EP3200083A1 (en) | Resource scheduling method and related apparatus | |
| CN114090219A (en) | Scheduling system, method, device, chip, computer device and storage medium | |
| Orland et al. | A case study on addressing complex load imbalance in OpenMP | |
| CN108268310B (en) | Method and device for determining minimum scheduling granularity | |
| Wang et al. | ACS: an effective admission control scheme with deadlock resolutions for workflow scheduling in clouds | |
| Kopanski et al. | Plan-based job scheduling for supercomputers with shared burst buffers | |
| Kim et al. | GPGPU-Perf: efficient, interval-based DVFS algorithm for mobile GPGPU applications | |
| Dumitru et al. | A queueing theory approach to pareto optimal bags-of-tasks scheduling on clouds |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |