CN102184124B - Task scheduling method and system - Google Patents
Task scheduling method and system Download PDFInfo
- Publication number
- CN102184124B CN102184124B CN 201110121393 CN201110121393A CN102184124B CN 102184124 B CN102184124 B CN 102184124B CN 201110121393 CN201110121393 CN 201110121393 CN 201110121393 A CN201110121393 A CN 201110121393A CN 102184124 B CN102184124 B CN 102184124B
- Authority
- CN
- China
- Prior art keywords
- task
- computing node
- input data
- scheduling
- allocated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a task scheduling method which comprises the following steps: a compute node requests to allocate tasks; whether the compute node requesting to allocate tasks contains input data or not is judged, if so, the task to be allocated, corresponding to the input data, is dispatched to the compute node, and if not, the dispatching probability is calculated according to the input data distribution and task traversal sequence of the task to be allocated; and the task to be allocated is dispatched to the compute node according to the dispatching probability sequence from high to low. In the task scheduling method and system, the task of the input data, stored in the compute node requesting to allocate tasks, is allocated to the compute node requesting to allocate tasks for executing; when a task of the input data, stored in the compute node requesting to allocate tasks, cannot be allocated, the dispatching probability is calculated according to the input data distribution and task traversal sequence of the task to be allocated, and the task with greatest dispatching probability is preferentially allocated to the compute node requesting to allocate tasks for executing so as to reduce the tasks causing remote data access delay and achieve the aim of reducing network load.
Description
[technical field]
The present invention relates to distributed computing technology, particularly relate to a kind of method for scheduling task and system.
[background technology]
Along with developing rapidly of internet, in traditional mass data intensive applications, the input data scale of wanting due to required by task is very large, has caused the expense that transmits data very large.Usually task scheduling is carried out to improve system performance to the back end of having stored the input data that this required by task wants.But the processing power of each node is limited, and the computational resource of vying each other between different tasks, exists to be dispatched to the task that back end is carried out, and these tasks must operate input data copy with correspondence in XM by remote I/O.
Yet in the remote I/O operation, a large amount of copies of input data have increased offered load.Traditional optimal way usually adopts prefetching technique.Prefetching technique maskable remote data access time delay, when XM is not stored the input data, to input data pre-fetching in XM, if XM Insufficient disk space, will input data copy to the node nearest apart from XM, this will cause the secondary transmission of input data, fails to solve the problem that offered load increases the weight of.
[summary of the invention]
Based on this, be necessary to provide a kind of method for scheduling task that reduces offered load.
In addition, also be necessary to provide a kind of task scheduling system that reduces offered load.
A kind of method for scheduling task comprises the following steps:
Computing node request allocating task;
Whether there are the input data in the computing node of judgement request allocating task, if, dispatch the to be allocated task corresponding with described input data to described computing node, if not, distribute according to task input data to be allocated and task traversal order calculating scheduling probability;
According to described scheduling probability order from high to low with task scheduling to be allocated to the described computing node.
Preferably, also comprise before described step according to task input data distribution to be allocated and task traversal order calculating scheduling probability:
Computing node according to the next request of Task Progress prediction allocating task;
Obtain the task that the input data are stored in the computing node of described next request allocating task;
The task of the input data being stored in the computing node of described next request allocating task keeps to described next computing node of asking allocating task.
Preferably, described the ask step of the computing node of allocating task next according to Task Progress prediction to be allocated is:
The reference input data scale is set;
The progress that each task that the input data scale is not equal to the reference input data scale calculates imaginary task according to Task Progress and the reference input data scale of correspondence;
Progress with imaginary task substitutes described Task Progress;
According to all tasks of large minispread of Task Progress, get and the computing node of the corresponding computing node of the task of Task Progress maximum as next one request allocating task.
Preferably, described all tasks of large minispread according to Task Progress, get the computing node corresponding with the task of Task Progress maximum and also comprise as after the step of the computing node of next one request allocating task:
In the task of arranging take the task of Task Progress maximum as the initial corresponding computing node of task of default number that extracts successively as the computing node of next one request allocating task.
Preferably, the described step of calculating the scheduling probability according to distribution and the task traversal order of task input data to be allocated is:
The computing node whether the input data of grade, task traversal order and the described task to be allocated of setting with respect to the computing node of request allocating task according to task to be allocated are stored in next request allocating task calculates the scheduling probability;
It is described that step to the described computing node is with task scheduling to be allocated according to described scheduling probability order from high to low:
Judge whether to exist the task to be allocated that belongs to described task grade according to the distance of the computing node of request allocating task and the memory location of input data successively in the task grade of dividing according to from the near to the remote order, if, according to dispatching the corresponding task to be allocated of probability sequential scheduling from high to low to the computing node of asking allocating task, if not, select next task dispatching level to enter the judgement that whether has task to be allocated in described next task grade.
A kind of task scheduling system comprises at least:
Computing node is for the treatment of task;
Control device, whether the computing node that is used for judgement request allocating task exists the input data, if, notify dispatching device, if not, notifier processes device;
Described dispatching device is used for the scheduling to be allocated task corresponding with described input data to described computing node, according to described scheduling probability order from high to low with task scheduling to be allocated extremely in described computing node;
Described treating apparatus is used for distributing and task traversal order calculating scheduling probability according to task input data to be allocated.
Preferably, also comprise:
Prediction unit is used for the computing node according to the next request of Task Progress prediction allocating task;
Described control device also is used for obtaining the task that the input data are stored in the computing node of described next request allocating task;
Described dispatching device also is used for input the task that data are stored in the computing node of described next request allocating task and keeps to described next computing node of asking allocating task.
Preferably, described prediction unit comprises:
Imagination Task Progress computing unit, be used for arranging with reference to input size, the input data scale is not equal to the progress that each task with reference to the input data scale calculates imaginary task according to Task Progress and the reference input data scale of correspondence, with the alternative described Task Progress of the progress of imaginary task;
Extraction unit is used for all tasks of large minispread according to Task Progress, get with the Task Progress maximum the corresponding computing node of task as the computing node of next one request allocating task.
Preferably, described extraction unit also is used in the task of arranging take the task of Task Progress maximum as the initial corresponding computing node of task of default number that extracts successively as the computing node of next one request allocating task.
Preferably, whether described treating apparatus is stored in next computing node calculating scheduling probability of asking allocating task for the input data of the grade of setting with respect to the computing node of asking allocating task according to task to be allocated, task traversal order and described task to be allocated;
Described dispatching device also is used for judging whether to exist in the task grade of dividing the task to be allocated that belongs to described task grade according to the distance of the computing node of request allocating task and the memory location of input data successively according to from the near to the remote order, if, according to dispatching the corresponding task to be allocated of probability sequential scheduling from high to low to the computing node of asking allocating task, if not, select next task dispatching level to enter the judgement that whether has task to be allocated in described next task grade.
In above-mentioned method for scheduling task and system, the task of the input data being stored in the computing node of request allocating task is distributed to the computing node execution of asking allocating task, in the time can not distributing the input data to be stored in the task of the computing node of asking allocating task, distribute and traversal order calculating scheduling probability according to task input data to be allocated, task to the computing node of asking allocating task of priority allocation scheduling maximum probability is carried out, cause the task of remote data access time-delay with minimizing, reach simultaneously the purpose that reduces offered load.
In above-mentioned method for scheduling task and system, calculate the computing node of the next request of prediction allocating task, the input data are stored in the next one task on the computing node of allocating task of asking to be kept to next requesting node execution, reduce offered load and remote data access time delay, improved system performance.
[description of drawings]
Fig. 1 is the process flow diagram of a method for scheduling task in embodiment;
Fig. 2 is the process flow diagram of the method for scheduling task in another embodiment;
Fig. 3 is method flow diagram of asking the computing node of allocating task next according to the Task Progress prediction in Fig. 1;
Fig. 4 is the structural representation of a task scheduling system in embodiment;
Fig. 5 is the structural representation of the task scheduling system in another embodiment;
Fig. 6 is the structural representation of prediction module in Fig. 5.
[embodiment]
Fig. 1 shows the method flow of a task scheduling in embodiment, comprises the following steps:
Step S101, computing node request allocating task.In the present embodiment, computing node is handled one during allocating task, and request distributes new task in the situation that the resources such as current CPU (central processing unit, Central Processing Unit), hard disk meet the demands.For example, when computing node handling one after allocating task in the situation that the demand of the free space of hard disk in satisfying the new task implementation initiated the request of allocating task.
Whether step S103 exists the input data in the computing node of judgement request allocating task, if, enter step S105, if not, enter step S107.In the present embodiment, the input data are the computing node necessary data of executing the task.In the task scheduling and implementation of reality, input data corresponding to each task are stored in any computing node, when computing node was carried out a certain task, corresponding input data may be stored in this computing node, also may be stored in other computing node.If the input data be stored in other computing node, the computing node of executing the task need to by remote I/O operate input data transmission to obtain be used to the input data of executing the task.
For reducing offered load, reduce the number of tasks of carrying out the remote I/O operation in processing procedure, preferred scheduling input data are stored in the task of this computing node, make computing node not need to input the transmission of data when executing the task, and then avoid causing the remote data access time-delay because of the remote I/O operation.
Step S105 dispatches the to be allocated task corresponding with the input data to computing node.In the present embodiment, when the input data of having stored in the computing node of request allocating task until scheduler task, to input the corresponding task scheduling of data to the computing node of asking allocating task with this, the transmission that makes this computing node not need to input data can be executed the task.
Step S107 distributes and task traversal order calculating scheduling probability according to task input data to be allocated.In the present embodiment, for task scheduling to be allocated to carrying out in the nearest computing node of the computing node of having stored the input data, need to be according to the memory location calculating of dispatching probability of input data in task to be allocated, the scheduling probability is higher, and the memory location of explanation input data is more approaching with the computing node of carrying out this task.
In a specific embodiment, according to the step that task to be allocated input data distribute and task traversal order calculates the scheduling probability be: the computing node whether the input data of grade, task traversal order and the task to be allocated of setting with respect to the computing node of request allocating task according to task to be allocated are stored in next request allocating task calculates dispatches probability.In the present embodiment, for task scheduling to carrying out from having stored the nearest computing node of input data, task is carried out the setting of task grade according to the memory location of input data with the distance of the computing node of request allocating task.In a preferred embodiment, the task division that the input data is stored in the computing node of request allocating task is the first estate, with the input data be stored in be connected in other computing node on same switch with the computing node of asking allocating task task as the second grade, the input data are stored in from the computing node of asking allocating task link task on the computing node of different switches as the tertiary gradient.The input data of the task of the second grade and the tertiary gradient all are not stored in computing node, if distribute the task of the second grade and the tertiary gradient for the computing node of request allocating task, described node needs to carry out data transmission in the process of carrying out these tasks, increase offered load.It inputs the node of data to storage if can keep task, and described node does not need to carry out data transmission in the process of executing the task.Therefore need the scheduling probability of calculation task.When the computing node request task, give lower probability by the task of being stored in the computing node of next request allocating task for the input data, and the highest task of priority allocation scheduling probability, can reduce causing the remote data access time-delay and increasing offered load of task.The scheduling probability is calculated by following formula:
P
R＝P
L-P
k+P
T
Wherein, P
RBe the scheduling probability of task, P
LBe the probability factor that is determined by the task grade, P
kThe probability factor that k the computing node of next request task of serving as reasons determines, P
TBe the probability factor that is determined by task traversal order.
Wherein, i is the task grade, and l is the task total number of grades.
If the input data of task are stored on k computing node,
If not, P
k=0.
Wherein, j is task traversal order, and n is the subtask sum in total task number or task.
With the P that calculates
L, P
kAnd P
TObtain the scheduling probability of each task in the computing formula of substitution scheduling probability.
In another embodiment, as shown in Figure 2, calculate the step of dispatching probability according to the memory location of input data in task to be allocated in above-mentioned method for scheduling task before further comprising the steps of:
Step S201 is according to the computing node of the next request of Task Progress prediction allocating task.In the present embodiment, the input data volume of having processed and input data total amount than Task Progress in value representation computing node.For example, in homogeneous environment, the processing power of each computing node is identical, under the identical prerequisite of all task input data scales, the fastest task of Task Progress will be done the earliest, process the computing node of this task and also will ask the earliest to carry out new task, namely this computing node is the computing node of next request allocating task.
In a specific embodiment, as shown in Figure 3, above-mentioned the ask step of the computing node of allocating task next according to the Task Progress prediction is:
Step S211 arranges the reference input data scale.In the present embodiment, in the computing node that all are being executed the task, each computing node has with it Task Progress one to one.But in the actual process of task, the input data of each task are all to be divided into some parts according to certain length, every portion is the input data of a subtask, therefore, if the input data scale of task is not the integral multiple of input data standard partition length, exist the input data scale not to wait in task and with reference to the task of input size.For the identical task of input data scale different task progress, input the pending input data of the little task of data scale less, will first complete, carry out the computing node of described task and first ask allocating task.
The difference of input data scale causes being difficult to computing node of asking allocating task next according to the Task Progress Accurate Prediction.For this reason, will input greatly the data standard partition length as reference input data scale, for example, the reference input data scale can be distributed file system file block size in computing environment.
Step S231, the progress that each task that the input data scale is not equal to the reference input data scale calculates imaginary task according to Task Progress and the reference input data scale of correspondence is with the alternative described Task Progress of the progress of imaginary task.In the present embodiment, be not equal to each task with reference to input size for the input data scale, in order to ask according to the Task Progress prediction next one computing node of allocating task, with each duty mapping to one input data scale imaginary task identical with the reference input data scale, with the progress replacement Task Progress of imaginary task.Model as follows can be used for calculating the progress of imaginary task:
Wherein, x and y represent respectively Task Progress and input data scale; α represents the reference input data scale, the progress of the imaginary task of f (x, y) expression.
Step S251 according to all tasks of large minispread of Task Progress, gets and the computing node of the corresponding computing node of the task of Task Progress maximum as next one request allocating task.In the present embodiment, with the Task Progress of each task according to large minispread, with the computing node of the task of the implementation progress maximum computing node as next one request allocating task.For example, each Task Progress can be stored in list by descending, at this moment, be the computing node of the next one request allocating task of prediction with the computing node that in list, first element is corresponding.
In addition, above-mentioned method for scheduling task is according to the large minispread task of progress, get the task of also having comprised after the step of computing node as the computing node of next one request allocating task of task of implementation progress maximum in the task of arranging take the Task Progress maximum as the initial corresponding computing node of task of default number that extracts successively as the computing node of next one request allocating task.In the present embodiment, owing to existing uncertainty and dynamic, situation about therefore predicting and actual conditions may be also incomplete same, suppose that the computing node request allocating task order of prediction and the error between PS are k, take the task of Task Progress maximum as initial k the task of extracting successively, the next computing node of allocating task of asking necessarily comprises in this k computing node in the task of arranging.Error k can measure by experiment and obtain, and adjusts flexibly according to the actual needs.
Step S203 obtains the task that the input data are stored in the computing node of next request allocating task.
Step S205, the task that the input data are stored in the computing node of next request allocating task keeps the computing node to next one request allocating task.
Step S109, according to scheduling probability order from high to low with task scheduling to be allocated to computing node.in the present embodiment, according to scheduling probability order from high to low with task scheduling to be allocated to the step in computing node be: judge whether to exist the task to be allocated that belongs to this task grade according to the distance of the computing node of request allocating task and the memory location of input data successively in the task grade of dividing according to from the near to the remote order, if, according to dispatching the corresponding task to be allocated of probability sequential scheduling from high to low to the computing node of asking allocating task, if not, select next task dispatching level to enter the judgement that whether has task to be allocated in this next task grade.
judge in each task grade and whether have task, because the task grade is to divide with the distance of the distance of the computing node of request allocating task according to the memory location of input data in task, therefore, whether there is the task of belonging to this task grade in the nearest task grade of the computing node of first judging distance request allocating task, if distance asks not have the task of belonging to this task grade in the nearest task grade of the computing node of allocating task, judge in each task grade successively according to from the near to the remote order according to the memory location of input data in task to be allocated distance with the computing node of request allocating task.For example, in the Three Estate of dividing, begin to judge whether have task to be allocated this grade, judge by that analogy successively from the first estate.
If have task to be allocated in a certain task grade, the task to be allocated in this task grade is distributed to task to be allocated successively the computing node of request allocating task according to scheduling probability order from high to low.If there is no task to be allocated in a certain task grade, enter the next task grade and judge in this task grade whether have task to be allocated.
Under the effect of scheduling probability, because the computing node priority allocation of asking allocating task arrives the high task of scheduling probability, therefore input in task that data are stored in the computing node of next request allocating task can not be assigned to the request allocating task because the scheduling probability is low computing node and carry out, and keep the computing node of asking allocating task to next, avoided described task to cause in the process of implementation remote data access time-delay and offered load.
Fig. 4 shows the detailed construction of task scheduling system in an embodiment, comprises at least computing node 10, control device 30, dispatching device 50 and treating apparatus 70.
Treating apparatus 70 is used for distributing and task traversal order calculating scheduling probability according to task input data to be allocated.In the present embodiment, for task scheduling to be allocated is carried out to inputting in the nearest computing node 10 of data from this task, need treating apparatus 70 to distribute according to input data in task to be allocated and task traversal order calculating scheduling probability, the scheduling probability is higher, and the computing node 10 of the memory location of explanation input data and current request allocating task is more approaching.
In a specific embodiment, whether treating apparatus 70 is stored in next computing node calculating scheduling probability of asking allocating task for the input data of the grade of setting with respect to the computing node of asking allocating task according to task to be allocated, task traversal order and task to be allocated.In the present embodiment, it is the first estate that the task division that data are stored in the computing node 10 of asking allocating task will input in scheduling probability calculation unit 703, with the input data be stored in be connected in other computing node on same switch with the computing node 10 of asking allocating task task as the second grade, the input data are stored in from the computing node 10 of asking allocating task link task on the computing node 10 of different switches as the tertiary gradient.The task on the computing node of asking allocating task of being stored in for the input data to be about to is given lower probability.When the computing node request task, dispatch the highest task of probability by priority allocation, can keep task to the clearing node of its input data of storage and carry out.The scheduling probability is calculated by following formula:
P
R＝P
L-P
k+P
T
Wherein, P
RBe the scheduling probability of task, P
LBe the probability factor that is determined by the task grade, P
kThe probability factor that k the computing node of next request task of serving as reasons determines, P
TBe the probability factor by task traversal order.
Wherein, i is the task grade, and l is the task total number of grades.
If the individual computing node that is about to request task of k has been stored the input data of task,
If not, P
k=0.
Wherein, j is task traversal order, and n is total task number or the sum of the subtask in task that comprises in operation.
With the P that calculates
L, P
kAnd P
TObtain the scheduling probability of each task in the computing formula of substitution scheduling probability.
In another embodiment, as shown in Figure 5, above-mentioned task scheduling system has also comprised prediction unit 90, and this prediction unit 90 is used for the computing node 10 according to the next request of Task Progress prediction allocating task.In the present embodiment, the processing progress than task in value representation computing node of the input data volume of having processed and input data total amount.For example, in homogeneous environment, the processing power of each computing node is identical, under the identical prerequisite of all task input data scales, the fastest task of Task Progress will be done the earliest, process the computing node of this task and also will ask the earliest to carry out new task, namely this computing node is the computing node of next request allocating task.
In a specific embodiment, as shown in Figure 6, prediction unit 90 comprises progress computing unit 901 and extraction unit 903.
Imagination Task Progress computing unit 901, be used for arranging with reference to input size, be not equal to for the input data scale progress that each task with reference to the input data scale calculates imaginary task according to Task Progress and the reference input data scale of correspondence, with the progress alternative tasks progress of imaginary task.In the present embodiment, in the computing node that all are being executed the task, each computing node has corresponding with it Task Progress.But in the actual process of task, the input data scale of each task differs to establish a capital and equals the reference input data scale.For the identical subtask of input data scale different task progress, input the pending input data of the little task of data scale less, will first complete, carry out the computing node of this task and first ask allocating task.
The difference of different task input data scale causes being difficult to computing node of asking allocating task next according to the Task Progress Accurate Prediction.For this reason, will input data criteria for classifying length as reference input data scale, for example, the reference input data scale can be distributed file system File block size.
Be not equal to reference to each task of inputting data scale for the input data scale, in order to process according to task the computing node of the next request of schedule forecasting allocating task, the imagination Task Progress computing unit 901 imaginary task that each duty mapping to one input size is identical with the reference input data scale is with the progress replacement Task Progress of imaginary task.Model as follows can be used for calculating the progress of imaginary task:
Wherein, x and y represent respectively Task Progress and input data scale; α represents the reference input data scale, and f (x, y) represents imaginary progress.
In addition, extraction unit 903 also is used in the task of arranging take the task of Task Progress maximum as the initial corresponding computing node of task of default number that extracts successively as the computing node of next one request allocating task.In the present embodiment, owing to existing uncertainty and dynamic, situation about predicting and actual conditions can be also incomplete same, the request allocating task order of assumed calculation node 10 prediction and the error between PS are k, and the computing node 10 of next request allocating task necessarily is included in the task of arrangement task take the progress maximum in the computing node 10 of initial k the task of extracting successively.Error k can measure by experiment and obtain, and adjusts flexibly according to the actual needs.
dispatching device 50 judges in each task grade whether have task, because the task grade is to divide with the distance of the distance of the computing node 10 of request allocating task according to the memory location of task input data, therefore, whether there is the task of belonging to this task grade in the nearest task grade of the computing node 10 of first judging distance request allocating task, if distance asks not have the task of belonging to this task grade in the nearest task grade of the computing node 10 of allocating task, judge in each task grade successively according to from the near to the remote order according to the memory location of input data in task to be allocated distance with the computing node 10 of request allocating task.For example, in the Three Estate of dividing, begin to judge whether have task to be allocated this grade, judge by that analogy successively from the first estate.
If have task to be allocated in a certain task grade, dispatching device 50 is distributed to the task to be allocated in this task grade the computing node of request allocating task successively with task to be allocated according to the order from high to low of scheduling probability.If there is no task to be allocated in a certain task grade, dispatching device 50 enters the next task grade and judges in this task grade whether have task to be allocated.
Under the effect of scheduling probability, because the computing node priority allocation of asking allocating task arrives the high task of scheduling probability, therefore input in task that data are stored in the computing node of next request allocating task can not be assigned to the current request allocating task because the scheduling probability is low computing node and carry out, carry out to the computing node of next request allocating task and keep, reach the minimizing data transmission, reduced the purpose of offered load.
In above-mentioned method for scheduling task and system, the task of the input data being stored in the computing node of request allocating task is distributed to the computing node execution of asking allocating task, in the time can not distributing the input data to be stored in the task of the computing node of asking allocating task, distribute and traversal order calculating scheduling probability according to task input data to be allocated, task to the computing node of asking allocating task of priority allocation scheduling maximum probability is carried out, cause the task of remote data access time-delay with minimizing, reach simultaneously the purpose that reduces offered load.
In above-mentioned method for scheduling task and system, calculate the computing node of the next request of prediction allocating task, the input data are stored in the next one task on the computing node of allocating task of asking to be kept to next requesting node execution, reduce offered load, reduced the remote data access time-delay, improved system performance.
The above embodiment has only expressed several embodiment of the present invention, and it describes comparatively concrete and detailed, but can not therefore be interpreted as the restriction to the scope of the claims of the present invention.Should be pointed out that for the person of ordinary skill of the art, without departing from the inventive concept of the premise, can also make some distortion and improvement, these all belong to protection scope of the present invention.Therefore, the protection domain of patent of the present invention should be as the criterion with claims.
Claims (8)
1. method for scheduling task comprises the following steps:
Computing node request allocating task;
Whether there are the input data in the computing node of judgement request allocating task, if, dispatch the to be allocated task corresponding with described input data to described computing node, if not, distribute according to task input data to be allocated and task traversal order calculating scheduling probability;
The described step of calculating the scheduling probability according to distribution and the task traversal order of task input data to be allocated is:
The computing node whether the input data of grade, task traversal order and the described task to be allocated of setting with respect to the computing node of request allocating task according to task to be allocated are stored in next request allocating task calculates the scheduling probability;
Described scheduling probability is calculated by following formula:
P
R＝P
L-P
k+P
T
Wherein, P
RBe the scheduling probability of task, P
LBe the probability factor that is determined by the task grade, P
kThe probability factor that k the computing node of next request task of serving as reasons determines, P
TBe the probability factor that is determined by task traversal order;
Wherein, i is the task grade, and l is the task total number of grades;
If the input data of task are stored on k computing node,
If not, P
k=0;
Wherein, j is task traversal order, and n is the subtask sum in total task number or task;
With the P that calculates
L, P
kAnd P
TObtain the scheduling probability of each task in the computing formula of substitution scheduling probability;
According to described scheduling probability order from high to low with task scheduling to be allocated to the described computing node.
2. method for scheduling task according to claim 1, is characterized in that, also comprises before described step according to task input data distribution to be allocated and task traversal order calculating scheduling probability:
Computing node according to the next request of Task Progress prediction allocating task;
Obtain the task that the input data are stored in the computing node of described next request allocating task;
The task of the input data being stored in the computing node of described next request allocating task keeps to described next computing node of asking allocating task;
Described the ask step of the computing node of allocating task next according to the Task Progress prediction is:
The reference input data scale is set;
The progress that each task that the input data scale is not equal to the reference input data scale calculates imaginary task according to Task Progress and the reference input data scale of correspondence is with the alternative described Task Progress of the progress of imaginary task;
According to all tasks of large minispread of Task Progress, get and the computing node of the corresponding computing node of the task of Task Progress maximum as next one request allocating task.
3. method for scheduling task according to claim 2, it is characterized in that, described all tasks of large minispread according to Task Progress, get the computing node corresponding with the task of Task Progress maximum and also comprise as after the step of the computing node of next one request allocating task:
In the task of arranging take the task of Task Progress maximum as the initial corresponding computing node of task of default number that extracts successively as the computing node of next one request allocating task.
4. method for scheduling task according to claim 1, is characterized in that, described step to the described computing node is with task scheduling to be allocated according to described scheduling probability order from high to low:
Judge whether to exist the task to be allocated that belongs to described task grade according to the distance of the computing node of request allocating task and the memory location of input data successively in the task grade of dividing according to from the near to the remote order, if, according to dispatching the corresponding task to be allocated of probability sequential scheduling from high to low to the computing node of asking allocating task, if not, select next task dispatching level to enter the judgement that whether has task to be allocated in described next task grade.
5. a task scheduling system, is characterized in that, comprises at least:
Computing node is for the treatment of task;
Control device, whether the computing node that is used for judgement request allocating task exists the input data, if, notify dispatching device, if not, notifier processes device;
Described dispatching device is used for the scheduling to be allocated task corresponding with described input data to described computing node, according to described scheduling probability order from high to low with task scheduling to be allocated extremely in described computing node;
Whether described treating apparatus is stored in next computing node calculating scheduling probability of asking allocating task for the input data of the grade of setting with respect to the computing node of asking allocating task according to task to be allocated, task traversal order and described task to be allocated;
Described scheduling probability is calculated by following formula:
P
R＝P
L-P
k+P
T
Wherein, P
RBe the scheduling probability of task, P
LBe the probability factor that is determined by the task grade, P
kThe probability factor that k the computing node of next request task of serving as reasons determines, P
TBe the probability factor by task traversal order;
Wherein, i is the task grade, and l is the task total number of grades;
If the individual computing node that is about to request task of k has been stored the input data of task,
If not, P
k=0;
Wherein, j is task traversal order, and n is total task number or the sum of the subtask in task that comprises in operation;
With the P that calculates
L, P
kAnd P
TObtain the scheduling probability of each task in the computing formula of substitution scheduling probability.
6. task scheduling system according to claim 5, is characterized in that, also comprises:
Prediction unit is used for the computing node according to the next request of Task Progress prediction allocating task;
Described control device also is used for obtaining the task that the input data are stored in the computing node of described next request allocating task;
Described dispatching device also is used for input the task that data are stored in the computing node of described next request allocating task and keeps to described next computing node of asking allocating task;
Described prediction unit comprises:
Imagination Task Progress computing unit, be used for arranging with reference to input size, the input data scale is not equal to the progress that each task with reference to the input data scale calculates imaginary task according to Task Progress and the reference input data scale of correspondence, with the alternative described Task Progress of the progress of imaginary task;
Extraction unit is used for all tasks of large minispread according to Task Progress, get with the Task Progress maximum the corresponding computing node of task as the computing node of next one request allocating task.
7. task scheduling system according to claim 6, it is characterized in that, described extraction unit also is used in the task of arranging take the task of Task Progress maximum as the initial corresponding computing node of task of default number that extracts successively as the computing node of next one request allocating task.
8. task scheduling system according to claim 5, is characterized in that,
Described dispatching device also is used for judging whether to exist in the task grade of dividing the task to be allocated that belongs to described task grade according to the distance of the computing node of request allocating task and the memory location of input data successively according to from the near to the remote order, if, according to dispatching the corresponding task to be allocated of probability sequential scheduling from high to low to the computing node of asking allocating task, if not, select next task dispatching level to enter the judgement that whether has task to be allocated in described next task grade.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110121393 CN102184124B (en) | 2011-05-11 | 2011-05-11 | Task scheduling method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110121393 CN102184124B (en) | 2011-05-11 | 2011-05-11 | Task scheduling method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102184124A CN102184124A (en) | 2011-09-14 |
| CN102184124B true CN102184124B (en) | 2013-06-05 |
Family
ID=44570304
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110121393 Active CN102184124B (en) | 2011-05-11 | 2011-05-11 | Task scheduling method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102184124B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104754007A (en) * | 2013-12-26 | 2015-07-01 | 伊姆西公司 | Method and device for managing network attached storage |
| CN104199738B (en) * | 2014-08-11 | 2018-05-25 | 阔地教育科技有限公司 | A kind of more data processing equipment collaboration working methods and system |
| CN107645541B (en) * | 2017-08-24 | 2021-03-02 | 创新先进技术有限公司 | Data storage method and device and server |
| CN111679904B (en) * | 2020-03-27 | 2023-10-31 | 北京世纪互联宽带数据中心有限公司 | Task scheduling method and device based on edge computing network |
| CN113176937B (en) * | 2021-05-21 | 2023-09-12 | 抖音视界有限公司 | Task processing method and device and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6393455B1 (en) * | 1997-03-28 | 2002-05-21 | International Business Machines Corp. | Workload management method to enhance shared resource access in a multisystem environment |
| CN101770402A (en) * | 2008-12-29 | 2010-07-07 | 中国移动通信集团公司 | Map task scheduling method, equipment and system in MapReduce system |
| CN102004670A (en) * | 2009-12-17 | 2011-04-06 | 华中科技大学 | Self-adaptive job scheduling method based on MapReduce |
-
2011
- 2011-05-11 CN CN 201110121393 patent/CN102184124B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6393455B1 (en) * | 1997-03-28 | 2002-05-21 | International Business Machines Corp. | Workload management method to enhance shared resource access in a multisystem environment |
| CN101770402A (en) * | 2008-12-29 | 2010-07-07 | 中国移动通信集团公司 | Map task scheduling method, equipment and system in MapReduce system |
| CN102004670A (en) * | 2009-12-17 | 2011-04-06 | 华中科技大学 | Self-adaptive job scheduling method based on MapReduce |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102184124A (en) | 2011-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108351805B (en) | Flow-based accelerator processing of computational graphs | |
| US10474504B2 (en) | Distributed node intra-group task scheduling method and system | |
| CN108885571B (en) | Input of batch processing machine learning model | |
| CN102902587B (en) | Distributed task dispatching mthods, systems and devices | |
| CN106020933B (en) | Cloud computing dynamic resource scheduling system and method based on ultralight amount virtual machine | |
| CN102063336B (en) | Distributed computing multiple application function asynchronous concurrent scheduling method | |
| JP5343523B2 (en) | Job management apparatus, job management method, and job management program | |
| CN111400022A (en) | Resource scheduling method and device and electronic equipment | |
| CN105487930A (en) | Task optimization scheduling method based on Hadoop | |
| CN109561148A (en) | Distributed task dispatching method in edge calculations network based on directed acyclic graph | |
| CN102184124B (en) | Task scheduling method and system | |
| CN103401939A (en) | Load balancing method adopting mixing scheduling strategy | |
| CN109857535B (en) | Spark JDBC-oriented task priority control implementation method and device | |
| Li et al. | An effective scheduling strategy based on hypergraph partition in geographically distributed datacenters | |
| CN114371926B (en) | Refined resource allocation method and device, electronic equipment and medium | |
| Delavar et al. | A synthetic heuristic algorithm for independent task scheduling in cloud systems | |
| CN104917839A (en) | Load balancing method used in cloud computing environment | |
| CN115237586A (en) | GPU resource configuration method for deep learning inference performance interference perception | |
| Zikos et al. | A clairvoyant site allocation policy based on service demands of jobs in a computational grid | |
| CN112559179A (en) | Job processing method and device | |
| CN113254200B (en) | Resource arrangement method and intelligent agent | |
| Sirohi et al. | Improvised round robin (CPU) scheduling algorithm | |
| CN112130979B (en) | Method, device, terminal and medium for scheduling task and training neural network model | |
| CN114896070A (en) | GPU resource allocation method for deep learning task | |
| JP2018181123A (en) | Resource allocation control system, resource allocation control method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |