WO2016197716A1

WO2016197716A1 - Task scheduling method and device

Info

Publication number: WO2016197716A1
Application number: PCT/CN2016/080057
Authority: WO
Inventors: 郑鹏飞
Original assignee: 中兴通讯股份有限公司
Priority date: 2016-01-18
Filing date: 2016-04-22
Publication date: 2016-12-15
Also published as: CN106980532A

Abstract

A task scheduling method, comprising: when free resources appear, calculating a priority of each task within a task queue according to a size factor of each task within the task queue and a waiting factor of each task within the task queue, and sorting the tasks within the task queue according to the priorities (S101); performing scheduling processing on the tasks within the task queue according to the order of the task queue after sorting (S102).

Description

Job scheduling method and device

Technical field

This document relates to, but is not limited to, the field of computer technology, and in particular, to a job scheduling method and apparatus.

Background technique

MapReduce is a widely used distributed system processing framework for processing large-scale data. The scheduler is a core component of MapReduce and has a critical impact on the performance of the entire system. Currently, common schedulers are: fair scheduler, FIFO scheduler.

According to the current situation, small jobs dominate the load. For example, in Facebook and Bing applications, jobs with less than 10 jobs accounted for 82% and 61% of the total number of jobs, respectively, while in Google, the average execution time for MapReduce jobs was 395 seconds, at Yahoo! In a production cluster, more than 80% of the job completion time is less than ten minutes. The large increase in small jobs is because small jobs such as random queries are starting to run more and more in MapReduce clusters, where latency is a critical factor influencing the experience of small jobs.

The fair scheduler is currently the most widely used scheduler. The core idea of the fair scheduler is to make all users and jobs as close as possible to the 'substantially equal' cluster resources. Since all users and jobs are emphasized to occupy resources fairly, The fair scheduler has a longer response time to interactive small jobs. Many distributed systems use the FIFO scheduling scheduler by default, but the FIFO scheduler also has some problems, such as the operation is prone to "starvation", can not be applied to multi-user scenarios, etc., therefore, the scope of application is narrow.

Aiming at the problem that the fair scheduler has a long response time to small jobs, a small job priority scheduling strategy is proposed. The small job priority scheduling strategy is compared with the fair scheduling strategy. The test results show that the average response time of the job using the small job priority scheduling strategy is shorter than the average response time of the job using the fair scheduling strategy, especially to a large extent. Shorten the response time of small jobs and have little impact on the response time of large jobs. However, it is also necessary to consider that in practical applications, small job priority strategies may cause “hunger” in large jobs, such as having large jobs. In the case of existence, a large number of small jobs are continuously submitted to the MapReduce system. Due to the priority scheduling of short jobs, large jobs may not be scheduled for processing, resulting in "starvation."

In summary, in the scheduling process of distributed systems, how to shorten the response time of small jobs to a large extent, and prevent "starvation" in large jobs is the problem to be solved.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

This paper provides a job scheduling method and device, which can shorten the response time of small jobs, prevent "starvation" in large jobs, and achieve the purpose of improving job scheduling efficiency.

An embodiment of the present invention provides a job scheduling method, where the method includes: when a free resource occurs, calculating the job queue according to a size factor of each job in the job queue and a waiting factor of each job in the job queue. a priority of each job, and sorting the jobs in the job queue according to the priority; scheduling the jobs in the job queue according to the order of the sorted job queues.

In the above solution, when the idle resource occurs, the priority of each job in the job queue is calculated according to the size factor of each job in the job queue and the waiting factor of each job in the job queue, and Sorting the jobs in the job queue according to the priority, including: when an idle resource occurs, calculating a size factor of each job in the job queue according to a task processing condition of each job in the job queue; Calculating a waiting factor of the job in the job queue in a time scheduled for each job in the job queue; calculating the job according to the calculated size factor of each job and the waiting factor of each job The priority of each job in the queue.

In the above solution, when an idle resource occurs, calculating a size factor of each job in the job queue according to a task processing condition of each job in the job queue, including: when an idle resource occurs, determining the job queue separately Whether each job has been scheduled to be processed; if the job in the job queue has never been scheduled, the size factor of the job is calculated according to the input data amount of the job; if the job in the job queue Has been scheduled for processing, based on the job The size of the job is calculated by the time it takes to perform the remaining tasks.

In the above solution, if the job in the job queue has been scheduled to be processed, calculating a size factor of the job according to a time required for the job to execute the remaining tasks, including: calculating the job queue The current task progress of the middle job and the time used by the job in the job queue to reach the current progress; according to the calculated current task progress of the job and the time taken by the calculated job to reach the current progress, calculate the Calculating an execution rate of the job; calculating a remaining task quantity of the job according to the calculated current task progress of the job; calculating according to the calculated execution rate of the job and the remaining task quantity of the calculated job The time required for the job to execute the remaining tasks; and the size factor of the job is calculated according to the time required for the calculated job to execute the remaining tasks.

In the above solution, when the idle resource occurs, the priority of the job in the job queue is calculated according to the size factor of the job in the job queue and the waiting factor of the job in the job queue, and the job queue is The jobs are sorted according to the priority, including: when an idle resource occurs, each job in the job queue is calculated according to a size factor of each job in the job queue and a wait factor of each job in the job queue. a priority, and sorting the jobs in the job queue according to the priority; wherein the wait factor is positively related to the priority, and the size factor is negatively correlated with the priority.

An embodiment of the present invention provides a job scheduling apparatus, where the apparatus includes: a calculating unit, a sorting unit, and a processing unit; wherein the calculating unit is configured to, according to a size factor of each job in the job queue, when an idle resource occurs Calculating a priority of each job in the job queue with a wait factor of each job in the job queue; the sorting unit is configured to set the job in the job queue according to the priority calculated by the calculation unit The processing unit is configured to perform scheduling processing on the jobs in the job queue according to the order of the sorted job queues.

In the above solution, the calculating unit is configured to calculate a size factor of each job in the job queue according to a task processing condition of each job in the job queue when an idle resource occurs; according to each of the job queues Calculating the waiting factor of the job in the job queue before the job is scheduled; calculating each job in the job queue according to the calculated size factor of each job and the waiting factor of each job Priority.

In the above solution, the calculating unit is configured to determine, when an idle resource occurs, whether each job in the job queue has been scheduled for processing; if the job in the job queue has not been scheduled, Calculating a size factor of the job according to the input data of the job; if the job in the job queue has been scheduled, the size of the job is calculated according to a time required for the job to execute the remaining tasks factor.

In the above solution, the calculating unit is configured to calculate a current task progress of the job in the job queue and a time used to reach the current progress; according to the calculated current task progress of the job and the calculated Calculating the execution rate of the job when the job reaches the current progress; calculating the remaining task number of the job according to the calculated current task progress of the job; and calculating the execution rate of the job according to the calculated Calculating the remaining tasks of the job, calculating the time required for the job to execute the remaining tasks; and calculating the size factor of the job according to the time required for the calculated job to execute the remaining tasks.

In the above solution, the sorting unit is configured to calculate each of the job queues according to a size factor of each job in the job queue and a wait factor of each job in the job queue when an idle resource occurs a priority of the job, and sorting the jobs in the job queue according to the priority; wherein the wait factor is positively correlated with the priority; the size factor is negatively correlated with the priority.

The embodiment of the invention further provides a computer readable storage medium storing computer executable instructions, which are implemented when executed by a processor.

The embodiment of the invention provides a job scheduling method and device, which first calculates the priority of the job by the size factor and the waiting factor of the job, and then performs scheduling processing according to the priority of the job, so that the response time of the small job can be shortened. At the same time, it prevents "starvation" in large operations and achieves the technical effect of improving job scheduling efficiency.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF abstract

1 is a schematic flowchart of a job scheduling method according to an embodiment of the present invention;

2 is a schematic flowchart of a calculation of a job size factor according to an embodiment of the present invention;

FIG. 3 is a schematic flowchart of a job scheduling method according to an embodiment of the present disclosure;

4 is a schematic flowchart of another job scheduling method according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a job scheduling apparatus according to an embodiment of the present invention.

Embodiments of the invention

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, in the case of no conflict, the features in the embodiments and the embodiments in the present application may be arbitrarily combined with each other.

Embodiment 1

Referring to FIG. 1 , a job scheduling method according to an embodiment of the present invention is shown, which may include:

S101: When an idle resource occurs, calculate a priority of each job in the job queue according to a size factor of each job in the job queue and a waiting factor of each job in the job queue, and queue the job The jobs in the order are sorted according to the priority;

It should be noted that the priority of the job is dynamically calculated. Whenever there is an idle resource, the priority of each job in the job queue needs to be recalculated. In the specific implementation process, the job queues can be sorted according to the order of priority from high to low.

In the specific implementation process, when idle resources appear, all users are arranged into user queues according to fairness, and the job queues under the user name are scheduled and processed in sequence according to the order of the user queues; Before scheduling the processing queue under the user name, it is necessary to recalculate each of the job queues under the current user name according to the size factor of each job in the job queue under the current user name and the waiting factor of each job in the job queue. The priority of the job, and sort the jobs in the job queue under the current user name according to the priority of each job; After the scheduling process of the job queue under the user name is completed, the job queue under the next user name in the user queue is scheduled.

The implementation process of step S101 may include:

S1011: When an idle resource occurs, calculating a size factor of each job in the job queue according to a task processing condition of each job in the job queue;

In the specific implementation process, it is necessary to separately determine whether the job in the job queue has been scheduled and processed.

If the job in the job queue has not been scheduled, the size factor of the job is calculated according to the input data amount of the job; wherein the input data amount of the job may specifically be the number of tasks of the job.

If the job in the job queue has been scheduled, the size factor of the job is calculated according to the time required for the job to execute the remaining tasks. The specific execution steps include:

S10111: Calculate the current task progress of the job in the job queue and the time used in the job queue to reach the current progress;

S10112: Calculate an execution rate of the job according to the calculated current task progress of the job and the calculated time used by the job to reach the current progress;

S10113: Calculate the remaining tasks of the job according to the calculated current task progress of the job;

S10114: Calculate, according to the calculated execution rate of the job and the calculated remaining task amount of the job, a time required for the job to execute the remaining tasks;

S10115: Calculate the size factor of the job according to the calculated time required for the job to execute the remaining tasks.

It should be noted that the time required for the job to perform the remaining tasks is positively related to the size factor.

S1012: Calculate a wait factor of the job in the job queue according to a time scheduled for each job in the job queue;

In the specific implementation process, the waiting time refers to the time when the job was last allocated resources to the current time.

It should be noted that the longer the waiting time of the job is proportional to the waiting factor.

S1013: Calculate the priority of each job in the job queue according to the calculated size factor of each job and the waiting factor of each job.

S102: Perform scheduling processing on the jobs in the job queue according to the order of the job queues after sorting.

In the specific implementation process, it is necessary to sequentially determine whether the job in the job queue satisfies the data locality according to the order of the sorted job queues, and perform scheduling processing on the first job satisfying data locality, that is, idle Resource allocation is scheduled for the first job that satisfies data locality.

It should be noted that the data locality refers to the data that needs to be scheduled and processed on the node of the idle resource. When the job does not satisfy the data locality, the data of the job needs to be transmitted to the node of the idle resource, which is time-consuming and resource-consuming. Therefore, if the job to be processed in the job queue does not satisfy the data locality, the next job is scheduled according to the order of the job queue; if the job queue under the current user name does not satisfy the data locality, then according to The order of the user queues is scheduled for the next user; if there is no job that satisfies the data locality in the job queue under all the user names in the user queue, the idle resources are directly allocated to the users who are to be executed the map task. The highest priority user, in turn, allocates idle resources to the first job in the job queue under the highest priority user name to be executed.

The following is a further description of the size factor of the job:

First, the current task progress of the job is calculated according to the number of tasks that have been completed in the job and the progress of the running task; wherein, the current task progress of the job can be calculated by using equation (1):

In the formula (1), the job j is taken as an example for description, the parameter jobPS _j represents the current task progress of the job j, and the parameter c represents the number of tasks that have been completed in the job j, and the parameters

Indicates the progress of the task being run in job j, parameter R _j represents the set of tasks running in job j, and parameter r represents the number of tasks running in job j.

Next, based on the total time spent on the currently completed tasks in the job and the running tasks The total time spent completing the current progress, calculating the time taken for the job to reach the current progress; where equation (2) can be used to calculate the time taken by the job to reach the current progress:

In the formula (2), the job j is taken as an example for description, and the parameter jobExT _j represents the linear time used by the job j to reach the current progress, and the parameter

The sum of the time spent representing the currently completed task in job j, the parameter

Represents the sum of the time taken by the running task in job j to complete the current progress, parameter C _j is the set of tasks that job j has completed, parameter R _j represents the set of tasks that are running in job j, and parameter c is already in job j The number of completed tasks, the parameter time _t is the time taken to complete the task t in the job j, the parameter r represents the number of tasks being run in the job j, the parameter curT is the current time, and the parameter start _t is the task t in the job j starts running. time.

It should be noted that the parameters in equation (2)

The specific calculation method is: first subtract the time of the task from the current time of the task to get the time taken by the task to complete the current progress, and then accumulate the time taken by all the running tasks in the job j to complete the current progress, You can get the total time spent on the current progress of all running tasks in job j.

Then, according to the current task progress of the job obtained by the formula (1) and the time taken by the job obtained by the formula (2) to reach the current progress, the execution rate of the job is calculated; wherein the execution rate of the job can be calculated by using the formula (3) :

jobExR _j =jobPS _j /jobExT (3)

In the formula (3), the job j is taken as an example. The parameter jobExR _j represents the execution rate of the job j, the parameter jobPS _j represents the current task progress of the job j, and the parameter jobExT _j represents the time used by the job j to reach the current progress.

Next, according to the total number of tasks of the job and the current task progress of the job obtained by the formula (1), the remaining number of tasks of the job is calculated; wherein, the formula (4) can be used to calculate the remaining tasks of the job:

jobRS _j =N-jobPS _j (4)

In the equation (4), the job j is taken as an example. The parameter jobRS _j indicates the number of remaining tasks of the job j, the parameter jobPS _j indicates the current task progress of the job j, and the parameter N is the total number of tasks of the job j.

Further, according to the execution rate of the job obtained by the equation (3) and the remaining task number of the job obtained by the equation (4), the time required for the job to execute the remaining tasks is calculated; wherein the job execution can be calculated by using the equation (5) Time required to complete the remaining tasks:

jobRT _j =jobRS _j /jobExR _j (5)

In the formula (5), the job j is taken as an example for explanation. The parameter jobRT _j represents the remaining time required for the job j to complete all tasks, the parameter jobRS _j represents the remaining task number of the job j, and the parameter jobExR _j represents the execution rate of the job j. .

Finally, according to the time required for the job obtained by the formula (5) to execute the remaining tasks or the number of tasks of the job, the size factor of the job is calculated; wherein the size factor of the job can be calculated by using equation (6) or (7):

In the formula (6), the job j is taken as an example. The parameter jobSizeFactor _j represents the size factor of the job j, the parameter jobRT _j represents the time required for the job j to execute the remaining tasks, and the parameter n represents the job queue in the current user name. The number of jobs; in equation (7), the parameter jobSizeFactor _j represents the size factor of job j, the parameter jobRTasksNum _j represents the number of remaining tasks of job j, and the parameter n represents the number of jobs in the job queue under the current user name.

In the specific implementation process, for a job that has just been submitted to the system and has not been scheduled for processing, since the history information of the task execution is not yet available, the execution time of the job cannot be predicted. Therefore, using the input data size of the job, specifically the number of tasks of the job to determine the size factor of the job, as shown in equation (7), but this situation only occurs at the very beginning, the next time after the job is scheduled When the resource is allocated, the formula (6) is used instead to calculate the size factor of the job.

Next, a calculation method of the size factor of the job will be described in a specific embodiment:

First, determine if the job has been scheduled for processing;

If the job has never been scheduled, the size factor of the job is calculated by equation (7) according to the number of tasks of the job;

If the job has been scheduled for processing, the size factor of the job is calculated based on the time required to execute the remaining tasks. In the case that the job has been scheduled to be processed, the specific steps of calculating the size factor of the job are as shown in FIG. 2, including:

S201: Calculate the current task progress of the job by using equation (1);

S202: Calculate, by formula (2), a time used by the job to reach a current progress;

S203: calculating a current execution schedule of the job according to the current task schedule calculated by the formula (1) and a time calculated by the formula (2) to reach the current progress, and calculating an execution rate of the job by the formula (3);

S204: Calculate the remaining number of tasks of the job according to the total task amount of the job, by using equation (4);

S205: an execution rate of the job calculated according to the formula (3) and a remaining task amount of the job calculated by the formula (4), and calculating, by the formula (5), a time required for the job to execute the remaining tasks;

S206: Calculate the time required for the job to execute the remaining tasks according to the formula (5), and calculate the size factor of the job by the formula (6).

Further, in order to prevent individual jobs from monopolizing cluster resources and causing other jobs to wait for acquisition of resources for too long, that is, a so-called "starvation phenomenon" occurs, the present invention adds a wait factor to the calculation of job priorities to balance idle resources. The allocation, in order to shorten the response time of small operations, to avoid "starvation" in other operations. It should be noted that the waiting time of the job is from the time when the job last obtained the idle resource to the current time. The larger the wait factor for a job, the longer the job waits for other jobs.

For the waiting factor, it should be further explained that the waiting factor of the job can be calculated according to the waiting time of the job; wherein, the waiting time of the job can be calculated by using equation (8):

In the equation (8), the job j is taken as an example. The parameter jobWT _j represents the waiting time of the job j, and the parameter n represents the number of jobs in the job queue under the current user name.

Finally, based on the size factor and wait factor of the job, the priority of the job can be calculated; where (9) can be used to calculate the priority of the job:

Priority _j =jobWaitFactor _j +(1/jobSizeFactor _j ) (9)

In the formula (9), the job j is taken as an example. The parameter jobWaitFactor _j is the waiting factor of the job j obtained by the equation (8), and the parameter jobSizeFactor is the size factor of the job j obtained by the equation (6) or the equation (7). The parameter priority _j is the priority of the calculated job j.

As can be seen from equation (9), the priority of the job is inversely proportional to the size factor of the job, and is proportional to the wait factor of the job. The shorter the remaining time required for the job to complete all tasks, the longer the waiting time, the higher the priority of the job; on the contrary, the longer the remaining time required for the job to complete all tasks, the shorter the waiting time. The lower the priority of the job.

In this way, during job scheduling, the remaining time required for the job to complete all tasks and the waiting time of the job are converted into the priority of the job, taking into account the remaining time required for the job to complete all tasks and the waiting time of the job. An important factor, at the same time, because the remaining execution time and job waiting time required for the job to perform all tasks are dynamically changed, the priority of the job is also dynamically changed. Idle resources can be allocated according to the priority of the job, and jobs with higher priority will get the idle resources first. In this way, the priority of the small job is increased by the size factor of the job, so as to achieve the technical effect of shortening the response time of the small job; the priority of the large job is improved by the waiting factor of the job, so as to avoid the technical effect of avoiding "starvation" in the large operation. . This scheduling method can greatly shorten the response time of small jobs, prevent "starvation" in large jobs, improve the user experience, and achieve the technical effect of improving job scheduling efficiency.

For example, on a social networking site, users have a large number of search queries every day. These search queries are generally similar to sql retrieval. This kind of job requires the system to give results quickly. Similar to sql, such jobs are generally small jobs. . In addition, the back-end system of the social networking site needs to analyze the network of each user and other users, so as to recommend friends to the user, such a job is a big job, it takes a long time to calculate and the response time of the job is not high. . In this scenario, the scheduling method of the present invention can not only shorten the homework response time of a small job, but also cause a big job "starvation" and improve the user experience.

The following describes the scheduling process of the job in another specific embodiment. The specific steps are as shown in FIG. 3, including:

S301: When idle resources appear, all users are arranged into a user queue according to fairness;

S302: Select a first user in the user queue according to a sequence of user queues for scheduling processing;

S303: Calculate a priority of each job in the job queue under the current user name before the scheduling process;

Among them, the specific process of calculating the priority includes:

It should be noted that there are 3 jobs in the job queue under the current user name. Among them, a job and b job have been scheduled, c job is just submitted, so it has not been scheduled, so calculate a job and b The size factor of the job uses equations (1) to (6), and the size factor of the c job is calculated using equation (7).

S3031: Calculate the current task progress of the two jobs a and b in the job queue by the formula (1), and the calculation results of a and b are all 5;

S3032: Calculate the time used by the two jobs a and b in the job queue to reach the current progress by the formula (2), and the calculation results of the a job and the b job are 100s and 5s, respectively;

S3033: The execution rates of two jobs a and b in the job queue are respectively calculated by the formula (3), and the calculation results of the a job and the b job are 0.05 and 1, respectively;

S3034: Calculate the remaining tasks of the two jobs a and b in the job queue by the formula (4), and the calculation results of both the a job and the b job are 5;

It should be noted that the total number of tasks for the three jobs a, b, and c in the job queue is 10.

S3035: Calculate the time required for the two tasks a and b in the job queue to execute the remaining tasks by the formula (5), and the calculation results of the a job and the b job are 100s and 5s, respectively;

S3036: Calculate the size factors of two jobs a and b in the job queue by equation (6), and the calculation results of a job and b job are 19.05 and 0.95, respectively;

S3037: Calculate the size factor of the c job according to the formula (7), and the calculation result is 1;

S3038: The waiting factors of three jobs a, b, and c in the job queue are respectively calculated by the formula (8), and the calculation results of the a job, the b job, and the c job are all 0s;

It should be noted that the waiting time for the a job, the b job, and the c job is 0 s.

S3039: The priority of three jobs a, b, and c in the job queue is calculated by the formula (9), and the calculation results of the a job, the b job, and the c job are respectively: 0.05, 1.05, and 1;

S304: Sort the job queues under the current user name from high to low according to the priority, sort The result is: c, b, a;

S305: Determine, according to the order of the sorted job queues, that the jobs in the job queue under the current user name satisfy the data locality;

S306: Perform scheduling processing on the first job c in the sorted job queue, that is, allocate idle resources to the c job in the job queue.

It can be seen that when the waiting time of the job under the user name is equal, the technical solution of the present invention can preferentially schedule the small job, and considers the execution rate of the job, which not only shortens the response time of the small job to a large extent, but also improves the response time. The efficiency of job scheduling has greatly improved the user experience of small jobs.

The scheduling process of the job is described in another embodiment. The specific steps are as shown in FIG. 4, including:

S401: When idle resources are present, all users are arranged into a user queue according to fairness;

S402: The first user in the user queue is selected for scheduling processing according to the sequence of the user queue, because the first user in the user queue has been scheduled to be processed.

S403: Calculate a priority of three jobs in the job queue under the current user name before the scheduling process;

Among them, the specific process of calculating the priority includes:

It should be noted that there are 3 jobs in the job queue under the current user name, and all 3 jobs have been scheduled for processing.

S4031: The current task progress of the three jobs a, b, and c in the job queue is calculated by the formula (1), and the calculation results of a, b, and c are respectively: 50, 8, and 1;

S4032: Calculate the time used by the three jobs a, b, and c in the job queue to reach the current progress by the formula (2), and the calculation results of a, b, and c are respectively: 90s, 20s, 2s;

S4033: The execution rates of three jobs a, b, and c in the job queue are calculated by equation (3), and the calculation results of a, b, and c are respectively: 0.6, 0.4, and 0.5;

S4034: Calculate the remaining tasks of three jobs a, b, and c in the job queue by equation (4) The quantity, wherein the total number of tasks of the first job, the second job, and the third job in the job queue are 100, 10, and 5, respectively, and the calculation results of a, b, and c are respectively: 50, 2, and 4;

S4035: Calculate the time required for the three tasks a, b, and c in the job queue to execute the remaining tasks by equation (5). The calculation results of a, b, and c are: 90s, 5s, and 8s, respectively;

S4036: The size factors of three jobs a, b, and c in the job queue are calculated by equation (6), and the calculation results of a, b, and c are respectively: 100, 5.6, and 8.9;

S4037: The waiting factors of three jobs a, b, and c in the job queue are calculated by equation (8), wherein the waiting times of a, b, and c are respectively: 300s, 0s, 2s, a, b, c The calculation results are: 115, 0, 0.77;

S4038: The priority of three jobs a, b, and c in the job queue is calculated by equation (9), and the calculation results of a, b, and c are: 115.01, 0.18, and 0.89, respectively;

S404: Sort the job queues under the current user name from high to low according to the priority, and the sort result is: a, c, b.

S405: Determine, according to the order of the sorted job queues, that the jobs in the job queue under the current user name satisfy the data locality;

S406: Perform scheduling processing on the first job a in the job queue according to the order of the sorted job queues, that is, allocate idle resources to the a job in the job queue.

It can be seen that when the waiting time of the large operation is long, the technical solution of the present invention can use the waiting factor to influence the priority of the operation, so as to realize the processing of such operations as soon as possible, and to avoid the phenomenon of “starvation” to a large extent.

Embodiment 2

Based on the same technical concept of the above embodiment, referring to FIG. 5, a job scheduling apparatus 50 according to an embodiment of the present invention is shown. The apparatus 50 includes: a calculating unit 501, a sorting unit 502, and a processing unit 503;

The calculating unit 501 is configured to: when a free resource occurs, calculate a priority of each job in the job queue according to a size factor of each job in the job queue and a waiting factor of each job in the job queue;

The sorting unit 502 is configured to sort the jobs in the job queue according to the priorities calculated by the calculating unit 501;

The processing unit 503 is configured to perform scheduling processing on the jobs in the job queue according to the order of the sorted job queues.

In the above solution, the calculating unit 501 is configured to calculate, according to the task processing condition of each job in the job queue, a size factor of each job in the job queue when an idle resource occurs; according to the job queue Calculating a waiting factor of the job in the job queue at a time scheduled once before each job; calculating each of the job queues according to the calculated size factor of each job and the waiting factor of each job The priority of the job.

In the above solution, the calculating unit 501 is configured to determine, when an idle resource occurs, whether each job in the job queue has been scheduled to be processed; if the job in the job queue has not been scheduled, Calculating a size factor of the job according to the input data of the job; if the job in the job queue has been scheduled to be processed, calculating the job according to the time required for the job to execute the remaining tasks Size factor.

In the above solution, the calculating unit 501 is configured to calculate a current task progress of the job in the job queue and a time used to reach the current progress; and calculate the current task progress and the calculated according to the calculated job. Calculating the execution rate of the job when the job reaches the current progress; calculating the remaining task number of the job according to the calculated current task progress of the job; and calculating the execution rate of the job according to the calculated job And calculating, by the calculated number of remaining tasks of the job, a time required for the job to execute the remaining tasks; and calculating a size factor of the job according to the time required for the calculated job to execute the remaining tasks .

In the above solution, the sorting unit 502 is configured to calculate, when an idle resource occurs, each of the job queues according to a size factor of each job in the job queue and a wait factor of each job in the job queue. The priority of the jobs, and the jobs in the job queue are sorted according to the priority; wherein the wait factor is positively related to the priority; the size factor is negatively correlated with the priority.

One of ordinary skill in the art will appreciate that all or part of the steps in the above methods may be passed through the program. The instructions are related to hardware (eg, a processor) that can be stored in a computer readable storage medium, such as a read only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps of the above embodiments may also be implemented using one or more integrated circuits. Correspondingly, each module/unit in the above embodiment may be implemented in the form of hardware, for example, by implementing an integrated circuit to implement its corresponding function, or may be implemented in the form of a software function module, for example, executing a program stored in the memory by a processor. Instructions to achieve their corresponding functions. This application is not limited to any specific combination of hardware and software.

It should be noted that various other embodiments and modifications may be made by those skilled in the art without departing from the spirit and scope of the application, Corresponding changes and modifications are intended to fall within the scope of the appended claims.

Industrial applicability

A job scheduling method and apparatus provided by an embodiment of the present invention first calculates a priority of a job by a size factor and a wait factor of the job, and then performs scheduling processing according to the priority of the job, so that the response time of the small job can be shortened. At the same time, it prevents "starvation" in large operations and achieves the technical effect of improving job scheduling efficiency.

Claims

A job scheduling method, the method comprising:

When an idle resource occurs, the priority of each job in the job queue is calculated according to the size factor of each job in the job queue and the waiting factor of each job in the job queue, and the job queue is Jobs are sorted according to the priority;

Scheduling processing of jobs in the job queue according to the order of the sorted job queues.
The method of claim 1 wherein:

When the idle resource occurs, calculating a priority of each job in the job queue according to a size factor of each job in the job queue and a waiting factor of each job in the job queue, and scheduling the job The jobs in are sorted according to the priority, including:

When an idle resource occurs, the size factor of each job in the job queue is calculated according to the task processing condition of each job in the job queue;

Calculating a wait factor of the job in the job queue according to a time scheduled for each job in the job queue;

A priority of each job in the job queue is calculated based on the calculated size factor of each job and the wait factor of each job.
The method of claim 2 wherein:

When the idle resource occurs, calculating a size factor of each job in the job queue according to the task processing condition of each job in the job queue, including:

When an idle resource occurs, it is determined whether each job in the job queue has been scheduled for processing;

If the job in the job queue has never been scheduled, the size factor of the job is calculated according to the input data amount of the job;

If the job in the job queue has been scheduled to be processed, the size factor of the job is calculated according to the time required for the job to execute the remaining tasks.
The method of claim 3 wherein:

If the job in the job queue has been scheduled, the size factor of the job is calculated according to the time required for the job to execute the remaining tasks, including:

Calculating a current task progress of the job in the job queue and a time used by the job in the job queue to reach a current progress;

Calculating an execution rate of the job according to the calculated current task progress of the job and the time taken by the calculated job to reach the current progress;

Calculating a remaining number of tasks of the job according to the calculated current task progress of the job;

Calculating, according to the calculated execution rate of the job and the remaining number of tasks of the calculated job, a time required for the job to execute the remaining tasks;

The size factor of the job is calculated according to the time required for the calculated job to execute the remaining tasks.
The method of claim 1 wherein:

When the idle resource occurs, calculating a priority of the job in the job queue according to a size factor of the job in the job queue and a wait factor of the job in the job queue, and the job in the job queue is as described Prioritize, including:

When an idle resource occurs, the priority of each job in the job queue is calculated according to the size factor of each job in the job queue and the waiting factor of each job in the job queue, and the job queue is The jobs are ordered according to the priority; wherein the wait factor is positively correlated with the priority, the size factor being negatively correlated with the priority.
A job scheduling device, the device comprising: a computing unit, a sorting unit, and a processing unit; wherein

The calculating unit is configured to: when a free resource occurs, calculate a priority of each job in the job queue according to a size factor of each job in the job queue and a waiting factor of each job in the job queue;

The sorting unit is configured to sort the jobs in the job queue according to the priority calculated by the calculating unit;

The processing unit is configured to perform scheduling processing on the jobs in the job queue in accordance with the order of the sorted job queues.
The device of claim 6 wherein:

The calculating unit is configured to calculate, according to a task processing condition of each job in the job queue, a size factor of each job in the job queue when an idle resource occurs; according to the previous scheduling of each job in the job queue The time is calculated as a waiting factor of the job in the job queue; and the priority of each job in the job queue is calculated according to the calculated size factor of each job and the waiting factor of each job.
The device of claim 7 wherein:

The calculating unit is configured to determine, when an idle resource occurs, whether each job in the job queue has been scheduled to be processed; if the job in the job queue has not been scheduled, according to the job The input data calculates a size factor of the job; if the job in the job queue has been scheduled to be processed, the size factor of the job is calculated according to the time required for the job to execute the remaining tasks.
The device of claim 8 wherein:

The calculating unit is configured to calculate a current task progress of the job in the job queue and a time used to reach the current progress; according to the calculated current task progress of the job and the calculated job reaching the current progress Calculating the execution rate of the job according to the time of use; calculating the remaining task number of the job according to the calculated current task progress of the job; calculating the execution rate of the job according to the calculation and the calculated The number of remaining tasks of the job, calculating the time required for the job to execute the remaining tasks; calculating the size factor of the job according to the time required for the calculated job to execute the remaining tasks.
The device of claim 6 wherein:

The sorting unit is configured to calculate a priority of each job in the job queue according to a size factor of each job in the job queue and a wait factor of each job in the job queue when an idle resource occurs, And sorting the jobs in the job queue according to the priority; wherein the wait factor is positively related to the priority; the size factor is negatively correlated with the priority.