CN107589985B - Two-stage job scheduling method and system for big data platform - Google Patents
Two-stage job scheduling method and system for big data platform Download PDFInfo
- Publication number
- CN107589985B CN107589985B CN201710590748.5A CN201710590748A CN107589985B CN 107589985 B CN107589985 B CN 107589985B CN 201710590748 A CN201710590748 A CN 201710590748A CN 107589985 B CN107589985 B CN 107589985B
- Authority
- CN
- China
- Prior art keywords
- job
- platform
- scheduling
- resource
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a big data platform-oriented two-stage job scheduling method and a big data platform-oriented two-stage job scheduling system, which are used for forming a job set to be scheduled by jobs submitted by a user, and scheduling the jobs based on the maximum profit of the latest starting time of the jobs: pre-allocating resources of the platform according to the deadline of the operation, and adjusting and scheduling the result of the pre-allocation of the resources according to the profit comparison of the operation to obtain a resource pre-allocation operation scheduling result queue which enables the profit of a service provider to be the maximum; job scheduling based on the maximum resource utilization rate of the platform: and according to the resource use condition of the platform, carrying out micro-adjustment on the scheduling result queue of the resource pre-allocation job to obtain a final scheduling result queue, and ensuring that the resource utilization rate of the platform is highest on the premise of ensuring the maximum platform benefit. Experimental results show that the method not only realizes the maximization of platform profit, but also improves the resource utilization rate of the platform and the comprehensive performance of the platform.
Description
Technical Field
The invention belongs to the technical field of big data calculation, and particularly relates to a big data platform-oriented two-stage job scheduling method and system.
Background
In recent years, with the rapid development of cloud computing and internet technologies, data shows an explosive continuous growth mode, and a big data era comes quietly. The traditional data processing technology and tool can not meet the data processing requirements of the new era, so that a large data platform is produced. The big data platform supports various computing frameworks and can provide services for a plurality of users simultaneously. However, in a large data platform, multiple users share the resources of the platform, and for a platform provider, how to efficiently schedule the multi-user job can not only fully utilize the resources of the platform, but also meet the SLA requirements of most users, so that the benefit of the platform is maximized, which has become a problem to be solved urgently.
Currently, many researchers have conducted intensive research on the job scheduling problem in a large data platform, and many solutions have been proposed. An optimization model of resource allocation performance based on deadline estimation is provided for Pig jobs by 'Optimizing Completion Time and resource planning of Pig Programs' published by Zhang Z et al, which eliminates the problem of non-determinacy in the execution of concurrent jobs by a Pig program, however, the model does not consider the problem of platform profit. Liu et al propose a priority-based job parallel scheduling method, which divides the computing capacity of each node into a foreground virtual machine (with higher CPU priority) layer and a background virtual machine (with lower CPU priority) layer by using a virtual technology, and balances the load of the platform by dividing the two layers, thereby fully utilizing the CPU resources of the platform, improving the execution efficiency of the job with high efficiency, and shortening the response time of the job, but the scheduling method does not consider the problem of platform profit. Therefore, the existing research results are deeply researched aiming at the job scheduling problems under different constraint conditions and different backgrounds, and a series of results are obtained, but the methods in the results do not consider the problem of platform profit, and do not consider the problems of platform profit maximization and platform resource utilization maximization at the same time.
The big data platform can serve a plurality of users at the same time, for a platform service provider, the user services can be reasonably scheduled, the requirements of multiple users can be met at the same time, the income of the platform can be increased, the utilization rate of the platform can be improved, and the execution process of multi-user operation is shown in figure 1.
As can be seen from fig. 1, there are 6 jobs of three users in the platform, each user has two jobs, and after the user submits the job, the platform service provider will schedule the job and generate an execution queue according to the resources of the platform, the SLA agreements signed with the user, and the like, and the result is shown in fig. 1. After the user's job is completed, the service provider can obtain corresponding benefits. Under ideal conditions, the platform has enough resources, can meet the requirements of all users, and has the maximum profit for the service provider; however, in reality, the resources of the platform are limited and most likely not meet the needs of all users, and for the platform service provider, the following problems will be faced:
(1) according to the available resources in the platform and the SLA requirements of users, how to schedule multi-user operation can maximize the benefit of the users;
(2) on the basis of (1), supposing that a job execution queue capable of maximizing the profit of a service provider is generated, how to adjust the job execution queue can ensure the maximum profit and further improve the resource utilization rate of the platform.
In summary, an effective solution is not yet available how to simultaneously solve the problem of maximizing platform profit and maximizing platform resource utilization in a large data platform.
Disclosure of Invention
In order to solve the above problems, the present invention provides a two-stage job scheduling method for a big data platform. The scheduling method comprehensively considers the constraint conditions of deadline constraint of operation, maximum profit of the platform, maximum resource utilization rate and the like, adopts a two-stage operation scheduling method based on profit and resources, and not only can meet the operation deadline requirement of a large data platform user, but also can realize the maximum platform resource utilization rate while ensuring the maximization of the platform profit.
In order to achieve the purpose, the invention adopts the following technical scheme:
a two-stage job scheduling method facing a big data platform comprises the following steps:
(1) forming a job set to be scheduled by the job submitted by the user, and performing maximum profit scheduling based on the latest starting time of the job: pre-allocating resources of the platform according to the deadline of the operation, and adjusting and scheduling the result of the pre-allocation of the resources according to the profit comparison of the operation to obtain a resource pre-allocation operation scheduling result queue which enables the profit of a service provider to be the maximum;
(2) and performing job scheduling based on the maximum resource utilization rate of the platform: and (3) according to the resource use condition of the platform, carrying out micro-adjustment on the scheduling result queue of the resource pre-allocation job in the step (1) to obtain a final scheduling result queue, and ensuring that the resource utilization rate of the platform is the highest on the premise of ensuring the maximum platform benefit.
Further, the specific steps of the maximum benefit scheduling based on the latest start time of the job in step (1) are as follows:
(1-1) calculating the initial latest starting time of each job in the job set to be scheduled, and performing resource pre-allocation on the resources of the platform according to the initial latest starting time;
(1-2) counting the total number of the calculation resources required by each time period according to the allocation result of the resource pre-allocation to obtain a resource pre-allocation result P _ R;
(1-3) judging whether an overload time period exists in the resource pre-allocation result P _ R, if so, entering the step (1-4), and if not, entering the step (1-5);
(1-4) adjusting the initial latest starting time of the overload time period operation in the resource pre-allocation result P _ R, updating the P _ R according to the adjustment result, and returning to the step (1-3);
and (1-5) outputting a resource pre-allocation result P _ R which maximizes the profit of the service provider.
Further, in the step (1-1), the specific steps of performing resource pre-allocation on the platform resources according to the initial latest start time are as follows:
letting each job start execution at its initial latest start time, obtaining the number of resources required for each time period when all jobs have just completed at their deadline;
the initial latest start time of the job is: for any job in the job set to be scheduled, when no other job contends for resources, the job can be completed at the job start time just at the deadline point;
in the invention, when the deadlines of a plurality of jobs are close, due to the limited computing resources of the platform, resource contention may occur among the jobs, and the jobs can not be completed before the deadlines when starting to execute at the initial latest starting time. Therefore, the initial latest starting time of the operation in the resource contention time period needs to be adjusted, and the resource pre-allocation method and the system can be used for pre-allocating the resources of all the operations according to the initial latest starting time of the operation, thereby effectively determining the time period in which the resource contention occurs.
Further, in the step (1-3), it is determined whether an overload time period exists in the resource pre-allocation result P _ R, that is, it is determined whether the total number of computing resources required by the job running in a certain time period is greater than the total number of computing resources of the large data platform; if the time interval is larger than the normal load time interval, the time interval is an overload time interval, otherwise, the time interval is a normal load time interval.
The total number of computing resources of the big data platform is the number of all standard containers in the big data platform.
Further, the specific steps of the step (1-4) are as follows:
(1-4-1) selecting the last overload time period, and acquiring all the jobs executed in the time period to form a job set;
(1-4-2) selecting a minimum proper subset in the operation set, wherein the minimum proper subset leads the initial latest starting time of all the operations in the operation set to a normal load time period which enables the time period to be in a normal load state, and the set of all the minimum proper subsets meeting the conditions forms a feasible adjustment strategy set of the time period;
(1-4-3) evaluating the evaluation value of each feasible adjustment strategy in the feasible adjustment strategy set according to the evaluation function, and selecting the optimal adjustment strategy;
(1-4-4) according to the optimal adjustment strategy, adjusting the initial latest starting time of the operation in the strategy, and updating the resource pre-allocation result P _ R according to the adjusted initial latest starting time of the operation.
Further, the evaluation function in the step (1-4-3) is:
wherein the content of the first and second substances,
to adjust the sum of the revenue estimates for all jobs in the strategy,
and lastsize is the percentage of the residual resources in the current time period accounting for the total number of the platform computing resources after adjustment according to the strategy, wherein Sp is the profit evaluation value of the operation, Sp is | a-b |, a is the profit value obtained when the operation is completed on time, and b is the profit value obtained when the operation is not completed on time.
Further, the specific step of the job scheduling based on the maximum resource utilization rate of the platform in the step (2) is as follows:
(2-1) initializing a variable final scheduling result queue and T time;
(2-2) searching a job set which can be executed at the time T and does not conflict with the resource pre-allocation result P _ R;
(2-3) judging whether the job set which does not conflict with the resource pre-allocation result P _ R is an empty set or not, and if the job set is the empty set, setting the time T as the initial time of the next time period in the resource pre-allocation result P _ R; if not, selecting the operation which minimizes the resource waste rate in the operation set to obtain the optimal starting time and updating the time T;
(2-4) repeatedly executing the step (2-2) -the step (2-3) until an optimal starting time is set for each job, and obtaining a final scheduling result;
and (2-5) outputting a final scheduling result.
Further, the resource waste rate in the step (2-3) is a ratio between the non-reusable resource after scheduling and the currently used computing resource;
the currently used computing resource is the sum of the used computing resource and the free computing resource.
In order to fully utilize the resources of the platform and improve the income of a service provider in the big data platform, the invention provides a two-stage operation scheduling system facing the big data platform, and the scheduling system is based on the two-stage operation scheduling method facing the big data platform.
In order to achieve the purpose, the invention adopts the following technical scheme:
a big data platform-oriented two-stage job scheduling system, comprising:
the system comprises a first-stage scheduling module, a second-stage scheduling module and a scheduling module, wherein the first-stage scheduling module is used for forming a job set to be scheduled by jobs submitted by a user, and performing first adjustment scheduling on jobs in the job set to be scheduled by adopting maximum profit scheduling based on the latest start time of each job and the total maximum profit of a service provider to obtain a resource pre-allocation job scheduling result queue which enables the profit of the service provider to be maximum;
and
and the second-stage scheduling module is used for carrying out micro-adjustment on the scheduling result queue of the resource pre-allocation job of the first-stage scheduling module according to the resource use condition of the platform to obtain a final scheduling result queue, the resource utilization rate of the platform is maximized on the premise that the final scheduling result queue ensures the platform to have the maximum profit, the resources of the platform can be fully utilized, and each job zone in the final scheduling result queue has the optimal starting time.
Further, the job set to be scheduled is: the platform service provider receives a batch of jobs in a certain time period and negotiates with a user to sign an SLA protocol related to the jobs, and the set of the jobs is a set J of jobs to be scheduled, which is expressed as J ═ J1,j2,…,jnN is the number of the operations in J;
for any job J in job set J to be schedulediIs represented by ji(ms, rs, mt, rt, dl, bf (t)), wherein ms is the number of Map tasks for the job; rs is the Reduce task number of the operation; mt is the average execution time of the operation Map task; rt is the average execution time of the Reduce task of the job; dl is the deadline constraint for the job; bf (t) is a revenue function for the job.
Further, the first scheduling module includes a first job scheduler and a first resource scheduler, and the second scheduling module includes a second job scheduler and a second resource scheduler.
The invention has the beneficial effects that:
the invention discloses a two-stage job scheduling method and system for a big data platform, and provides the two-stage job scheduling system and method for jobs with deadline constraints based on a MapReduce calculation framework. The scheduling method calculates and adjusts the latest starting time of each job according to the deadline constraint of the job and the profit information of the job, and performs resource pre-allocation according to the adjustment result to ensure that the job with high profit can be completed before the deadline so as to maximize the total profit of the platform; in the second stage, on the premise of ensuring the maximum platform benefit, a job scheduling method based on the maximum platform resource utilization rate is provided to improve the platform resource utilization rate. Experimental results show that the two-stage operation scheduling method provided by the invention not only realizes the maximization of platform profit, but also improves the resource utilization rate of the platform and the comprehensive performance of the platform.
Drawings
FIG. 1 is a schematic diagram of a big data platform multi-user job execution process;
FIG. 2 is a flow chart of a method of the present invention;
FIG. 3 is a flowchart of a method of maximum revenue scheduling based on the latest start time of a job in accordance with the present invention;
FIG. 4 is a flowchart of a method for scheduling jobs based on maximum resource utilization of a platform according to the present invention;
FIG. 5 is a schematic diagram of the system of the present invention;
FIG. 6 is a diagram illustrating the relationship between the resource utilization and the average job size according to the present invention;
FIG. 7 is a diagram illustrating the relationship between job completion rate and average job size according to the present invention;
FIG. 8 is a graph illustrating the relationship between total revenue and average job size according to the present invention;
FIG. 9 is a schematic diagram illustrating the effect of job set size on resource utilization according to the present invention;
FIG. 10 is a schematic diagram illustrating the effect of job set size on job completion rate;
FIG. 11 is a schematic representation of the effect of job set size on revenue in accordance with the present invention;
FIG. 12 is a diagram illustrating the impact of the total number of computing resources on resource utilization according to the present invention;
FIG. 13 is a diagram illustrating the effect of the total number of computing resources on the completion rate of a job according to the present invention;
FIG. 14 is a graphical illustration of the impact of the total number of computing resources on revenue in accordance with the present invention;
FIG. 15 is a diagram illustrating the impact of job urgency on resource utilization according to the present invention.
The specific implementation mode is as follows:
it should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
The embodiments and features of the embodiments in the present application may be combined with each other without conflict. The invention is further described with reference to the following figures and examples.
Example 1:
as introduced by the background art, the problems of maximization of platform profit and maximization of platform resource utilization rate cannot be effectively solved in the prior art, and a two-stage job scheduling method for a big data platform is provided. The scheduling method comprehensively considers the constraint conditions of deadline constraint of operation, maximum profit of the platform, maximum resource utilization rate and the like, adopts a two-stage operation scheduling method based on profit and resources, and not only can meet the operation deadline requirement of a large data platform user, but also can realize the maximum platform resource utilization rate while ensuring the maximization of the platform profit.
In order to achieve the purpose, the invention adopts the following technical scheme:
as shown in figure 2 of the drawings, in which,
a two-stage job scheduling method facing a big data platform comprises the following steps:
(1) forming a job set to be scheduled by the job submitted by the user, and scheduling based on the maximum profit of the latest start time of the job: pre-allocating resources of the platform according to the deadline of the operation, and adjusting and scheduling the result of the pre-allocation of the resources according to the profit comparison of the operation to obtain a resource pre-allocation operation scheduling result queue which enables the profit of a service provider to be the maximum;
(2) job scheduling based on the maximum resource utilization rate of the platform: and (3) according to the resource use condition of the platform, carrying out micro-adjustment on the scheduling result queue of the resource pre-allocation job in the step (1) to obtain a final scheduling result queue, and ensuring that the resource utilization rate of the platform is the highest on the premise of ensuring the maximum platform benefit.
The first stage is as follows: maximum revenue scheduling based on the latest start time of a job
Firstly, forming a job set to be scheduled by jobs submitted by a user, wherein the job set to be scheduled is as follows:
the platform service provider receives a batch of jobs in a certain time period and negotiates with a user to sign an SLA protocol related to the jobs, and the set of the jobs is a set J of jobs to be scheduled, which is expressed as J ═ J1,j2,…,jnN is the number of the operations in J;
the invention relates to job scheduling in a big data platform based on a MapReduce computing framework under a homogeneous cluster, so that the hardware configuration of each node is assumed to be approximately same, the processing performance and the stability are consistent, and the running time of any task is consistent no matter which node runs. For any job, the method and the device assume that the number of Map tasks and the number of Reduce tasks of the job are known, and the average execution time of the Map tasks and the average execution time of the Reduce tasks are known, and in addition, the method and the device do not consider the data skew condition of the job, and the processing time of each Map task (or each Reduce task) of the job is consistent by default.
For each job in J, the invention pays attention to the information of execution time, occupied resource number, deadline, income and the like of the job, and for any job J in the job set J to be schedulediIs represented by ji(ms, rs, mt, rt, dl, bf (t)), wherein ms is the number of Map tasks for the job; rs is the Reduce task number of the operation; mt is the average execution time of the operation Map task; rt is the average execution time of the Reduce task of the job; dl is the deadline constraint for the job; bf (t) is a revenue function for the job.
The revenue function bf (t) of a job is a function of the actual completion time j of the jobiPiecewise function of end:
wherein, a and b respectively represent the profit obtained by completing the job within the deadline and the profit obtained by failing to complete the job within the deadline (for non-loss of generality, the profit value b is represented by a negative number when the corresponding amount of the user is to be paid when the job cannot be completed by the deadline).
The invention provides that the deadline of the operation is soft deadline, namely the operation can not cause serious consequences when the operation can not be completed before the deadline, and the execution of the operation can not be abandoned.
As shown in fig. 3, the specific steps of the maximum benefit scheduling based on the latest start time of the job in the first stage of step (1) are as follows:
(1-1) calculating a set J ═ { J) of jobs to be scheduled1,j2,…,jnEach job j iniInitial latest start time j ofi.TolsAnd according toInitial latest start time ji.TolsPerforming resource pre-allocation on resources of the platform;
(1-2) according to the allocation result of the resource pre-allocation, counting the sum of the number of the computing resources required by the operation of the operation in each time period to obtain a resource pre-allocation result P _ R;
(1-3) judging whether an overload time period exists in the resource pre-allocation result P _ R, if so, entering the step (1-4), and if not, entering the step (1-5);
(1-4) adjusting the initial latest starting time of the overload time period operation in the resource pre-allocation result P _ R, updating the P _ R according to the adjustment result, and returning to the step (1-3);
and (1-5) outputting a resource pre-allocation result P _ R which maximizes the profit of the service provider.
In this embodiment, the specific steps of performing resource pre-allocation on the resources of the platform according to the initial latest start time in step (1-1) are as follows:
letting each job start execution at its initial latest start time, obtaining the number of resources required for each time period when all jobs have just completed at their deadline;
initial latest start time j of the jobi.TolsComprises the following steps: for job set to be scheduled, J ═ J1,j2,…,jnAny job j in (1)iWhen no other job contends for the resource, the job j is enablediA job start time that can just be completed at the end time point; if operation jiHas a cutoff time of jiDl, the initial latest start time calculation for this job is as follows:
in the present invention, j is applied to an arbitrary jobiWhen there is no resource contention, the job must start at the initial latest start time ji.TolsBefore starting execution can ensure that the job is completed before the deadline, when the job is started at the initial latest starting time ji.TolsThe moment beginsExecution can just be completed at the deadline point. When the deadlines of a plurality of jobs are close, due to limited computing resources of the platform, resource contention may occur between jobs, job jiAt its initial latest start time ji.TolsThe start of execution cannot be completed by the deadline. Therefore, a job j requiring a time period for contending for resourcesiInitial latest start time j ofi.TolsMaking adjustments in accordance with the present invention based on the initial latest start time j of the jobi.TolsFor all jobs jiPerforming pre-allocation of resources, i.e. letting job jiAt ji.TolsThe execution is started at the moment, so that the total number of computing resources required in each time period when all the jobs are just finished at the deadline of the jobs is obtained, and the time period for which the resource contention occurs is effectively determined.
In this embodiment, the resource pre-allocation result P _ R in step (1-2) is:
P_R＝{(P1,R1),(P2,R2),…,(Pt,Rt) In which P isiFor a certain period of time, RiThe sum of the number of computing resources required for a job running within a certain time period;
in this embodiment, in the total number of computing resources required by a job running in a certain time period, the present invention defines the states that may occur in each time period in the resource pre-allocation result P _ R:
and (3) normal load state: in the resource pre-allocation result P _ R, for a certain period P, the total number R of computing resources required by the job running in the period P is less than the total number TS of computing resources of the big data platform, the state of the period is referred to as a normal load state, and the period in the normal load state is referred to as a normal period.
Overload state: in the resource pre-allocation result P _ R, for a certain period P, if the total number R of computing resources required by the job running in the period P is greater than the total number TS of computing resources of the large data platform, the state of the period is referred to as an overload state, and the period in the overload state is referred to as an overload period.
In an overload time period, Computing Resources (Computing Resources) of a large data platform cannot meet the requirements of all jobs, resource contention occurs among the jobs, and finally the jobs are finished in a delayed mode.
In this embodiment, in the step (1-3), it is determined whether an overload time period exists in the resource pre-allocation result P _ R, that is, it is determined whether the total number of computing resources required by the job running in a certain time period is greater than the total number TS of computing resources of the large data platform; if the time interval is larger than the normal load time interval, the time interval is an overload time interval, otherwise, the time interval is a normal load time interval.
The Total Number of Computing Resources (TS) of the big data platform is the Number of all standard containers in the big data platform.
For example, in a system comprising 1 master node and NdnIn the big data platform of each slave node, the configuration of each slave node is c-core CPU and mG memory, and the size of each Container is set as c1Nuclear CPU \ m1G, memory, the total number of computing resources of the big data platform is:
TS＝min(c/c1,m/m1)*Ndn
wherein, min (c/c)1,m/m1) The maximum number of containers owned for each compute node.
Aiming at the overload time period, an optimal adjustment strategy needs to be found to adjust the overload time period and ensure that the platform gains are maximum:
in this embodiment, the specific steps of the steps (1-4) are as follows:
(1-4-1) selecting the last overload time period, and acquiring all the jobs executed in the time period to form a job set Ju;
(1-4-2) selecting a minimum proper subset from the job set Ju
The minimum proper subset Js advances the initial latest start time of all the jobs in the job set Ju to a normal load time period in which the time period is in a normal load state, and all the satisfaction items are setThe set of the smallest proper subset Js of pieces constitutes the set of feasible tuning strategies for that time period CL ═ Js1,Js2,…,Jsm}；
(1-4-3) evaluating the set of feasible tuning strategies CL ═ Js according to an evaluation function1,Js2,…,JsmEvaluating value of each feasible adjustment strategy in the data acquisition system, and selecting the optimal adjustment strategy;
(1-4-4) according to the optimal adjustment strategy, adjusting the initial latest starting time of the operation in the strategy, and updating the resource pre-allocation result P _ R according to the adjusted initial latest starting time of the operation.
To find the optimal adjustment strategy, first define a set of feasible adjustment strategies as follows:
and (3) feasible adjustment strategy set: in the resource pre-allocation result P _ R based on the operation deadline, for the operation set Ju running in a certain overload time period, a minimum proper subset is selected from the operation set Ju
The latest start time of all the jobs in the Js is advanced so that the total number of computing resources occupied by the jobs in the time slot does not exceed the total number TS of computing resources of the large data platform (i.e. the time slot is adjusted from the overload state to the normal load state), and such a minimum proper subset Js is called a feasible adjustment strategy. Set CL ═ Js of all Js satisfying the condition1,Js2,…,JsmThe set of feasible adjustment strategies referred to as this time period.
For a feasible tuning strategy set CL ═ { Js }1,Js2,…,JsmEvery job set Js iniTo adjust the overload state to the normal load state, Js should be adjustediThe latest start time of all jobs in (1) is advanced to:
wherein ji∈Jsi，TcsIs the start time of the timeout period.
For a time slot in an overload state, more than one feasible adjustment strategy is used, any feasible adjustment strategy is selected from the feasible adjustment strategy set to adjust the overload time slot, in order to ensure the maximum benefit of the large data platform, all feasible scheduling strategies in the feasible adjustment strategy set need to be subjected to benefit evaluation, and the optimal adjustment strategy is selected according to the evaluation result. The following two aspects are mainly considered in the evaluation:
(1) and (4) evaluating the profit.
To handle the timeout period (overload period), the present invention advances the latest start time of the job to ji.TlsWhen j isi.TlsWhen the time is less than 0, the current time point cannot guarantee that the operation can be completed before the deadline time, and at this time, some operations need to be properly abandoned, and the operation with a large profit evaluation value is preferentially executed to guarantee the maximum total profit of the platform. Therefore, the job that should preferentially ensure the profit evaluation value is large when the adjustment is performed according to the adjustment policy can be completed before the deadline, that is, before the feasible adjustment policy set CL { Js { (Js) }1,Js2,…,JsmIn the method, the optimal adjustment strategy should be to adjust the job set with the smallest total profit evaluation value. For any job, the profit estimate Sp for the job is the difference between the profit value a obtained when the job is completed on time and the profit b obtained when the job is not completed on time:
Sp＝|a-b|。
(2) and adjusting the cost.
When adjusting the operation, the invention not only needs to consider the profit evaluation value, but also needs to consider the influence of the number of resources occupied by the adjusted operation on the operation adjustment, namely the adjustment cost. Consider a case where there are two jobs j in a time period in a timeout stateaAnd jbAnd j isa.Sp＞jbSp, j should be set from the benefit point of view onlybTo ensure at least jaCan be completed. However due to jaThe required computing resources are very small and jbThe required computing resources are muchWill j isbThe last time before or after the last start time leads to the previous time period becoming overtime, and the previous time period needs to be continuously adjusted to finally lead to jbOr T for other worklsLess than 0 and thus discarded; and if j isaThe latest start time of is advanced is due to jaLess resources are occupied and all jobs can be completed before the deadline.
In summary, considering two factors, the invention provides an adjustment strategy evaluation function taking platform profit maximization as an optimization target, and an adjustment strategy with a smaller score is an optimal adjustment strategy, and the evaluation function is as follows:
wherein the content of the first and second substances,
to adjust the sum of the revenue estimates for all jobs in the strategy,
lastsize is the percentage of the total number of computing resources of the platform occupied by the remaining resources in the current time period after being adjusted according to the strategy.
In this process, the total profit needs to be calculated. Job set J ═ J received by service provider in big data platform1,j2,…,jnEach job in the queue has its own completion time constraint jiDl and revenue function j over timeiBf (t), actual completion time j of the jobiEnd at jiBefore or j dliDifferent earnings are obtained after dl, and for all jobs the total earnings calculation function is:
from the above description, the present invention proposes a maximum benefit scheduling algorithm based on the latest start time of a job, as shown below.
In algorithm 1, lines 1-3 execute step (1-1) to calculate the set J of jobs to be scheduled ═ J1,j2,…,jnEach job j iniInitial latest start time j ofi.TolsAnd according to the initial latest start time ji.TolsAnd performing resource pre-allocation on the resources of the platform.
And (4) executing the step (1-2), and counting the total number of the computing resources required by the operation of the job in each time period according to the allocation result of the resource pre-allocation to obtain a resource pre-allocation result P _ R.
And (5) lines 5 to 23 execute the steps (1-3) and (1-4), judge whether the overload time slot exists in the resource pre-allocation result P _ R, and if so, adjust the initial latest starting time of the time slot operation.
When the initial latest starting time of the overload time period job in the resource pre-allocation result P _ R in the step (1-4) is specifically adjusted, the 6 th to 7 th rows execute the step (1-4-1), the 6 th row selects the last overload time period, and the 7 th row acquires a job set Ju executed in the time period; line 8 performs step (1-4-2), and according to the job set Ju, combines the set of all the smallest proper subsets Js that satisfy the condition into the feasible adjustment policy set CL ═ Js for the time period { Js }1,Js2,…,Jsm}; the 9 th to 16 th lines execute the step (1-4-3), evaluate the evaluation value of each feasible adjustment strategy according to the evaluation function and select the optimal adjustment strategy; the step (1-4-4) is executed in the lines 17-23, after the optimal adjustment strategy is selected, the latest starting time adjustment is carried out on the operation in the strategy, and the P _ R is updated according to the adjustment result;
lines 6-23 are cycled until there is no overload period in P _ R.
Line 24 performs steps (1-5) outputting P _ R as a return result.
In algorithm 1, there are two loops outside and one loop inside, so the time complexity of algorithm 1 is n + n (n + n), i.e. 2n2+n。
And a second stage: job scheduling based on maximum resource utilization of platform
As shown in fig. 4, the specific steps of the job scheduling based on the platform maximum resource utilization rate in step (2) are as follows:
(2-1) initializing a variable final scheduling result queue and T time;
(2-2) searching a job set which can be executed at the time T and does not conflict with the resource pre-allocation result P _ R;
(2-3) judging whether the job set which does not conflict with the resource pre-allocation result P _ R is an empty set or not, and if the job set is the empty set, setting the time T as the initial time of the next time period in the resource pre-allocation result P _ R; if not, selecting the operation which minimizes the resource waste rate in the operation set to obtain the optimal starting time and updating the time T;
(2-4) repeatedly executing the step (2-2) -the step (2-3) until an optimal starting time is set for each job, and obtaining a final scheduling result;
and (2-5) outputting a final scheduling result.
In this embodiment, the resource wasting rate in the step (2-3) is a ratio between the non-reusable resource after scheduling and the currently used computing resource;
the currently used computing resource is the sum of the used computing resource and the free computing resource.
Through a maximum profit scheduling algorithm based on the latest starting time of the operation in the first stage, the invention obtains a resource pre-allocation result P _ R which enables the platform profit to be maximum, and the operation execution according to the result can ensure that the operation with large profit is completed before the deadline, but cannot ensure that the resource utilization rate of the platform is the highest. Therefore, the second stage of the two-stage job scheduling method facing the big data platform of the invention provides a secondary adjustment scheduling algorithm based on the maximum resource utilization rate of the platform, and the resource utilization rate of the platform is maximized on the premise of ensuring the maximum benefit of the platform.
In the scheduling algorithm of the second stage, the invention considers the resource pre-allocation result obtained according to the algorithm of the first stage to ensure the on-time completion of all the jobs, and simultaneously considers the resource utilization rate of the scheduled platform. To maximize platform resource utilization, the present invention uses a wasted resource rate (W)rr) Evaluating the utilization rate of computing resources and wasting the resource rate WrrI.e. resource W is not available for reuse after schedulingrWith currently used computing resource Ar(the sum of the computational used computational resources and the free computational resources), i.e.:
waste resource rate WrrSmaller means the resource utilization of the current platform is the greatest.
From the above description, a job scheduling algorithm based on the maximum resource utilization of the platform is proposed herein, as follows:
in the algorithm 2, the algorithm is described,
lines 1-2 execute step (2-1), initialize variables;
4-10 lines of execution step (2-2), find job set E that can be executed at time T and does not conflict with pre-allocated resourcesj；
Lines 11-20 perform step (2-3), line 11 determines whether the job set that does not conflict with the resource pre-allocation result P _ R is an empty set,
lines 12-18, if EjIf not empty, then at EjSelecting the operation which minimizes the resource waste rate and setting the optimal starting time as the current time T;
executing 3-20 lines circularly until an optimal starting time is set for each job;
line 21 performs step (2-5), returning the result.
There are two cycles in algorithm 2, so the time complexity is n 2.
After the secondary adjustment scheduling, the starting time of one job is determined for each job in the job set to be scheduled, the resource utilization rate of the job is increased after the secondary adjustment scheduling, the actual starting time and the finishing time of the job are advanced, and a lot of jobs abandoned due to insufficient resources in a resource pre-allocation algorithm obtain new execution opportunities, so after the secondary adjustment scheduling, the benefit of a cluster is increased once again.
Example 2:
in order to fully utilize the resources of the platform and improve the profit of the service provider in the big data platform, the invention provides a two-stage job scheduling system for the big data platform, which is based on the two-stage job scheduling method for the big data platform in embodiment 1.
In order to achieve the purpose, the invention adopts the following technical scheme:
as shown in figure 5 of the drawings,
a big data platform-oriented two-stage job scheduling system, comprising:
the system comprises a first-stage scheduling module, a second-stage scheduling module and a scheduling module, wherein the first-stage scheduling module is used for forming a job set to be scheduled by jobs submitted by a user, each job in the job set has own SLA information, and based on the latest starting time constraint of each job and the total maximum income of a service provider, the maximum income based on the latest starting time of the job is used for scheduling the jobs in the job set to be scheduled for the first time to perform adjustment scheduling so as to obtain a resource pre-allocation job scheduling result queue which enables the income of the service provider to be maximum;
and
and the second-stage scheduling module is used for carrying out secondary adjustment scheduling aiming at improving the resource utilization rate of the platform on the premise of ensuring the maximum profit of a service provider according to the resource use condition of the platform, carrying out micro-adjustment on the scheduling result queue of the resource pre-allocation job of the first-stage scheduling module to obtain a final scheduling result queue, ensuring that the resource utilization rate of the platform is the highest on the premise of ensuring the maximum profit of the platform by the final scheduling result queue, fully utilizing the resources of the platform, and ensuring that each job zone in the final scheduling result queue has the optimal starting time.
In this embodiment, the first scheduling module includes a first job scheduler and a first resource scheduler, and the second scheduling module includes a second job scheduler and a second resource scheduler. The scheduling of the jobs is completed under the assistance of a job scheduler, and after the job scheduler generates an optimal job execution queue, the jobs are started according to the optimal starting time of each job; after the jobs are started, the resource scheduler starts to allocate resources for each job and executes, as shown in the lower part of fig. 5, the jobs may acquire resources from different nodes and may run on different nodes.
Example 3:
in this embodiment 3, the two-stage job scheduling method and system for large data platforms in embodiments 1 and 2 of the present invention are compared with the EDF algorithm and the original FIFO scheduling algorithm in Hadoop, respectively, in terms of the influence of each factor, such as the average job size, the job set size, the total number of platform resources, and the job urgency degree, on the algorithm.
Platform configuration:
in this example, the present invention and comparative examples 1 and 2 were tested in a big data platform based on the MapReduce computing framework.
The platform comprises 1 main node and 20 slave nodes with the same configuration information. The configuration information of each node is CPUInter (R) core (TM) i 5-24003.10 GHz, memory 8GB, hard disk 1TB, Red Hat Enterprise Linux6.2Systems, and Hadoop version is 2.7.1. We use the number of containers to represent the number of computing resources, each Container is 1 core and 2G memory, so that there are 4 containers on each node, and there are 80 containers in the whole platform.
And (3) comparison algorithm:
in order to verify the effectiveness of the two-stage job scheduling method and system for the big data platform in the embodiments 1 and 2 of the present invention, the two-stage scheduling algorithm-TPS (two-phase scheduling) provided by the present invention is compared with the EDF algorithm and the original FIFO scheduling algorithm in Hadoop in the experiment.
The EDF algorithm is a classical job scheduling algorithm aiming at cut-off time constraint, and the principle of the algorithm is to determine the execution sequence of jobs based on the cut-off time in sequence and preferentially execute the jobs with early cut-off time. In the execution process, in order to avoid the prolongation of the execution time of the job caused by the resource contention among the jobs, in each comparative example of the embodiment, the job is started when the remaining resources in the platform are not less than the resource requirement of the job.
In the FIFO scheduling algorithm, the jobs are executed according to the order of the priority (or the order of the submission time), and the jobs cannot be executed in parallel. In the tests herein, jobs would be prioritized by the size of the benefit ratio (the ratio of benefit value to resource demand) for comparison with the algorithms presented herein.
Test jobs and data:
the experiment was tested using classical MapReduce jobs Sort and Grep as input jobs.
Evaluation indexes are as follows:
in this embodiment, the algorithm is evaluated through three indexes, namely, the platform resource utilization rate, the operation completion rate and the platform profit, and the calculation formulas of the three indexes are as follows:
the platform resource utilization rate is the number of resources used for executing the job/the total number of platform resources.
The job completion rate is the number of jobs completed on time/total number of jobs to be scheduled.
The platform profit is the profit of the job completed on time-the amount of the claim paid for the job not completed on time.
Experiment design:
during the experiment, the influence of various factors such as the average operation size, the operation set scale, the total platform resource number and the operation urgency degree on the algorithm is considered, and the experiment analysis is respectively carried out:
in the experiment (1), in order to test the influence of the average job size on the algorithm, the number of data blocks of the job is used as a standard for evaluating the job size, the total number of the job is set to be 30, and the performance of the algorithm is tested when the number of the average data blocks of the job is 20, 40, 80, 100 and 200 respectively.
Experiment (2) is to test the influence of the scale of the operation set on the algorithm, the average size of the operation is set to be fixed to 40, and the performance of the algorithm is tested when the number of the operations in the operation set is 10, 20, 30 and 40 respectively.
Experiment (3) is to test the influence of the total number of platform resources on the algorithm, set the number of jobs and the average size to be respectively fixed at 20 and 40 by using the number of contacts as a standard for evaluating the total number of platform resources, and test the performance of the algorithm when the number of contacts is respectively 4, 8, 16, 32, 48, 64 and 80.
Experiment (4) to test the influence of the job urgency level on the algorithm, the ratio of the distance length of the job deadline to the job execution length was used as a criterion for evaluating the urgency level, and the larger the ratio, the lower the job urgency level. The number of the set jobs and the average size are respectively fixed at 20 and 40, and the performance of the algorithm when the average urgency degree is 3, 4, 5, 6 and 7 is respectively tested.
Because the scheduling algorithm in the invention has higher complexity and needs to consume a certain amount of time during scheduling, and FIFO scheduling and EDF scheduling almost have no scheduling time, the execution time of the algorithm is required to be taken into account during algorithm comparison.
And (3) analyzing an experimental result:
(1) effect of average Job size on scheduling Algorithm
As shown in fig. 6-8, this example first performed experiment (1) to test the effect of average job size on algorithm performance.
As shown in fig. 6, the experiment (1) tests the influence of the average job size on the resource utilization rate, and it can be known from the experimental results that when the number of jobs is small, the resource utilization rate of FIFO scheduling is low because jobs are executed serially, and the resource utilization rates of TPS and EDF algorithms are lower than those of TPS algorithms because the job parallel resource utilization rate is higher and because EDF algorithms only consider the cutoff time factor; with the increase of the size of the operation, the resource utilization rate of FIFO scheduling is increased, and the resource fragments generated by scheduling are increased due to the increase of the operation in the TPS algorithm and the EDF algorithm, so that the resource utilization rate is reduced; finally, when the job size reaches a certain degree, the platform resources cannot meet the parallel requirements of the jobs, and the jobs scheduled by the three algorithms are executed in series, so that the resource utilization rates of the three algorithms are approximately the same.
As shown in fig. 7, experiment (1) also tests the relationship between the job completion rate and the average job size, and since the completion time limit is not changed and the computing capacity of the system in a fixed time period is fixed, the job completion rates of the 3 algorithms are obviously reduced as the jobs increase, but since the EDF algorithm only considers the deadline factor and the TPS algorithm considers the platform gain, the job completion rate of the EDF algorithm is slightly higher than that of the TPS algorithm, and the FIFO algorithm only considers the gain, the job completion rate of the algorithm is the lowest.
As shown in fig. 8, experiment (1) also tests the influence of the average job size on the total profit, where the FIFO has the smallest early profit due to its low resource utilization rate, and although the EDF algorithm has a job completion rate slightly higher than that of the TPS algorithm, the TPS algorithm has a job profit slightly larger than that of the EDF algorithm because the TPS takes the job profit into consideration, and as the job size increases, the job completion rate decreases, the job profits of the three algorithms all decrease, and the profit of the TPS algorithm is always larger than that of the other two algorithms.
(2) Impact of Job set size on scheduling algorithms
As shown in fig. 9-11, this example performed experiment (2) to test the effect of job set size on scheduling algorithm performance.
As shown in fig. 9, the experiment (2) tests the influence of the job set scale on the resource utilization rate, and it can be seen from the experimental results that the calculation resource utilization rates of the three algorithms are not greatly influenced by the job set scale, and the resource utilization rates do not change significantly even though the job scale is increased. The TPS algorithm has the highest utilization rate, the EDF algorithm has the resource utilization rate slightly lower than that of the TPS algorithm, and the FIFO algorithm has the lowest resource utilization rate.
As shown in fig. 10, experiment (2) also tests the effect of the job set scale on the job completion rate, and since the calculation capability is limited, when the job set scale is increased, the job completion rates of the three algorithms are reduced, which is the same as the effect of the job size on the job completion rate and the total profit, and is not repeated here.
As shown in fig. 11, the experiment (2) also tests the influence of the scale of the flat job set on the profit, and it can be known from the experimental results that the profits of the three algorithms all show a trend of increasing first and then decreasing when the scale of the job set increases, because the number of jobs that can be completed by the platform increases when the number of jobs is within the platform computing capacity, so that more profits can be obtained, whereas the number of jobs that are not completed on time increases when the number of jobs increases as the number of jobs exceeds the platform computing capacity, because the number of uncompleted jobs obtains negative profits, the total profit decreases instead. The FIFO algorithm and the TPS algorithm in the three algorithms have the minimum job completion rate and the minimum profit due to low resource utilization rate, and the EDF algorithm and the TPS algorithm have the high resource utilization rate, so when the job number is in the computing capacity range of the platform, the job completion rate and the profit of the two algorithms are approximately the same, the EDF algorithm continuously increases along with the job number, the EDF algorithm only considers the cut-off time factor of the job, and the TPS algorithm preferentially completes the job with large profit, so the job completion rate of the EDF algorithm is slightly higher than that of the TPS algorithm, but the job profit of the TPS algorithm is far greater than that of the EDF algorithm.
(3) Impact of computing resource quantity on scheduling algorithm
As shown in fig. 12-14, this example performed experiment (3) to test the effect of job set size on scheduling algorithm performance.
As shown in fig. 12, the experiment (3) tests the influence of the number of computing resources on the resource utilization rate, and it can be known from the experimental results that when the number of computing resources is small, the resource waste rates of the three algorithms are substantially the same and decrease as the number of computing resources increases, however, when the number of computing resources decreases to a certain extent, the resource utilization rate of the FIFO algorithm continues to decrease as the number of computing resources increases, while the resource utilization rates of the EDF and the TPS increase as the number of computing resources increases, and the TPS rise is larger than the rise of the EDF. The reason for the above phenomenon is that when the number of computing resources is small, the jobs scheduled by the three algorithms are all executed in series basically, so the resource utilization rate is similar and is reduced along with the increase of resources; when the number of containers is continuously increased to be larger than the average job size, the jobs scheduled by the EDF and TPS algorithms can be executed in parallel, so that the resource utilization rate is increased.
As shown in fig. 13, experiment (3) also tested the effect of the number of computing resources on the job completion rate; as shown in fig. 14, the experiment (3) also tests the influence of the number of computing resources on the profit, and it can be known from the experiment result that the operation completion rate and the profit of the TPS algorithm and the EDF algorithm are both increased with the increase of the number of computing resources, and the profit of the TPS algorithm is much greater than that of the EDF algorithm; the FIFO algorithm increases the job completion rate and the profit with the increase of the resources when the platform resources are small, however, after the platform resources are larger than the average job size, since the jobs are executed serially, a large amount of resources are wasted, and the job completion rate and the profit are not increased with the increase of the resources but are stabilized at a fixed value.
(4) Impact of job urgency on scheduling algorithms
As shown in fig. 15, this example performed experiment (4) to test the influence of the degree of urgency of the work on the three algorithms. Because the operation has more time to be executed under the condition that the information such as the operation quantity is not changed along with the reduction of the urgency degree, the completion rate and the yield of the three algorithms are increased inevitably, and the experiment only tests the influence of the operation urgency degree on the resource utilization rate. From the experimental results in the figure, it can be known that the resource utilization of the FIFO algorithm and the EDF algorithm is not substantially affected by the urgency of the job, while the resource utilization of the TPS algorithm increases as the urgency becomes lower. The reason is that the FIFO algorithm and the EDF algorithm only concern the profit of the operation or the precedence relationship of the cut-off time between the operations, while the TPS algorithm considers the utilization rate of the resources on the premise of meeting the profit, and when the urgency degree of the operation becomes low, the degree of freedom of the operation in the resource utilization rate maximization algorithm is higher, so that the start time of the operation can be better adjusted to maximize the resource utilization rate.
The invention has the beneficial effects that:
the invention discloses a two-stage job scheduling method and system for a big data platform, and provides the two-stage job scheduling system and method for jobs with deadline constraints based on a MapReduce calculation framework. The scheduling method calculates and adjusts the latest starting time of each job according to the deadline constraint of the job and the profit information of the job, and performs resource pre-allocation according to the adjustment result to ensure that the job with high profit can be completed before the deadline so as to maximize the total profit of the platform; in the second stage, on the premise of ensuring the maximum platform benefit, a job scheduling method based on the maximum platform resource utilization rate is provided to improve the platform resource utilization rate. Experimental results show that the two-stage operation scheduling method provided by the invention not only realizes the maximization of platform profit, but also improves the resource utilization rate of the platform and the comprehensive performance of the platform.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (8)
1. A two-stage job scheduling method facing a big data platform is characterized in that: the method comprises the following steps:
(1) forming a job set to be scheduled by the job submitted by the user, and scheduling based on the maximum profit of the latest start time of the job: pre-allocating resources of the platform according to the deadline of the operation, and adjusting and scheduling the result of the pre-allocation of the resources according to the profit comparison of the operation to obtain a resource pre-allocation operation scheduling result queue which enables the profit of a service provider to be the maximum;
(2) job scheduling based on the maximum resource utilization rate of the platform: according to the resource use condition of the platform, carrying out micro-adjustment on the scheduling result queue of the resource pre-allocation job in the step (1) to obtain a final scheduling result queue, and ensuring that the resource utilization rate of the platform is the highest on the premise of ensuring the maximum platform benefit;
the specific steps of the job scheduling based on the platform maximum resource utilization rate in the step (2) are as follows:
(2-1) initializing a variable final scheduling result queue and T time;
(2-2) searching a job set which can be executed at the time T and does not conflict with the resource pre-allocation result P _ R;
(2-3) judging whether the job set which does not conflict with the resource pre-allocation result P _ R is an empty set or not, and if the job set is the empty set, setting the time T as the initial time of the next time period in the resource pre-allocation result P _ R; if not, selecting the operation which minimizes the resource waste rate in the operation set to obtain the optimal starting time and updating the time T;
(2-4) repeatedly executing the step (2-2) -the step (2-3) until an optimal starting time is set for each job, and obtaining a final scheduling result;
(2-5) outputting a final scheduling result;
the resource waste rate in the step (2-3) is the ratio of the non-reusable resources after scheduling to the currently used computing resources;
the currently used computing resource is the sum of the used computing resource and the free computing resource.
2. The big data platform-oriented two-stage job scheduling method according to claim 1, wherein: the specific steps of the maximum benefit scheduling based on the latest start time of the job in the step (1) are as follows:
(1-1) calculating the initial latest starting time of each job in the job set to be scheduled, and performing resource pre-allocation on the resources of the platform according to the initial latest starting time;
(1-2) counting the total number of the calculation resources required by each time period according to the allocation result of the resource pre-allocation to obtain a resource pre-allocation result P _ R;
(1-3) judging whether an overload time period exists in the resource pre-allocation result P _ R, if so, entering the step (1-4), and if not, entering the step (1-5);
(1-4) adjusting the initial latest starting time of the overload time period operation in the resource pre-allocation result P _ R, updating the P _ R according to the adjustment result, and returning to the step (1-3);
and (1-5) outputting a resource pre-allocation result P _ R which maximizes the profit of the service provider.
3. The big data platform-oriented two-stage job scheduling method according to claim 2, wherein: in the step (1-1), the specific steps of performing resource pre-allocation on the platform resources according to the initial latest start time are as follows:
letting each job start execution at its initial latest start time, obtaining the number of resources required for each time period when all jobs have just completed at their deadline;
the initial latest start time of the job is: for any job in the set of jobs to be scheduled, the job start time at which the job can be completed at the deadline point in time when no other jobs contend for the resource.
4. The big data platform-oriented two-stage job scheduling method according to claim 2, wherein: in the step (1-3), whether an overload time period exists in the resource pre-allocation result P _ R is judged, that is, whether the total number of computing resources required by the operation running in a certain time period is greater than the total number of computing resources of the large data platform is judged; if the time interval is larger than the normal load time interval, the time interval is an overload time interval, otherwise, the time interval is a normal load time interval;
the total number of computing resources of the big data platform is the number of all standard containers in the big data platform.
5. The big data platform-oriented two-stage job scheduling method according to claim 2, wherein: the steps (1-4) are as follows:
(1-4-1) selecting the last overload time period, and acquiring all the jobs executed in the time period to form a job set;
(1-4-2) selecting a minimum proper subset in the operation set, wherein the minimum proper subset leads the initial latest starting time of all the operations in the operation set to a normal load time period which enables the time period to be in a normal load state, and the set of all the minimum proper subsets meeting the conditions forms a feasible adjustment strategy set of the time period;
(1-4-3) evaluating the evaluation value of each feasible adjustment strategy in the feasible adjustment strategy set according to the evaluation function, and selecting the optimal adjustment strategy;
(1-4-4) according to the optimal adjustment strategy, adjusting the initial latest starting time of the operation in the strategy, and updating the resource pre-allocation result P _ R according to the adjusted initial latest starting time of the operation.
6. The big data platform-oriented two-stage job scheduling method according to claim 5, wherein: the evaluation function in the step (1-4-3) is:
wherein the content of the first and second substances,
to adjust the sum of the revenue estimates for all jobs in the strategy,
lastsize is the sum of the profit evaluation values of all the jobs in the current time period, and the remaining resources in the current time period account for the platform after being adjusted according to the strategyAnd calculating the percentage of the total number of resources, wherein Sp is the income assessment value of the operation, Sp is | a-b |, a is the income value obtained when the operation is completed on time, and b is the income value obtained when the operation is not completed on time.
7. A big data platform-oriented two-stage job scheduling system based on the big data platform-oriented two-stage job scheduling method according to any one of claims 1 to 6, characterized in that: the system comprises:
the system comprises a first-stage scheduling module, a second-stage scheduling module and a scheduling module, wherein the first-stage scheduling module is used for forming a job set to be scheduled by jobs submitted by a user, and performing first adjustment scheduling on jobs in the job set to be scheduled by adopting maximum profit scheduling based on the latest start time of each job and the total maximum profit of a service provider to obtain a resource pre-allocation job scheduling result queue which enables the profit of the service provider to be maximum;
and
the second-stage scheduling module is used for carrying out micro-adjustment on a resource pre-allocation job scheduling result queue of the first-stage scheduling module according to the resource use condition of the platform to obtain a final scheduling result queue, the resource utilization rate of the platform is maximized on the premise that the final scheduling result queue ensures the maximum platform profit, the resources of the platform can be fully utilized, and each job zone in the final scheduling result queue has the optimal starting time;
the specific steps of job scheduling based on the maximum resource utilization rate of the platform are as follows:
(2-1) initializing a variable final scheduling result queue and T time;
(2-2) searching a job set which can be executed at the time T and does not conflict with the resource pre-allocation result P _ R;
(2-3) judging whether the job set which does not conflict with the resource pre-allocation result P _ R is an empty set or not, and if the job set is the empty set, setting the time T as the initial time of the next time period in the resource pre-allocation result P _ R; if not, selecting the operation which minimizes the resource waste rate in the operation set to obtain the optimal starting time and updating the time T;
(2-4) repeatedly executing the step (2-2) -the step (2-3) until an optimal starting time is set for each job, and obtaining a final scheduling result;
(2-5) outputting a final scheduling result;
the resource waste rate in the step (2-3) is the ratio of the non-reusable resources after scheduling to the currently used computing resources;
the currently used computing resource is the sum of the used computing resource and the free computing resource.
8. The big data platform-oriented two-stage job scheduling system of claim 7, wherein: the set of jobs to be scheduled is: the platform service provider receives a batch of jobs in a certain time period and negotiates with a user to sign an SLA protocol related to the jobs, and the set of the jobs is a set J of jobs to be scheduled, which is expressed as J ═ J1,j2,…,jnN is the number of the operations in J;
for any job J in job set J to be schedulediIs represented by ji(ms, rs, mt, rt, dl, bf (t)), wherein ms is the number of Map tasks for the job; rs is the Reduce task number of the operation; mt is the average execution time of the operation Map task; rt is the average execution time of the Reduce task of the job; dl is the deadline constraint for the job; bf (t) is a revenue function of the job;
the first stage scheduling module comprises a first job scheduler and a first resource scheduler, and the second stage scheduling module comprises a second job scheduler and a second resource scheduler.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710590748.5A CN107589985B (en) | 2017-07-19 | 2017-07-19 | Two-stage job scheduling method and system for big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710590748.5A CN107589985B (en) | 2017-07-19 | 2017-07-19 | Two-stage job scheduling method and system for big data platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107589985A CN107589985A (en) | 2018-01-16 |
| CN107589985B true CN107589985B (en) | 2020-04-24 |
Family
ID=61041646
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710590748.5A Active CN107589985B (en) | 2017-07-19 | 2017-07-19 | Two-stage job scheduling method and system for big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107589985B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112328383A (en) * | 2020-11-19 | 2021-02-05 | 湖南智慧畅行交通科技有限公司 | Priority-based job concurrency control and scheduling algorithm |
| CN114690719B (en) * | 2021-12-21 | 2024-05-14 | 中国科学院沈阳自动化研究所 | Platform-oriented efficient information physical production system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778079A (en) * | 2014-01-10 | 2015-07-15 | 国际商业机器公司 | Method and device used for dispatching and execution and distributed system |
| CN105159769A (en) * | 2015-09-11 | 2015-12-16 | 国电南瑞科技股份有限公司 | Distributed job scheduling method suitable for heterogeneous computational capability cluster |
| CN105721565A (en) * | 2016-01-29 | 2016-06-29 | 南京邮电大学 | Game based cloud computation resource allocation method and system |
| CN105718316A (en) * | 2014-12-01 | 2016-06-29 | 中国移动通信集团公司 | Job scheduling method and apparatus |
| CN105740051A (en) * | 2016-01-27 | 2016-07-06 | 北京工业大学 | Cloud computing resource scheduling realization method based on improved genetic algorithm |
| CN106293893A (en) * | 2015-06-26 | 2017-01-04 | 阿里巴巴集团控股有限公司 | job scheduling method, device and distributed system |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8752036B2 (en) * | 2011-10-31 | 2014-06-10 | Oracle International Corporation | Throughput-aware software pipelining for highly multi-threaded systems |
| CN103077438A (en) * | 2012-12-27 | 2013-05-01 | 深圳先进技术研究院 | Control method and system for scheduling multiple robots |
| CN104349459B (en) * | 2013-07-25 | 2019-04-19 | 索尼公司 | Method, base station and the terminal of dynamic uplink configuration are carried out in a wireless communication system |
| CN104317650B (en) * | 2014-10-10 | 2018-05-01 | 北京工业大学 | A kind of job scheduling method towards Map/Reduce type mass data processing platforms |
| CN104731662B (en) * | 2015-03-26 | 2018-02-23 | 华中科技大学 | A kind of resource allocation methods of variable concurrent job |
| CN105808334B (en) * | 2016-03-04 | 2016-12-28 | 山东大学 | A kind of short optimization of job system and method for MapReduce based on resource reuse |
-
2017
- 2017-07-19 CN CN201710590748.5A patent/CN107589985B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778079A (en) * | 2014-01-10 | 2015-07-15 | 国际商业机器公司 | Method and device used for dispatching and execution and distributed system |
| CN105718316A (en) * | 2014-12-01 | 2016-06-29 | 中国移动通信集团公司 | Job scheduling method and apparatus |
| CN106293893A (en) * | 2015-06-26 | 2017-01-04 | 阿里巴巴集团控股有限公司 | job scheduling method, device and distributed system |
| CN105159769A (en) * | 2015-09-11 | 2015-12-16 | 国电南瑞科技股份有限公司 | Distributed job scheduling method suitable for heterogeneous computational capability cluster |
| CN105740051A (en) * | 2016-01-27 | 2016-07-06 | 北京工业大学 | Cloud computing resource scheduling realization method based on improved genetic algorithm |
| CN105721565A (en) * | 2016-01-29 | 2016-06-29 | 南京邮电大学 | Game based cloud computation resource allocation method and system |
Non-Patent Citations (3)
| Title |
|---|
| An Ant Colony Optimization for Grid Task Scheduling with Multiple QoS Dimensions;J Hu等;《2009 Eighth International Conference on Grid and Cooperative Computing》;20090829;第415-419页 * |
| MapReduce集群中最大收益问题的研究;王习特等;《计算机学报》;20150131;第38卷(第1期);第109-121页 * |
| 基于最小费用最大流的大规模资源调度方法;陈晓旭等;《软件学报》;20161129;第598-610页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107589985A (en) | 2018-01-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10474504B2 (en) | Distributed node intra-group task scheduling method and system | |
| CN108345501B (en) | Distributed resource scheduling method and system | |
| US8997107B2 (en) | Elastic scaling for cloud-hosted batch applications | |
| CN107911478B (en) | Multi-user calculation unloading method and device based on chemical reaction optimization algorithm | |
| US9575691B2 (en) | Scheduling user requests in a distributed resource system having a plurality of schedulers and coordinators | |
| US9218213B2 (en) | Dynamic placement of heterogeneous workloads | |
| CN112416585B (en) | Deep learning-oriented GPU resource management and intelligent scheduling method | |
| CN111782355B (en) | Cloud computing task scheduling method and system based on mixed load | |
| CN108123998B (en) | Heuristic request scheduling method for delay sensitive application in multi-cloud data center | |
| CN114610474B (en) | Multi-strategy job scheduling method and system under heterogeneous supercomputing environment | |
| CN108428051B (en) | MapReduce job scheduling method and device facing big data platform and based on maximized benefits | |
| US20140137122A1 (en) | Modified backfill scheduler and a method employing frequency control to reduce peak cluster power requirements | |
| CN114968510A (en) | Multi-target dynamic task scheduling method and system based on improved ant colony algorithm | |
| CN109710372B (en) | Calculation intensive cloud workflow scheduling method based on owl search algorithm | |
| CN113821318A (en) | Internet of things cross-domain subtask combined collaborative computing method and system | |
| CN107589985B (en) | Two-stage job scheduling method and system for big data platform | |
| Derakhshan et al. | Optimization of tasks in cloud computing based on MAX-MIN, MIN-MIN and priority | |
| CN112860390A (en) | Job scheduling program, information processing apparatus, and job scheduling method | |
| CN108958919B (en) | Multi-DAG task scheduling cost fairness evaluation method with deadline constraint in cloud computing | |
| Karmakar et al. | Resource scheduling for tasks of a workflow in cloud environment | |
| CN112306642B (en) | Workflow scheduling method based on stable matching game theory | |
| Du et al. | Energy-efficient scheduling for best-effort interactive services to achieve high response quality | |
| CN104731662B (en) | A kind of resource allocation methods of variable concurrent job | |
| Chen et al. | Deadline-constrained MapReduce scheduling based on graph modelling | |
| CN110088730A (en) | Task processing method, device, medium and its equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |