CN107797863A - Fine granularity reso urce matching method in a kind of cloud computing platform - Google Patents
Fine granularity reso urce matching method in a kind of cloud computing platform Download PDFInfo
- Publication number
- CN107797863A CN107797863A CN201710909672.8A CN201710909672A CN107797863A CN 107797863 A CN107797863 A CN 107797863A CN 201710909672 A CN201710909672 A CN 201710909672A CN 107797863 A CN107797863 A CN 107797863A
- Authority
- CN
- China
- Prior art keywords
- resource
- mrow
- task
- msub
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5033—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering data affinity
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses fine granularity reso urce matching method in a kind of cloud computing platform, belong to computer realm, execution logic same task speculates task execution time and various resources requirements in being loaded based on cloud computing, the problems such as resource fragmentation, over-allocation are avoided according to task resource demand division tasks execution phases and simulation stage by stage resource requirement, computing resource is abstracted as compressible and incompressible two types during resource matched, and compressed resource distribution further improves resource utilization when necessary.The present invention has relatively low scheduling average response time while distributing the time, the aspect of stock number two improves matching granularity.The present invention can be applied in each cloud computing platform resource management, job scheduling etc., the problems such as avoiding resource fragmentation and over-allocation, improve the utilization ratio of computing resource in platform, the final overall throughput for lifting cloud computing platform.
Description
Technical field
The invention belongs to computer realm, is related to the resource management in cloud computing platform and job scheduling method, specifically relates to
A kind of and fine granularity reso urce matching method in cloud computing platform.
Background technology
Cloud computing is a kind of calculation based on internet, and in this way, user can obtain calculating money on demand
Source, computing capability.The infrastructure of cloud computing platform is typically formed by many computer nodes by high performance network interconnection, its
By numerous node organizations into high-performance, High Availabitity, expansible single image, there is provided give user to use.
A kind of resource (be typically internal memory) is typically based in the resource management of cloud computing platform or by two kinds of fixed amounts
Computing resource binding is defined as slot, and the unit of resource allocation is used as using slot.Due to loading the need to resource in cloud computing
To ask with diversity, the resource distribution mode based on fixed unit is easy to produce resource fragmentation and cause the wasting of resources, or because
Trigger bad resource-sharing for over-allocation.The bad shared competitive resource between causing task of some resources (such as CPU),
The execution time is significantly increased.In the data center, the backwardness task more than 53% is by the high utilization of resources of bad shared initiation
Caused by rate, and 4%-6% improper task affects 37%-49% operation, causes the operation deadline significantly
Extend.And the over-allocation of some resources (such as internal memory) can directly result in mission failure or server crash.
The cloud computing resources such as existing Yarn, Fuxi, Borg management platform and Apollo, Omega, Tetris, DRF,
Resource bid distribution fixed amount resource of the cloud computing such as Carbyne dispatching algorithm based on operation, to avoid resource fragmentation, excessively divide
With the problems such as.Because the resource bid amount of operation is generally by artificially specifying, very big difference be present between resource bid and actual use
It is different.In addition, the resource usage amount fluctuation of task is very big, peak value will not be always held at, operation is used using task maximum resource
When amount is used as applications, still had differences between resource bid amount and actual use amount.Therefore resource management platform or tune
When degree device is allocated according to resource bid amount, resource fragmentation is still present.The method of salary distribution based on resource bid is difficult to obtain
High resource utilization, which limit the resource utilization of cluster to a certain extent.
Therefore, how resource fragmentation and over-allocation are avoided in cloud computing resources management and job scheduling, becomes cloud
Major issue in calculating platform research.
The content of the invention
It is an object of the invention to provide fine granularity reso urce matching method in a kind of cloud computing platform, and this method is in distribution
Between, the aspect of stock number two is while improve matching granularity, there is relatively low scheduling average response time, platform can be effectively improved
The utilization ratio of middle computing resource, lift the overall throughput of cloud computing platform.
The present invention is to be achieved through the following technical solutions：
The invention discloses fine granularity reso urce matching method in a kind of cloud computing platform, comprise the following steps：
Step 1：The role of server in cloud computing platform is divided into two kinds of calculation server and management server, calculates clothes
Business device is responsible for the execution specifically loaded, and periodically reports resource status to management server；Management server is responsible for whole cloud meter
The management work of platform is calculated, including calculating task is distributed to calculation server；
Step 2：Management server receives the information that each calculation server is periodically reported, and according to similar tasks and resource pressure
Shrinkage speculates various resource requirements and the duration of some task；
Wherein, similar tasks refer in load has identical execution logic and input data amount identical with some described task
Task；
Step 3：Management server analysis speculates the CPU of each progress of task drawn, memory headroom resource requirement, by task
It is divided into multiple execution stages；
Step 4：Management server is from treating to select task in scheduling set, simulation stage by stage task resource demand and server
Available computational resources, and compression fit resource requirement as needed；
Step 5：If the match is successful for resource requirement, management server assumes that resource is the allocated, checks the calculating service
Whether all tasks can be affected and can not meet constraints on device, if all task restriction conditions are satisfied by, distribute
Computing resource.
Preferably, following operation is also included after step 5：Whether management server checks remaining enough on the calculation server
Resource matches into next round, if surplus resources meet condition, is matched into next round, that is, repeats step 4 and step
Rapid 5.
Preferably, in step 1, calculation server is responsible for the execution specifically loaded, and periodically reports and provide to management server
Source state, in particular to：Task resource service condition in the operation of calculation server taken at regular intervals book server, calculates book server
Available resource information is simultaneously reported to management server；
Wherein, the available volume of resources r of certain resource is calculated by formula (1) on calculation server：
In formula, riFor the stock number of ith sampling, tiThe duration sampled for ith, when T is that the calculating of sampling is total
Between, n is sampling number.
Preferably, in step 2, speculate the various resource requirements of some task according to similar tasks and resource compression ratio and continue
Time, resource requirement type include CPU, memory headroom, disk space, Disk bandwidth and network bandwidth；
Wherein, the disk of task and network bandwidth resources demand are divided into three classes according to the relative position of task and data
Speculated, the first kind：Task is with data in same server；Second class：Task is with data in same frame；3rd class：Other；
Concrete operations are：
Task is calculated in the resource requirement needed for certain progress and duration by formula (2)：
In formula, αnFor n-th estimation result, βnFor the similar tasks resource information of n-th, rcFor the resource of mission bit stream
Compression ratio, ThrFor maximum compression rate limit coefficient, e is the nature truth of a matter.
Preferably, step 3 concrete operations are：
Management server travels through CPU, memory headroom resources requirement of the task in each progress, and records both moneys respectively
The maximum and minimum value of source amount；
Threshold value is divided when the difference of the maximum and minimum value of CPU or memory headroom resources requirement is more than, and is traveled through
Progress reach partition length, then the progress of traversal is divided into an execution stage of task；
Tasks execution phases CPU, memory headroom and disk space resource requirement value are each progress money in the execution stage
The maximum of source demand, Disk bandwidth, the requirements of network bandwidth resources are the flat of each progress resource requirement in the execution stage
Average.
It is further preferred that referring to Fig. 1, the division methods of tasks execution phases, following steps are specifically included：
1)P、Ps、PeTask Progress is represented respectively, the stage starts progress and the stage terminates progress, Cmax、Cmin、MmaxAnd Mmin
CPU, the maximum and minimum value of memory headroom resource requirement in the expression stage respectively；Wherein, P, Ps、PeIt is initialized as 0；Cmin、
MminFor 100, Cmax、MmaxIt is initialized as 0；
2) Task Progress P and stage terminate progress PeIncrease by 1, if Task Progress P reaches 100%, by PsTo 100%
It is divided into after the new stage and terminates, otherwise continues step 3)；
If 3) current schedules cpu demand CpMore than Cmax, then by CmaxValue be updated to Cp, into step 5)；
If 4) current schedules cpu demand CpLess than Cmin, then by CminValue be updated to Cp；
If 5) current schedules memory requirements MpMore than Mmax, then by MmaxValue be updated to Mp, into step 7)；
If 6) current schedules memory requirements MpLess than Mmin, then by MminValue be updated to Mp；
If 7) CmaxWith CminDifference be more than cpu resource total amount C and threshold value ThcProduct or MmaxWith MminDifference
More than memory amount M and threshold value ThmProduct, then into step 8, otherwise into step 2)；
If 8) PeWith PsDifference be more than threshold value Thp, then by PsTo PeThe new stage is divided into, by PsIt is updated to Pe, again
Initialize Cmin、MminFor 100, Cmax、MmaxFor 0；Otherwise, step 2) is returned to.
Preferably, step 4 concrete operations are：
First, management server checks whether available computational resources reach matching requirement on calculation server；
Secondly, management server treats scheduling set by resource allocation fairness and the sequence of data locality strategy；
Finally, resource management server takes out task, and the supposition information of task is obtained from estimation result；
Wherein, if acquisition of information failure is speculated, according to task resource applications matching primitives resource；If speculating, information obtains
Success is taken, then the resource requirement of matching task successively stage by stage.
Preferably, computing resource is abstracted as compressible resource and incompressible money by management server according to resource characteristicses
Source；Wherein, computing resource includes cpu resource, memory source, disk resource and Internet resources；
In task implementation procedure, if certain resource that task is assigned to is less than task to the resources requirement, task energy
Enough to be normally completed by extending the execution time, then the resource is compressible resource, is otherwise incompressible resource；
Resource compression ratio r on calculation servercCalculated by formula (3)：
rcFor resource compression ratio, RrFor resources requirement, RuFor resource allocation；
If resource is incompressible resource, its resource compression ratio rcIt is always 0.
Preferably, in the matching process, if demand is more than available volume of resources, it fails to match；
For incompressible resource, calculation server does not process to the available volume of resources of the resource；
For compressible resource, management server calculates each rank of the server according to calculation server resource and loading condition
The maximum compression rate of the various resources of section, the maximum compression rate r of calculation server certain compressible resource within each stagemaxBy formula
(4) calculate：
N task is matched in formula, on server, current matching task is (n+1)th, and the resource of n+1 task always needs
Ask and be more than total resources, the workload that i-th of task is completed within the stage is wi, Δ p is performance change caused by resource compression；
It is compressible it is resource matched in, the actually available stock number of the calculation server resource is calculated by formula (5),
ai=ni+ri×Ni(5)；
In formula, riFor the maximum compressible in the current matching stage of this kind of resource on the server, NiFor on the server
The total amount of this kind of resource, niFor the available volume of resources of this kind of resource collected；
If it fails to match for the demand of all execution stage all resources of task, the match is successful.
Compared with prior art, the present invention has technique effect beneficial below：
Fine granularity reso urce matching method in cloud computing platform disclosed by the invention, computing resource is abstracted according to resource characteristicses
For compressible and incompressible two types；According to cloud computing load present in similar tasks and resource compression ratio speculate task
Resource requirement and the duration；Task is divided into by multiple execution stages according to task resource demand, and divides resource stage by stage
Type matches resource requirement and available resources respectively；In the matching process, by way of resource compression, with mild prolonged task
Deadline is the lifting that cost exchanges overall resource utilization and load performance for.This method is in distribution time, the side of stock number two
While matching granularity is improved in face, there is relatively low scheduling average response time, thus can apply and be provided in each cloud computing platform
Source control, job scheduling etc., the problems such as avoiding resource fragmentation and over-allocation, that improves computing resource in platform utilizes effect
Rate, the final overall throughput for lifting cloud computing platform.
Brief description of the drawings
Fig. 1 is the FB(flow block) of tasks execution phases partitioning algorithm.
Fig. 2 is the Organization Chart that fine granularity reso urce matching method is realized in Yarn platforms.
Embodiment
With reference to specific embodiment, the present invention is described in further detail, it is described be explanation of the invention and
It is not to limit.
Fine granularity reso urce matching method in cloud computing platform disclosed by the invention, speculate that task resource needs according to similar tasks
Ask and the duration；Resource is abstracted as compressible and incompressible two types according to resource characteristicses；Task is divided into more
Individual execution stage and the demand for matching each execution stage all kinds resource successively.This method can be in cloud computing platform resource pipe
By improving the time granularity and stock number granularity of resource allocation when reason and job scheduling matching resource, avoid in resource management
The problems such as resource fragmentation and over-allocation, resource utilization and performance are improved so as to realize.
Fine granularity reso urce matching method, comprises the following steps in the cloud computing platform of disclosure of the invention：
Step 1：The role of server is divided into two kinds of calculation server and management server in cloud computing platform：Calculate service
Device is responsible for the execution specifically loaded, and periodically reports resource status to management server；Management server is responsible for whole cloud platform
Various management works, including to calculation server distribute calculating task.
Task resource service condition in the operation of calculation server taken at regular intervals book server, calculates book server available resources
Information is simultaneously reported to management server.
The available volume of resources of certain resource calculates as follows on calculation server：
Wherein, riFor the stock number of ith sampling, tiThe duration sampled for ith, when T is that the calculating of sampling is total
Between, n is sampling number.
Step 2：Management server receives the information that each calculation server is periodically reported, and according to similar tasks and resource pressure
Shrinkage speculates various resource requirements and the duration of task.Herein, similar tasks refer in load has identical hold with the task
Row logic and input data amount identical task.The disk of task, network bandwidth resources demand according to task and data phase
It is divided into task to position with data in same server, in same frame and other 3 classes respectively to speculate.
Task calculates as follows in a kind of computational resource requirements amount needed for certain progress and time：
Wherein, αnFor n-th estimation result, βnFor the similar tasks resource information of n-th, rcFor the resource of mission bit stream
Compression ratio, ThrFor maximum compression rate limit coefficient, e is the nature truth of a matter.
Step 3：The CPU of each progress of management server analysis task, memory headroom resource requirement, task are divided into multiple
The execution stage.
Management server travels through CPU, memory headroom resources requirement of the task in each progress, and records both moneys respectively
The maximum and minimum value of source amount.When the difference of the maximum and minimum value of CPU or memory headroom resources requirement is more than necessarily
Threshold value, and the progress traveled through reaches certain length, then the progress of traversal is divided into an execution stage of task.Task is held
The resource requirement values such as row order section CPU, memory headroom, disk space be the stage in each progress resource requirement maximum, internal memory
The requirements of the resources such as bandwidth, Disk bandwidth, network bandwidth are the average value of each progress resource requirement in the stage.
Step 4：When available computational resources reach a certain amount of on calculation server, management server is from treating in scheduling set
Select task, simulation stage by stage task resource demand and server available computational resources.
Management server first checks for whether available resources on calculation server reach matching requirement.Secondly, management service
Device treats scheduling set according to the strategy sequence such as resource allocation fairness, data locality.Again, resource management server, which takes out, appoints
Business, and attempt to obtain the supposition information of task from estimation result.If acquisition of information failure is speculated, according to task resource Shen
Please flux matched computing resource.If speculate acquisition of information success, the stage by stage resource requirement of matching task successively.
The computing resources such as CPU, internal memory, disk, network are abstracted as compressible and not by management server according to resource characteristicses
Compressible two class.In task implementation procedure, if certain resource that task is assigned to is less than demand of the task to the resource,
Task can be normally completed by extending the execution time, then the resource is compressible resource, is otherwise incompressible resource.
Resource compression ratio calculates as follows on calculation server：
Wherein, rcFor resource compression ratio, RrFor the demand of resource, RuFor resource allocation.If resource is incompressible
Resource, then its compression ratio rcIt is always 0.
In the matching process, if demand is more than available volume of resources, it fails to match.For incompressible resource, meter
The available volume of resources of the calculation server resource does not process；For compressible resource, management server provides according to calculation server
Source and loading condition calculate the maximum compression rate of the server each stage various resources.
Calculation server maximum compression rate of certain compressible resource within the stage calculates as follows：
Wherein, n task has been matched on server, current matching task is (n+1)th, and the resource of n+1 task always needs
Ask and be more than total resources, the workload that i-th of task is completed within the stage is wi, △ p are performance change caused by resource compression.
It is compressible it is resource matched in, the actually available stock number of the calculation server resource presses formula ai=ni+ri×NiMeter
Calculate；Wherein, riFor the maximum compressible in the current matching stage of this kind of resource on the server, NiFor this kind on the server
The total amount of resource, niFor the available volume of resources of this kind of resource collected.If all execution stage all resources of task
Demand it fails to match, then the match is successful.
Specifically, matching process is carried out resource matched according to the following steps：
Step1), server available volume of resources is checked, if available volume of resources is less than scheduling thresholds, terminates to dispatch；
Step2), scheduler task set is treated according to the strategy sequence such as resource allocation fairness, data locality.
Step3), from scheduler task set one task of selection is treated, acquisition task respectively performs the resource requirement in stage and held
The continuous time；If obtained successfully, into step5), it is otherwise flux matched by applying, enter step7 by rear)；Not by then entering
Enter step5)；
Step4), using task resource applications as the flux matched resource of demand, the match is successful then enters step7)；
Step5), s is tasks execution phases, and n is the server corresponding available resources stage, compares i-th kind of resource successively and exists
The resource requirement s in the execution stageiWith the resource availability a in available resources stageiIf resource is incompressible money in i-th
Source, then aiIt is worth for respective resources value n in the available resources stagei；If resource is compressible resource, aiValue is according to formula ai=ni
+ri×NiCalculate, wherein riFor the maximum compressible in the corresponding available resources stage of this kind of resource on the server, NiFor this
The total amount of this kind of resource on server.If resource requirement is more than available resources, it fails to match, into step3), otherwise enter
Enter step6)；
Step6), if s, which is task, finally performs the stage, the match is successful, into step 7), otherwise repeat
Step5), next stage is matched.
Step7), check whether the matching influences other tasks carryings on server, completed if checking to match if, it is no
Then return to step3).
Step 5：If resource matched success, management server assumes that resource is the allocated, checks the calculation server
Upper all tasks can be affected and can not meet constraints.If all task restriction conditions can meet, distribute
Computing resource.
Step 6：Management server checks whether remaining resource enough matches into next round on the calculation server, if
Surplus resources meet condition, then are matched into next round.
It can be achieved to match the fine granularity of cloud computing platform computing resource as stated above.
A concrete application example using the inventive method is given below：
Referring to Fig. 2, the example, which is incorporated in, increases income on cloud computing platform Yarn, it is necessary to illustrate, method of the invention is simultaneously
The cloud computing platform Yarn that increases income is used not only for, other meet that the application platform of condition is also suitable.
Step 1：Application Master to Resource Manager register when, there is provided application code and ginseng
Several MD5 values and input data amount information.Application Master apply for required by task meter to Resource Manager
When calculating resource, Application Master and task type message identification task are used.
Step 2：Node Manager are appointed under Linux Proc files by analyzing on this node of acquisition of information in operation
Information of being engaged in and residual resource information.
The available volume of resources of certain resource of Node Manager calculates as follows：
Wherein, riFor the stock number of ith sampling, tiFor the duration of ith sampling, T is the total time of sampling, and n is
Sampling number.
The resource information of collection and analysis are calculated the relevant informations such as available volume of resources and pass through the heart by Node Manager
The mode of jump, which is returned, gives Resource Manager.
Step 3：Resource Manager receive Application Master and Node Manager request and remittance
Report, and Application Master application registration is transferred into Scheduler processing with resource bid information, by Node
The resource information that Manager is reported transfers to Estimator processing.
Step 4：Estimator identifies according to Application Master using log-on message, resource request information
Similar tasks.Similar tasks refer in load has identical execution logic and input data amount identical task with the task.
Estimator handles Node Manager report informations, thus it is speculated that the resource requirement of task and duration.Task
Disk, network bandwidth resources demand according to the relative position of task and data be divided into task and data in same server, same
Frame and other 3 classes speculate respectively.
Task calculates as follows in a kind of required computational resource requirements amount of certain progress and time：
Wherein, αnFor n-th estimation result, βnFor the similar tasks resource information of n-th, rcFor similar tasks information
Resource compression ratio, ThrFor maximum compression rate limit coefficient, e is the nature truth of a matter.
Estimator analysis tasks CPU, memory space requirement information, task is divided into multiple execution stages.
Estimator travels through CPU, memory headroom resources requirement of the task in each progress, and records both stock numbers respectively most
Big value and minimum value.Threshold value is divided when the difference of the maximum and minimum value of CPU or memory headroom resources requirement is more than, and
The progress of traversal reaches partition length, then the progress of traversal is divided into an execution stage of task.Division threshold value is write from memory herein
Recognize and take the 20% of total resources, partition length is defaulted as the 5% of Task Progress overall length.Divide threshold value and partition length should be according to tool
Body platform and adjustment of load.The resource requirement such as tasks execution phases CPU, memory headroom, disk space value is the resource in the stage
The requirements of the resource such as the maximum of demand, memory bandwidth, Disk bandwidth, network bandwidth are being averaged for the stage interior resource requirement
Value.
Step 5：It is available in Scheduler matching Application Master resource bid and Node Manager
Computing resource.
Application Master resource bid is added and treated in scheduling set by Scheduler.
Scheduler checks whether the available computational resources on Node Manager reach matching and require.
Scheduler treats scheduling set according to the strategy sequence such as resource allocation fairness, data locality
Scheduler treats that scheduler task is treated in taking-up in scheduling set from after sequence, and obtains and speculate from Estimator
The information such as the task resource demand gone out and duration.If acquisition of information fails, matched according to resource bid amount.Such as
The success of fruit acquisition of information, then all resource requirements in all execution stages of matching task successively.
In matching, according to resource characteristicses the computing resources such as CPU, internal memory, disk, network are abstracted as compressible and can not pressed
Contract two classes.In task implementation procedure, if certain resource that task is assigned to is less than task to the demand of the resource, task
It can be normally completed by extending the execution time, then the resource is compressible resource, is otherwise incompressible resource.
The degree of Node Manager resources compression calculates as follows：
Wherein, rcFor resource compression ratio, RrFor the demand of resource, RuFor resource allocation.If resource is incompressible
Resource, then its compression ratio rcIt is always 0.
Node Manager maximum compression rates of certain compressible resource within certain stage calculate as follows：
Wherein, there are n matching tasks on Node Manager, current matching task is (n+1)th, n+1 task
Resource aggregate demand is more than total resources, and the workload that i-th of task is completed within the stage is wi, △ p are caused by resource compression
Performance change.
In matching process, if certain resources requirement of task is more than corresponding available volume of resources on Node Manager,
It fails to match.The available volume of resources of incompressible resource is collection capacity on Node Manager；The available resources of compressible resource
Amount presses formula ai=ni+ri×NiCalculate, wherein riFor the maximum compressible of this kind of resource matching stage on the server, Ni
For the total amount of this kind of resource on the server, niFor the available volume of resources of this kind of resource collected.If task is all
It fails to match for the demand of execution stage all resources, then the match is successful.
Step 6：After the match is successful, if Scheduler can check that the match decision comes into force institute on Node Manager
Whether the constraints for having task in performing can be met.Pass through if checked, Scheduler is to sending resource request
Application Master distribute computing resource.
Distribute after resource, the surplus resources of server, decide whether where Scheduler can check Node Manager
Dispatched into next round.
Step 7：The Application Master for obtaining resource allocation communicate with Node Manager where computing resource,
Start corresponding task.
Actual test result shows that the resource matched result that this method is drawn can avoid resource fragmentation, over-allocation, carry
The level of resources utilization of high cloud computing platform, the final overall throughput for lifting cloud computing platform.
It can be seen by above-described embodiment, the present invention can be used in cloud computing platform resource management and job scheduling.This
Invention speculates task resource demand and duration according to similar tasks, and task is divided into multiple execution ranks based on resource requirement
Section, divide resource characteristicses difference matching task resource requirement and server computing resource stage by stage, if necessary in tolerance interval
The deadline for extending individual task exchanges higher resource utilization and tasks in parallel quantity for, finally improves overall performance.
The present invention cannot be only used in resource management and job scheduling in cloud computing platform, the resource management based on cluster
Platform can all use for reference improvement.
Claims (9)
1. fine granularity reso urce matching method in a kind of cloud computing platform, it is characterised in that comprise the following steps：
Step 1：The role of server in cloud computing platform is divided into two kinds of calculation server and management server, calculation server
It is responsible for the execution specifically loaded, and periodically reports resource status to management server；Management server is responsible for whole cloud computing and put down
The management work of platform, including distribute calculating task to calculation server；
Step 2：Management server receives the information that each calculation server is periodically reported, and according to similar tasks and resource compression ratio
Speculate various resource requirements and the duration of some task；
Wherein, similar tasks refer to has identical execution logic and input data amount identical times with some described task in load
Business；
Step 3：Management server analysis speculates the CPU of each progress of task drawn, memory headroom resource requirement, and task is divided
For multiple execution stages；
Step 4：From treating to select task in scheduling set, simulation stage by stage task resource demand and server can use management server
Computing resource, and compression fit resource requirement as needed；
Step 5：If the match is successful for resource requirement, management server assumes that resource is the allocated, checks on the calculation server
Whether all tasks can be affected and can not meet constraints, if all task restriction conditions are satisfied by, distribution calculates
Resource.
2. fine granularity reso urce matching method in cloud computing platform according to claim 1, it is characterised in that after step 5 also
Including following operation：Management server checks whether remaining resource enough matches into next round on the calculation server, if
Surplus resources meet condition, then are matched into next round, that is, repeat step 4 and step 5.
3. fine granularity reso urce matching method in cloud computing platform according to claim 1, it is characterised in that in step 1, meter
Calculate server and be responsible for the execution that specifically loads, and periodically to management server report resource status, in particular to：Calculation server
Task resource service condition in the operation of taken at regular intervals book server, calculate book server available resource information and be reported to management clothes
Business device；
Wherein, the available volume of resources r of certain resource is calculated by formula (1) on calculation server：
<mrow>
<mi>r</mi>
<mo>=</mo>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>i</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mi>n</mi>
</munderover>
<mrow>
<mo>(</mo>
<msub>
<mi>r</mi>
<mi>i</mi>
</msub>
<mo>&times;</mo>
<msub>
<mi>t</mi>
<mi>i</mi>
</msub>
<mo>)</mo>
</mrow>
<mo>/</mo>
<mi>T</mi>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>1</mn>
<mo>)</mo>
</mrow>
<mo>;</mo>
</mrow>
In formula, riFor the stock number of ith sampling, tiFor the duration of ith sampling, T is the calculating total time of sampling, and n is
Sampling number.
4. fine granularity reso urce matching method in cloud computing platform according to claim 1, it is characterised in that in step 2, according to
Similar tasks and resource compression ratio speculate various resource requirements and the duration of some task, resource requirement type include CPU,
Memory headroom, disk space, Disk bandwidth and network bandwidth；
Wherein, the disk of task and network bandwidth resources demand are divided into three classes progress according to the relative position of task and data
Speculate, the first kind：Task is with data in same server；Second class：Task is with data in same frame；3rd class：Other；
Concrete operations are：
Task is calculated in the resource requirement needed for certain progress and duration by formula (2)：
<mrow>
<msub>
<mi>&alpha;</mi>
<mi>n</mi>
</msub>
<mo>=</mo>
<mfenced open = "{" close = "">
<mtable>
<mtr>
<mtd>
<mrow>
<mi>a</mi>
<mi>v</mi>
<mi>e</mi>
<mi>r</mi>
<mi>a</mi>
<mi>g</mi>
<mi>e</mi>
<mrow>
<mo>(</mo>
<msub>
<mi>&beta;</mi>
<mi>n</mi>
</msub>
<mo>,</mo>
<msub>
<mi>&beta;</mi>
<mrow>
<mi>n</mi>
<mo>+</mo>
<mn>1</mn>
</mrow>
</msub>
<mo>,</mo>
<msub>
<mi>&beta;</mi>
<mrow>
<mi>n</mi>
<mo>+</mo>
<mn>2</mn>
</mrow>
</msub>
<mo>)</mo>
</mrow>
</mrow>
</mtd>
<mtd>
<mrow>
<mi>n</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mrow>
<msup>
<mi>e</mi>
<mrow>
<mi>min</mi>
<mrow>
<mo>(</mo>
<msub>
<mi>Th</mi>
<mi>r</mi>
</msub>
<mo>&times;</mo>
<msub>
<mi>r</mi>
<mi>c</mi>
</msub>
<mo>-</mo>
<mn>5</mn>
<mo>/</mo>
<mn>60</mn>
<mo>,</mo>
<mn>0</mn>
<mo>)</mo>
</mrow>
</mrow>
</msup>
<mo>&times;</mo>
<msub>
<mi>&alpha;</mi>
<mrow>
<mi>n</mi>
<mo>-</mo>
<mn>1</mn>
</mrow>
</msub>
<mo>+</mo>
<mrow>
<mo>(</mo>
<mn>1</mn>
<mo>-</mo>
<msup>
<mi>e</mi>
<mrow>
<mi>min</mi>
<mrow>
<mo>(</mo>
<msub>
<mi>Th</mi>
<mi>r</mi>
</msub>
<mo>&times;</mo>
<msub>
<mi>r</mi>
<mi>c</mi>
</msub>
<mo>-</mo>
<mn>5</mn>
<mo>/</mo>
<mn>60</mn>
<mo>,</mo>
<mn>0</mn>
<mo>)</mo>
</mrow>
</mrow>
</msup>
<mo>)</mo>
</mrow>
<mo>&times;</mo>
<msub>
<mi>&beta;</mi>
<mrow>
<mi>n</mi>
<mo>+</mo>
<mn>2</mn>
</mrow>
</msub>
</mrow>
</mtd>
<mtd>
<mrow>
<mi>n</mi>
<mo>&GreaterEqual;</mo>
<mn>1</mn>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>2</mn>
<mo>)</mo>
</mrow>
<mo>;</mo>
</mrow>
In formula, αnFor n-th estimation result, βnFor the similar tasks resource information of n-th, rcCompressed for the resource of mission bit stream
Rate, ThrFor maximum compression rate limit coefficient, e is the nature truth of a matter.
5. fine granularity reso urce matching method in cloud computing platform according to claim 1, it is characterised in that step 3 is specific
Operate and be：Management server travels through CPU, memory headroom resources requirement of the task in each progress, and records both moneys respectively
The maximum and minimum value of source amount；
Threshold value is divided when the difference of the maximum and minimum value of CPU or memory headroom resources requirement is more than, and what is traveled through enters
Degree reaches partition length, then the progress of traversal is divided into an execution stage of task；
Tasks execution phases CPU, memory headroom and disk space resource requirement value are that each progress resource needs in the execution stage
The maximum asked, Disk bandwidth, the average value that the requirements of network bandwidth resources are each progress resource requirement in the execution stage.
6. fine granularity reso urce matching method in cloud computing platform according to claim 5, it is characterised in that tasks carrying rank
The division methods of section, specifically include following steps：
1)P、Ps、PeTask Progress is represented respectively, the stage starts progress and the stage terminates progress, Cmax、Cmin、MmaxAnd MminRespectively
CPU, the maximum and minimum value of memory headroom resource requirement in the expression stage；Wherein, P, Ps、PeIt is initialized as 0；Cmin、MminFor
100, Cmax、MmaxIt is initialized as 0；
2) Task Progress P and stage terminate progress PeIncrease by 1, if Task Progress P reaches 100%, by PsTo 100% division
To terminate after the new stage, otherwise continue step 3)；
If 3) current schedules cpu demand CpMore than Cmax, then by CmaxValue be updated to Cp, into step 5)；
If 4) current schedules cpu demand CpLess than Cmin, then by CminValue be updated to Cp；
If 5) current schedules memory requirements MpMore than Mmax, then by MmaxValue be updated to Mp, into step 7)；
If 6) current schedules memory requirements MpLess than Mmin, then by MminValue be updated to Mp；
If 7) CmaxWith CminDifference be more than cpu resource total amount C and threshold value ThcProduct or MmaxWith MminDifference be more than
Memory amount M and threshold value ThmProduct, then into step 8, otherwise into step 2)；
If 8) PeWith PsDifference be more than threshold value Thp, then by PsTo PeThe new stage is divided into, by PsIt is updated to Pe, reinitialize
Cmin、MminFor 100, Cmax、MmaxFor 0；Otherwise, step 2) is returned to.
7. fine granularity reso urce matching method in cloud computing platform according to claim 1, it is characterised in that step 4 is specific
Operate and be：
First, management server checks whether available computational resources reach matching requirement on calculation server；
Secondly, management server treats scheduling set by resource allocation fairness and the sequence of data locality strategy；
Finally, resource management server takes out task, and the supposition information of task is obtained from estimation result；
Wherein, if acquisition of information failure is speculated, according to task resource applications matching primitives resource；If speculate acquisition of information into
Work(, the then resource requirement of matching task successively stage by stage.
8. fine granularity reso urce matching method in cloud computing platform according to claim 7, it is characterised in that management server
Computing resource is abstracted as compressible resource and incompressible resource according to resource characteristicses；Wherein, computing resource provides including CPU
Source, memory source, disk resource and Internet resources；
In task implementation procedure, if certain resource that task is assigned to can lead to less than task to the resources requirement, task
Cross extension perform the time normally complete, then the resource is compressible resource, is otherwise incompressible resource；
Resource compression ratio r on calculation servercCalculated by formula (3)：
<mrow>
<msub>
<mi>r</mi>
<mi>c</mi>
</msub>
<mo>=</mo>
<mfenced open = "{" close = "">
<mtable>
<mtr>
<mtd>
<mrow>
<mo>(</mo>
<msub>
<mi>R</mi>
<mi>r</mi>
</msub>
<mo>-</mo>
<msub>
<mi>R</mi>
<mi>u</mi>
</msub>
<mo>)</mo>
<mo>/</mo>
<msub>
<mi>R</mi>
<mi>r</mi>
</msub>
</mrow>
</mtd>
<mtd>
<mrow>
<msub>
<mi>R</mi>
<mi>r</mi>
</msub>
<mo>></mo>
<msub>
<mi>R</mi>
<mi>u</mi>
</msub>
</mrow>
</mtd>
</mtr>
<mtr>
<mtd>
<mn>0</mn>
</mtd>
<mtd>
<mrow>
<msub>
<mi>R</mi>
<mi>r</mi>
</msub>
<mo>&le;</mo>
<msub>
<mi>R</mi>
<mi>u</mi>
</msub>
</mrow>
</mtd>
</mtr>
</mtable>
</mfenced>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>3</mn>
<mo>)</mo>
</mrow>
<mo>;</mo>
</mrow>
rcFor resource compression ratio, RrFor resources requirement, RuFor resource allocation；
If resource is incompressible resource, its resource compression ratio rcIt is always 0.
9. fine granularity reso urce matching method in cloud computing platform according to claim 8, it is characterised in that in matching process
In, if demand is more than available volume of resources, it fails to match；
For incompressible resource, calculation server does not process to the available volume of resources of the resource；
For compressible resource, it is each that management server according to calculation server resource and loading condition calculates the server each stage
The maximum compression rate of kind resource, the maximum compression rate r of calculation server certain compressible resource within each stagemaxBased on formula (4)
Calculate：
<mrow>
<msub>
<mi>r</mi>
<mrow>
<mi>m</mi>
<mi>a</mi>
<mi>x</mi>
</mrow>
</msub>
<mo>=</mo>
<mi>m</mi>
<mi>a</mi>
<mi>x</mi>
<mrow>
<mo>(</mo>
<mn>0</mn>
<mo>,</mo>
<mfrac>
<mrow>
<msub>
<mi>w</mi>
<mrow>
<mi>n</mi>
<mo>+</mo>
<mn>1</mn>
</mrow>
</msub>
<mo>-</mo>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>i</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mi>n</mi>
</munderover>
<msub>
<mi>w</mi>
<mi>i</mi>
</msub>
<mo>&times;</mo>
<mi>&Delta;</mi>
<mi>p</mi>
</mrow>
<mrow>
<munderover>
<mo>&Sigma;</mo>
<mrow>
<mi>i</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mrow>
<mi>n</mi>
<mo>+</mo>
<mn>1</mn>
</mrow>
</munderover>
<msub>
<mi>w</mi>
<mi>i</mi>
</msub>
</mrow>
</mfrac>
<mo>)</mo>
</mrow>
<mo>-</mo>
<mo>-</mo>
<mo>-</mo>
<mrow>
<mo>(</mo>
<mn>4</mn>
<mo>)</mo>
</mrow>
<mo>;</mo>
</mrow>
N task is matched in formula, on server, current matching task is (n+1)th, and the resource aggregate demand of n+1 task is big
In total resources, the workload that i-th of task is completed within the stage is wi, Δ p is performance change caused by resource compression；
It is compressible it is resource matched in, the actually available stock number of the calculation server resource is calculated by formula (5),
ai=ni+ri×Ni(5)；
In formula, riFor the maximum compressible in the current matching stage of this kind of resource on the server, NiFor this kind on the server
The total amount of resource, niFor the available volume of resources of this kind of resource collected；
If it fails to match for the demand of all execution stage all resources of task, the match is successful.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710909672.8A CN107797863B (en) | 2017-09-29 | 2017-09-29 | Fine-grained resource matching method in cloud computing platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710909672.8A CN107797863B (en) | 2017-09-29 | 2017-09-29 | Fine-grained resource matching method in cloud computing platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107797863A true CN107797863A (en) | 2018-03-13 |
| CN107797863B CN107797863B (en) | 2020-07-28 |
Family
ID=61532969
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710909672.8A Active CN107797863B (en) | 2017-09-29 | 2017-09-29 | Fine-grained resource matching method in cloud computing platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107797863B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108572875A (en) * | 2018-04-28 | 2018-09-25 | 辽宁工程技术大学 | Resource allocation methods, apparatus and system |
| CN110247979A (en) * | 2019-06-21 | 2019-09-17 | 北京邮电大学 | A kind of scheduling scheme determines method, apparatus and electronic equipment |
| CN110659126A (en) * | 2018-06-29 | 2020-01-07 | 中兴通讯股份有限公司 | Resource management method, device and computer readable storage medium |
| CN110780977A (en) * | 2019-10-25 | 2020-02-11 | 杭州安恒信息技术股份有限公司 | Task issuing method, device and system based on cloud computing and readable storage medium |
| WO2020211300A1 (en) * | 2019-04-16 | 2020-10-22 | 苏宁云计算有限公司 | Resource allocation method and apparatus, and computer device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102460393A (en) * | 2009-05-01 | 2012-05-16 | 思杰系统有限公司 | Systems and methods for establishing a cloud bridge between virtual storage resources |
| CN103533086A (en) * | 2013-10-31 | 2014-01-22 | 中国科学院计算机网络信息中心 | Uniform resource scheduling method in cloud computing system |
| US20140207955A1 (en) * | 2013-01-22 | 2014-07-24 | International Business Machines Corporation | Storage managment in a multi-tiered storage architecture |
| CN105718364A (en) * | 2016-01-15 | 2016-06-29 | 西安交通大学 | Dynamic assessment method for ability of computation resource in cloud computing platform |
| US20170163764A1 (en) * | 2015-12-07 | 2017-06-08 | International Business Machines Corporation | Managing provisioning for cloud resource allocations |
-
2017
- 2017-09-29 CN CN201710909672.8A patent/CN107797863B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102460393A (en) * | 2009-05-01 | 2012-05-16 | 思杰系统有限公司 | Systems and methods for establishing a cloud bridge between virtual storage resources |
| US20140207955A1 (en) * | 2013-01-22 | 2014-07-24 | International Business Machines Corporation | Storage managment in a multi-tiered storage architecture |
| CN103533086A (en) * | 2013-10-31 | 2014-01-22 | 中国科学院计算机网络信息中心 | Uniform resource scheduling method in cloud computing system |
| US20170163764A1 (en) * | 2015-12-07 | 2017-06-08 | International Business Machines Corporation | Managing provisioning for cloud resource allocations |
| CN105718364A (en) * | 2016-01-15 | 2016-06-29 | 西安交通大学 | Dynamic assessment method for ability of computation resource in cloud computing platform |
Non-Patent Citations (1)
| Title |
|---|
| 周墨颂等: "采用资源划分的云环境下Hadoop资源许可调度方法", 《西安交通大学学报》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108572875A (en) * | 2018-04-28 | 2018-09-25 | 辽宁工程技术大学 | Resource allocation methods, apparatus and system |
| CN108572875B (en) * | 2018-04-28 | 2020-09-04 | 辽宁工程技术大学 | Resource allocation method, device and system |
| CN110659126A (en) * | 2018-06-29 | 2020-01-07 | 中兴通讯股份有限公司 | Resource management method, device and computer readable storage medium |
| CN110659126B (en) * | 2018-06-29 | 2023-04-14 | 中兴通讯股份有限公司 | Resource management method, device and computer readable storage medium |
| WO2020211300A1 (en) * | 2019-04-16 | 2020-10-22 | 苏宁云计算有限公司 | Resource allocation method and apparatus, and computer device and storage medium |
| CN110247979A (en) * | 2019-06-21 | 2019-09-17 | 北京邮电大学 | A kind of scheduling scheme determines method, apparatus and electronic equipment |
| CN110780977A (en) * | 2019-10-25 | 2020-02-11 | 杭州安恒信息技术股份有限公司 | Task issuing method, device and system based on cloud computing and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107797863B (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107797863A (en) | Fine granularity reso urce matching method in a kind of cloud computing platform | |
| US11989647B2 (en) | Self-learning scheduler for application orchestration on shared compute cluster | |
| CN112416585B (en) | Deep learning-oriented GPU resource management and intelligent scheduling method | |
| CN104951372B (en) | A kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation methods based on prediction | |
| US20070226743A1 (en) | Parallel-distributed-processing program and parallel-distributed-processing system | |
| CN110543352B (en) | Task allocation method of scheduling system and related device thereof | |
| CN104834564A (en) | Task scheduling method and device | |
| CN109918201A (en) | The control method and system of task unloading | |
| Jangiti et al. | Scalable and direct vector bin-packing heuristic based on residual resource ratios for virtual machine placement in cloud data centers | |
| CN104794239A (en) | Cloud platform data processing method | |
| CN104965762B (en) | A kind of scheduling system towards hybrid task | |
| CN113568759B (en) | Cloud computing-based big data processing method and system | |
| CN105187531A (en) | Cloud computing virtualized server cluster load balancing system and method | |
| CN108132840A (en) | Resource regulating method and device in a kind of distributed system | |
| Perepelitsyn et al. | Method of QoS evaluation of FPGA as a service | |
| Benoit et al. | Max-stretch minimization on an edge-cloud platform | |
| Lin et al. | Two-tier project and job scheduling for SaaS cloud service providers | |
| CN112256436B (en) | Resource allocation method, device, equipment and computer storage medium | |
| Cao et al. | Online cost-rejection rate scheduling for resource requests in hybrid clouds | |
| CN114579284A (en) | Task scheduling method and device | |
| CN114786156A (en) | Internet of vehicles optimization task edge unloading method based on deep learning | |
| CN104796493A (en) | Information processing method based on cloud computing | |
| Lin et al. | A multi-centric model of resource and capability management in cloud simulation | |
| CN117453376B (en) | Control method, device, equipment and storage medium for high-throughput calculation | |
| US20230004440A1 (en) | Allocating of computing resources for applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |