CN107291539A - Cluster program scheduler method based on resource significance level - Google Patents
Cluster program scheduler method based on resource significance level Download PDFInfo
- Publication number
- CN107291539A CN107291539A CN201710462836.7A CN201710462836A CN107291539A CN 107291539 A CN107291539 A CN 107291539A CN 201710462836 A CN201710462836 A CN 201710462836A CN 107291539 A CN107291539 A CN 107291539A
- Authority
- CN
- China
- Prior art keywords
- resource
- program
- significance level
- server node
- capacity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/485—Resource constraint
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/504—Resource capping
Abstract
The invention discloses the cluster program scheduler method based on resource significance level, the dispatching method is for each program by resource according to the important procedure sequence to program, resource is searched according to resource significance level sequence when searching node, so as to ensure the performance of polytype program.Present invention, avoiding the shortcoming being scheduled based on processor resource and memory source of blindness, it is ensured that disk-intensive type, the performance of programs of network intensive model；Performing environment is searched according to different resource sequences to each program, the effectiveness for playing resource in data center can be maximized.
Description
Technical field
It is more particularly to important based on resource the present invention relates to the scheduling of Parallel ＆ Distributed Computing, especially cluster Program
The cluster program scheduler method of degree.
Background technology
Data center can be with the carrier of win-win, with net with cloud service operator as the infrastructure of cloud computing and user
Network accesses the non-local increase for calculating service, is moved to maturity from concept.But data center resource utilization rate is typically not enough
30%.And low-resource utilization rate causes compared with low-energy-efficiency, the investigation display data center of the New York Times in 2012 wastes a large amount of energy consumptions,
Only 6% to the 12% of total energy consumption has been used as effective calculating.How resource utilization is improved as cloud computing operator needs
The key issue of consideration, all receives much concern from business perspective and academic angle.
Load aggregation (Workload Consolidation) is the important means for improving data center resource utilization rate, is born
It is that multiple programs are assigned in a calculate node to carry polymerization, so that server node node improves calculating, storage, disk
The utilization rate of the resources such as I/O, it is possible to close more idle nodes to reduce energy consumption expense.Investigation display in the recent period is needed with calculating
The increase asked, the data center operator more than 60% can use load aggregation.Load aggregation can be in program feature and system
Balance is realized between resource utilization.
Diversified program is run at current data center, and such as processor is intensive, disk-intensive type.However,
The current general load aggregation method of data center is only to consider processor and memory source when doing scheduling decision, according to program
Occupancy to both resources realizes scheduling, so as to reach the load balancing of processor and memory source in cluster.But it is this
Method have ignored program to disk and the occupancy capacity of network bandwidth resources, often lead to disk-intensive or the intensive journey of the network bandwidth
The relatively low performance of sequence.Accordingly, it would be desirable to which a kind of scheduling strategy for being more suitable for data center is it can be considered that processor, internal memory, disk
The multiple resources such as bandwidth, the network bandwidth, it is ensured that the performance of the program of various features.
The content of the invention
The purpose of the present invention is to propose to a kind of cluster program scheduler method based on resource significance level, the dispatching method pin
To each program by resource according to the significance level sequence to program, this sequence is called resource significance level sequence.Searching
Resource is searched according to resource significance level sequence during node, so as to ensure the performance of polytype program.
Cluster program scheduler method based on resource significance level, processor, disk reading, magnetic are considered when realizing scheduling
Disk write-in, five resources of internal memory and the network bandwidth, comprise the following steps：
Step (1)：Collection of resources：Obtain processor, disk reading, disk write-in, five resources of internal memory and the network bandwidth
Idling-resource information；
Step (2)：Resource sorts：First, computing resource is to program significance level；Then, for every in task queue
One program, is ranked up according to resource to the significance level of program to resource；Obtain the resource significance level of each program
Sequence；
Step (3)：Scheduling：For each program in task queue, first is chosen from resource significance level sequence
Individual resource, and be first resource lookup several server node；The service for searching several server nodes to be searched
It is standard to the occupancy capacity of first resource that the idling-resource capacity of device node, which is more than program,；
Then, it is that second resource lookup meets demand in the server node found out from first resource
Server node, equally, is more than occupancy capacity of the program to second resource with the idling-resource for the server node searched
For standard；
By that analogy, until going out to meet the server node of demand for last resource lookup in sequence, by last
The server node that individual resource lookup goes out is stored into service list；
When program is performed, server node is selected to enter line program operation directly from service list.
The step of step (1) is：
Step (101)：Obtain processor, disk reading, disk write-in, internal memory, the network bandwidth five on server node
Resource use capacity；
Step (102)：It is respective total using processor, disk reading, disk write-in, internal memory, five resources of the network bandwidth
Capacity, correspondence subtract each resource it is respective use capacity, obtain the idling-resource information of each resource；
Step (103)：Periodically report the idling-resource information of Current resource.
In the step (101) using obtained by performance analysis tool collectl processor on server node,
Disk reading, disk write-in, internal memory, five resources of the network bandwidth use capacity.
The resource is as follows to the computational methods of program significance level：
Step (201)：For each resource, it is maximum to set stock number of the program when the unrestricted condition of resource is performed
Maximum resource limitation point in the range of resource constraint point, setting ratio limits point for least resource；
Step (202)：Acquisition program limits the performance under point and least resource limitation point in maximum resource；
Step (203)：Calculation procedure limits point and the performance ratio under maximum resource limitation point in least resource；The property
Energy ratio is exactly importance value of the resource to program；Resource is bigger to the importance value of program, illustrates that resource size changes
Insensitive to program feature, resource is inessential to program；Resource is smaller to program importance value, illustrates resource size change pair
Program feature is sensitive, and resource is important to program；Sorted from small to large according to the importance value of program, obtain the resource weight of program
Want degree of sequence.
The step (202)：Resource is limited by using resource constraint instrument Cgroups, program is obtained in maximum resource
Limitation point and least resource limit the performance under point；
Described program least resource limit point with maximum resource limit point under performance ratio be one between [0,1] it
Between constant.
The prior information of each program, including：The resource significance level sequence that takes capacity and program of the program to resource
Row.
The resource, including：Processor, disk reading, disk write-in, internal memory and the network bandwidth.
The step of step (3) is：
For each program in task queue, first resource is chosen from resource significance level sequence, and search
Several first kind server nodes, searching the standard of first kind server node is：The appearance of each first kind server node
Amount is more than occupancy capacity of the program to first resource；
Then, second resource is chosen from resource significance level sequence, and from several first kind server nodes
Search several Equations of The Second Kind server nodes；Searching the standard of Equations of The Second Kind server node is：Each Equations of The Second Kind server node
Capacity be more than program to the occupancy capacity of second resource；
Then, the 3rd resource is chosen from resource significance level sequence, and from several Equations of The Second Kind server nodes
Search several the 3rd class server nodes；Searching the standard of the 3rd class server node is：Each 3rd class server node
Capacity be more than program to the occupancy capacity of the 3rd resource；
Then, the 4th resource is chosen from resource significance level sequence, and from several the 3rd class server nodes
Search several the 4th class server nodes；Searching the standard of the 4th class server node is：Each 4th class server node
Capacity be more than program to the occupancy capacity of the 4th resource；
Then, the 5th resource is chosen from resource significance level sequence, and from several the 4th class server nodes
Search several the 5th class server nodes；Searching the standard of the 5th class server node is：Each 5th class server node
Capacity be more than program to the occupancy capacity of the 5th resource；
Finally, by the storage of all 5th class server node titles into server list, the node in server list
For the node that can be mapped.
The present invention is based on the advantage that resource significance level is dispatched：
1st, the valuable source of the invention by targetedly analysis program, the valuable source based on program is scheduled.
This method avoid the shortcoming being scheduled based on processor resource and memory source of blindness, it is ensured that disk-intensive type, net
The performance of network intensive procedure.
2nd, the present invention searches performing environment to each program according to different resource sequences, can maximize performance data
The effectiveness of resource in center.
Brief description of the drawings
Fig. 1 is to obtain resource significance level schematic diagram.
Fig. 2 is scheduling strategy flow chart.
Embodiment
The invention will be further described with embodiment below in conjunction with the accompanying drawings.
Fig. 1 descriptions obtain method of the processor resource to program Data Caching significance level.Processor has 8
Core, we set 8 cores to limit point for maximum resource, and it is that least resource limits point to set 2 cores.And program takes 2 cores and held
Capable performance is with taking significance level of the processor resource to program that performance ratio when 8 cores are performed is our requirements.
In this example, processor resource importance value is 0.92.Resource is bigger to the importance value of program, illustrates that resource size becomes
Change is insensitive to program feature, and the resource is inessential to program.Resource is smaller to program importance value, illustrates that resource size becomes
Change to program feature sensitivity, the resource is important to program.In this way, can be with computation processor, disk reading, disk write
Enter, internal memory, network bandwidth resources to the importance value of program, sorted from small to large according to numerical value, you can obtain for program
Data Caching resource significance level sequence.
Fig. 2 describes scheduling strategy.
One：Idling-resource information gathering
On each node utility analysis tool collectl obtain the processor of present node, disk read,
Resource using information in disk write-in, five resource dimensions of internal memory and the network bandwidth；And idling-resource is then equal to server section
The total resources of point subtract the resource used；And periodically report the idling-resource on present node to fallout predictor.
Two：Resource sorts
Resource significance level sequence is obtained according to Fig. 1 description.
Three：Scheduling
For each program in task queue, all comprising two groups of prior informations：Program processor, disk read,
Resource using information in disk write-in, five resource dimensions of internal memory and the network bandwidth, the resource significance level sequence of program.
First resource is chosen from resource significance level sequence, and checks occupancy capacity of the program to the resource, with this
For standard filtration server node list.Previous step is repeated, until the resource in resource significance level sequence has been listed, then
Stop searching, the node in server list is the node that can be mapped.Mapped.
Although above-mentioned the embodiment of the present invention is described with reference to accompanying drawing, not to present invention protection model
The limitation enclosed, one of ordinary skill in the art should be understood that on the basis of technical scheme those skilled in the art are not
Need to pay various modifications or deform still within protection scope of the present invention that creative work can make.
Claims (9)
1. the cluster program scheduler method based on resource significance level, it is characterized in that, consider processor, disk when realizing scheduling
Read, disk writes, five resources of internal memory and the network bandwidth, comprise the following steps：
Step (1)：Collection of resources：Obtain processor, disk reading, disk write-in, the sky of five resources of internal memory and the network bandwidth
Not busy resource information；
Step (2)：Resource sorts：First, computing resource is to program significance level；Then, for each in task queue
Program, is ranked up according to resource to the significance level of program to resource；Obtain the resource significance level sequence of each program；
Step (3)：Scheduling：For each program in task queue, first money is chosen from resource significance level sequence
Source, and be first resource lookup several server node；The server section for searching several server nodes to be searched
It is standard to the occupancy capacity of first resource that the idling-resource capacity of point, which is more than program,；
Then, it is the service that second resource lookup meets demand in the server node found out from first resource
Device node, equally, the idling-resource of the server node to be searched are more than program to the occupancy capacity of second resource as mark
It is accurate；
By that analogy, until going out to meet the server node of demand for last resource lookup in sequence, last is provided
The server node that source is found out is stored into service list；
When program is performed, server node is selected to enter line program operation directly from service list.
2. the cluster program scheduler method as claimed in claim 1 based on resource significance level, it is characterized in that, the step
(1) the step of is：
Step (101)：Obtain processor, disk reading, disk write-in, internal memory, five resources of the network bandwidth on server node
Use capacity；
Step (102)：Using processor, disk reading, disk write-in, internal memory, five respective total capacities of resource of the network bandwidth,
Correspondingly subtract each resource respective using capacity, obtain the idling-resource information of each resource；
Step (103)：Periodically report the idling-resource information of Current resource.
3. the cluster program scheduler method as claimed in claim 2 based on resource significance level, it is characterized in that, the step
(101) in using obtained by performance analysis tool collectl processor on server node, disk reading, disk write
Enter, internal memory, five resources of the network bandwidth use capacity.
4. the cluster program scheduler method as claimed in claim 1 based on resource significance level, it is characterized in that, the resource pair
The computational methods of program significance level are as follows：
Step (201)：For each resource, it is maximum resource to set stock number of the program when the unrestricted condition of resource is performed
Limit the maximum resource limitation point in the range of point, setting ratio and limit point for least resource；
Step (202)：Acquisition program limits the performance under point and least resource limitation point in maximum resource；
Step (203)：Calculation procedure limits point and the performance ratio under maximum resource limitation point in least resource；The performance ratio
Value is exactly importance value of the resource to program；Resource is bigger to the importance value of program, illustrates resource size change to journey
Sequence can be insensitive, and resource is inessential to program；Resource is smaller to program importance value, illustrates resource size change to program
Performance sensitive, resource is important to program；Sorted from small to large according to the importance value of program, obtain the important journey of resource of program
Degree series.
5. the cluster program scheduler method as claimed in claim 4 based on resource significance level, it is characterized in that, the step
(202)：Resource is limited by using resource constraint instrument Cgroups, program is obtained and limits point and least resource in maximum resource
Performance under limitation point.
6. the cluster program scheduler method as claimed in claim 4 based on resource significance level, it is characterized in that, described program exists
Performance ratio under least resource limitation point and maximum resource limitation point is a constant between [0,1].
7. the cluster program scheduler method as claimed in claim 1 based on resource significance level, it is characterized in that, each program
Prior information, including：The resource significance level sequence that takes capacity and program of the program to resource.
8. the cluster program scheduler method as claimed in claim 7 based on resource significance level, it is characterized in that, the resource,
Including：Processor, disk reading, disk write-in, internal memory and the network bandwidth.
9. the cluster program scheduler method as claimed in claim 1 based on resource significance level, it is characterized in that, the step
(3) the step of is：
For each program in task queue, first resource is chosen from resource significance level sequence, and search some
Individual first kind server node, searching the standard of first kind server node is：The capacity of each first kind server node is big
In occupancy capacity of the program to first resource；
Then, second resource is chosen from resource significance level sequence, and is searched from several first kind server nodes
Several Equations of The Second Kind server nodes；Searching the standard of Equations of The Second Kind server node is：The appearance of each Equations of The Second Kind server node
Amount is more than occupancy capacity of the program to second resource；
Then, the 3rd resource is chosen from resource significance level sequence, and is searched from several Equations of The Second Kind server nodes
Several the 3rd class server nodes；Searching the standard of the 3rd class server node is：The appearance of each 3rd class server node
Amount is more than occupancy capacity of the program to the 3rd resource；
Then, the 4th resource is chosen from resource significance level sequence, and is searched from several the 3rd class server nodes
Several the 4th class server nodes；Searching the standard of the 4th class server node is：The appearance of each 4th class server node
Amount is more than occupancy capacity of the program to the 4th resource；
Then, the 5th resource is chosen from resource significance level sequence, and is searched from several the 4th class server nodes
Several the 5th class server nodes；Searching the standard of the 5th class server node is：The appearance of each 5th class server node
Amount is more than occupancy capacity of the program to the 5th resource；
Finally, by the storage of all 5th class server node titles into server list, the node in server list is energy
The node enough mapped.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710462836.7A CN107291539B (en) | 2017-06-19 | 2017-06-19 | Cluster program scheduler method based on resource significance level |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710462836.7A CN107291539B (en) | 2017-06-19 | 2017-06-19 | Cluster program scheduler method based on resource significance level |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107291539A true CN107291539A (en) | 2017-10-24 |
| CN107291539B CN107291539B (en) | 2019-11-01 |
Family
ID=60096816
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710462836.7A Active CN107291539B (en) | 2017-06-19 | 2017-06-19 | Cluster program scheduler method based on resource significance level |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107291539B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457138A (en) * | 2019-08-20 | 2019-11-15 | 网易（杭州）网络有限公司 | Management method, device and the electronic equipment of game server cluster |
| CN110740070A (en) * | 2019-11-22 | 2020-01-31 | 国网四川省电力公司经济技术研究院 | intelligent power grid site bandwidth estimation method based on multivariate nonlinear fitting |
| CN111475277A (en) * | 2019-01-23 | 2020-07-31 | 阿里巴巴集团控股有限公司 | Resource allocation method, system, equipment and machine readable storage medium |
| CN113377521A (en) * | 2020-02-25 | 2021-09-10 | 先智云端数据股份有限公司 | Method for establishing system resource prediction and management model through multi-level correlation |
| CN116048773A (en) * | 2022-10-25 | 2023-05-02 | 北京京航计算通讯研究所 | Distributed collaborative task assignment method and system based on wave function collapse |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103019853A (en) * | 2012-11-19 | 2013-04-03 | 北京亿赞普网络技术有限公司 | Method and device for dispatching job task |
| CN105430027A (en) * | 2014-09-04 | 2016-03-23 | 中国石油化工股份有限公司 | Load balance dynamic pre-allocating method based on a plurality of resource scales |
| CN105630575A (en) * | 2015-12-23 | 2016-06-01 | 一兰云联科技股份有限公司 | Performance evaluation method aiming at KVM virtualization server |
-
2017
- 2017-06-19 CN CN201710462836.7A patent/CN107291539B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103019853A (en) * | 2012-11-19 | 2013-04-03 | 北京亿赞普网络技术有限公司 | Method and device for dispatching job task |
| CN105430027A (en) * | 2014-09-04 | 2016-03-23 | 中国石油化工股份有限公司 | Load balance dynamic pre-allocating method based on a plurality of resource scales |
| CN105630575A (en) * | 2015-12-23 | 2016-06-01 | 一兰云联科技股份有限公司 | Performance evaluation method aiming at KVM virtualization server |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111475277A (en) * | 2019-01-23 | 2020-07-31 | 阿里巴巴集团控股有限公司 | Resource allocation method, system, equipment and machine readable storage medium |
| CN110457138A (en) * | 2019-08-20 | 2019-11-15 | 网易（杭州）网络有限公司 | Management method, device and the electronic equipment of game server cluster |
| CN110740070A (en) * | 2019-11-22 | 2020-01-31 | 国网四川省电力公司经济技术研究院 | intelligent power grid site bandwidth estimation method based on multivariate nonlinear fitting |
| CN110740070B (en) * | 2019-11-22 | 2022-07-01 | 国网四川省电力公司经济技术研究院 | Intelligent power grid station bandwidth estimation method based on multivariate nonlinear fitting |
| CN113377521A (en) * | 2020-02-25 | 2021-09-10 | 先智云端数据股份有限公司 | Method for establishing system resource prediction and management model through multi-level correlation |
| CN113377521B (en) * | 2020-02-25 | 2024-01-30 | 先智云端数据股份有限公司 | Method for establishing system resource prediction and management model through multi-level correlation |
| CN116048773A (en) * | 2022-10-25 | 2023-05-02 | 北京京航计算通讯研究所 | Distributed collaborative task assignment method and system based on wave function collapse |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107291539B (en) | 2019-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107291539B (en) | Cluster program scheduler method based on resource significance level | |
| CN107734052B (en) | Load balancing container scheduling method facing component dependence | |
| Sethi et al. | RecShard: statistical feature-based memory optimization for industry-scale neural recommendation | |
| CN102857560B (en) | A kind of cloud towards multi-service application stores data distribution method | |
| WO2010024027A1 (en) | Virtual server system and physical server selection method | |
| CN106020715A (en) | Storage pool capacity management | |
| CN105808358B (en) | A kind of data dependence thread packet mapping method for many-core system | |
| CN107273200B (en) | Task scheduling method for heterogeneous storage | |
| CN113010576A (en) | Method, device, equipment and storage medium for capacity evaluation of cloud computing system | |
| CN109327844A (en) | A kind of cell capacity-enlarging method and device | |
| CN109144670A (en) | A kind of resource regulating method and device | |
| CN112632025A (en) | Power grid enterprise management decision support application system based on PAAS platform | |
| CN114721835A (en) | Method, system, device and medium for predicting energy consumption of edge data center server | |
| Li et al. | Cost-aware automatic scaling and workload-aware replica management for edge-cloud environment | |
| CN109889370A (en) | A kind of network equipment location determining method, device and computer readable storage medium | |
| CN103699534A (en) | Display method and device for data object in system directory | |
| CN107193940A (en) | Big data method for optimization analysis | |
| CN110134646A (en) | The storage of knowledge platform service data and integrated approach and system | |
| JP4479431B2 (en) | Information life cycle management system and data arrangement determination method thereof | |
| CN103324577A (en) | Large-scale itemizing file distributing system based on minimum IO access conflict and file itemizing | |
| CN111124313A (en) | Data reading and writing method and device for power acquisition terminal and electronic equipment | |
| CN116244085A (en) | Kubernetes cluster container group scheduling method, device and medium | |
| CN107203554A (en) | A kind of distributed search method and device | |
| CN108762684A (en) | Hot spot data migrates flow control method, device, electronic equipment and storage medium | |
| CN108304549A (en) | A kind of big data Intelligent processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |