CN107220114A - Distributed resource scheduling method based on resource United Dispatching - Google Patents
Distributed resource scheduling method based on resource United Dispatching Download PDFInfo
- Publication number
- CN107220114A CN107220114A CN201710374158.9A CN201710374158A CN107220114A CN 107220114 A CN107220114 A CN 107220114A CN 201710374158 A CN201710374158 A CN 201710374158A CN 107220114 A CN107220114 A CN 107220114A
- Authority
- CN
- China
- Prior art keywords
- resource
- available resources
- distributed
- scheduling
- list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of distributed resource scheduling method based on resource United Dispatching, wherein, including：Determine the content of available resources；The available resources being collected into are packaged into list of available resources；Determine the priority of each distributed computing framework；According to the order from high to low of the priority of distributed computing framework, resource in list of available resources is selected by distributed computing framework, after each distributed computing framework selection is finished, update list of available resources, until all distributed computing frameworks all select to finish or list of available resources in there is no available resources；Restart this method from the content for determining available resources.The distributed resource scheduling method based on resource United Dispatching of the present invention, realizes the resource of shared cluster, and resource is uniformly used.
Description
Technical field
The present invention relates to distributed arithmetic framework technology field, more particularly to a kind of distribution based on resource United Dispatching
Resource regulating method.
Background technology
Different task in different distributed arithmetic frameworks (Spark, Hadoop, ES, MPI, Cassandra etc.) is often
The resource (internal memory, CPU, network I/O etc.) needed is different, and they are operated in same cluster, can be interfered, between task
Efficiency is caused to decline by contention for resources, O＆M cost rise.
The content of the invention
It is an object of the invention to provide a kind of method of the distributed resource scheduling of unified resource scheduling, for solving to keep away
Exempt to cause the problem of efficiency declines by contention for resources between task.
A kind of method of the distributed resource scheduling of unified resource scheduling of the present invention, wherein, including：Determine available resources
Content；The available resources being collected into are packaged into list of available resources；Determine the priority of each distributed computing framework；According to point
The order from high to low of the priority of cloth Computational frame, the resource in list of available resources is selected by distributed computing framework,
After each distributed computing framework selection is finished, list of available resources is updated, until all distributed computing frameworks are all selected
Finish or list of available resources in there is no available resources；Restart this method from the content for determining available resources.
One embodiment of the method for the distributed resource scheduling dispatched according to the unified resource of the present invention, wherein, money can be used
Source includes available CPU and free memory of the scheduling of resource from node.
One embodiment of the method for the distributed resource scheduling dispatched according to the unified resource of the present invention, wherein, resource row
Table includes：Available scheduling of resource is from node set, and each scheduling of resource includes from node set：Volume of the scheduling of resource from node
Number, CPU and free memory information can be used.
One embodiment of the method for the distributed resource scheduling dispatched according to the unified resource of the present invention, wherein, Mei Gefen
After the selection of cloth Computational frame is finished, list of available resources is updated, and it is excellent that the list of available resources after renewal is sent into time one
The distributed computing framework of first level.
One embodiment of the method for the distributed resource scheduling dispatched according to the unified resource of the present invention, wherein, it is distributed
Computational frame is according to its internal calculating demand, the resource information in list of available resources, the available resources needed for selection.
The method of the distributed resource scheduling of the unified resource scheduling of the present invention avoids appointing there is provided a kind of Resource Scheduling Mechanism
Efficiency is caused to decline by contention for resources between business, while in view of resource utilization, O＆M cost, the factor, one such as data sharing
As wish all these frameworks being deployed in a public cluster, allow the resource of their shared clusters, and resource is carried out
Uniformly use.
Brief description of the drawings
Fig. 1 show the module map for the system of distributed resource scheduling that the present invention is dispatched based on unified resource.
Embodiment
To make the purpose of the present invention, content and advantage clearer, with reference to the accompanying drawings and examples, to the present invention's
Embodiment is described in further detail.
Fig. 1 show the module map for the system of distributed resource scheduling that the present invention is dispatched based on unified resource, such as Fig. 1 institutes
Show, including：Scheduling of resource host node M, scheduling of resource from node S1, scheduling of resource from node S2, distributed computing framework f1, point
Cloth Computational frame f2.
As shown in figure 1, scheduling of resource host node M is responsible for the overall statistics of resource of cluster, the schedule level one of resource and specific
Execution task is issued.Including three below component：Resource statistics device：It is responsible for receiving what scheduling of resource in cluster was reported from node
Available resource information；Schedule level one device：The resource information collected is sent to registered distributed computing framework；Mission dispatching
Device：It is responsible for distributed computing task is issued into scheduling of resource being performed from node.
As shown in figure 1, scheduling of resource gives scheduling of resource master from node S1 and S2 for reporting the available resources on this node
Node, and be responsible for performing specific task.Including three below component：Resource collectors：Monitor available money on present node
Source；Task monitor：It is responsible for receiving and starts the task that scheduling of resource host node is sent, establishing resource spacing container performs task；
Resource isolation container：The resource independent operating that responsible basis is specified, the lightweight virtual machine of mutual process isolation.
As shown in figure 1, distributed computing framework f2 is responsible for the second-level dispatching and Task-decomposing of specific resource, it is necessary to be registered to
On scheduling of resource host node.Including three below primary clustering：Register：Registered to scheduling of resource host node, application monitoring collection
Group's resource；Task：The specific works performed in scheduling of resource from node, required resource letter when each task is comprising operation
Breath；Second-level dispatching device：The available resource information that scheduling of resource host node schedule level one device is sent is received, according to itself required by task
Resource by task determine performed to some scheduling of resource from node.
As shown in figure 1, an embodiment of the distributed resource scheduling method of the invention based on resource United Dispatching includes：Money
Source scheduling is from node s1 collecting robot available resources, and the present embodiment can include 4 CPU, 4G internal memories and announcing resource scheduling master
Node M.Scheduling of resource includes 2 CPU and 1G internal memories from available resources in node s2 collecting robot, and announcing resource dispatches main section
Point M.
The distributed resource scheduling method based on resource United Dispatching of dispatching sequence 1-8 and the present embodiment in Fig. 1
Step correspondence, determines distributed computing framework f1 and distributed computing framework f2 priority order, such as distributed computing framework
F1 priority is high, and the available resources being collected into are packaged into list of available resources by scheduling of resource host node M, and the present embodiment is
[{ s1,4cpu, 4g }, { s2,2cpu, 1g }], i.e. scheduling of resource are from node s1 and scheduling of resource from the node s2 CPU called
And internal memory, distributed computing framework f1 is sent to first according to certain priority.
Distributed computing framework f1 determines which task is each scheduling of resource perform from node according to the content of the Resources list.
Distributed computing framework f1 determines that (this task is consumed in 2 CPU, 1G from operation task task1 on node s2 in scheduling of resource
Deposit), in scheduling of resource from operation task task2 on node s1 (this task consumes 1 CPU, 2G internal memories), and this decision is encapsulated
Host node is dispatched into task assignment list [{ f1, task1, s2,2cpu, 1g }, { f1, task2, s1,1cpu, 2g }] announcing resource
M。
Scheduling of resource host node M receives distributed computing framework f1 task assignment list by task distribution formula calculation block
Frame f1 task2 is issued to scheduling of resource from node s1, is responsible for establishing resource spacing container from node s1 by scheduling of resource and performs
Task.
Scheduling of resource host node M receives distributed computing framework f1 task assignment list by distributed computing framework f1
Task task1 be issued to scheduling of resource from node s2, being responsible for establishing resource spacing container from node s2 by scheduling of resource performs
Task.
Scheduling of resource host node M recalculates remaining available resource, by remaining available resource list [{ s1,3cpu, 2g }]
It is sent to the Distributed Architecture f2 of next priority.
Distributed computing framework f2 is determined in scheduling of resource from operation task task1 (this on node s1 according to the Resources list
Task consumes 2 CPU, 2G internal memories), and this decision is packaged into task assignment list [{ f2, task1, s1,2cpu, 2g }] announcement
Know scheduling of resource host node M.
Scheduling of resource host node M receives distributed computing framework f2 task assignment list by task distribution formula calculation block
Frame f2 task1 is issued to s1, and being responsible for establishing resource spacing container from node s1 by scheduling of resource performs task.
Etc. a period of time to be specified, repeat step 1.
Distributed resource scheduling method of the invention based on resource United Dispatching, completion rapidly and efficiently is distributed to some
The distribution of resource needed for calculating most crucial is exactly scheduling mechanism, and this method is using two layer scheduling algorithms：First by scheduling of resource
Some distributed computing framework is given to the schedule level one device coarseness of host node by resource, it is then real by the second-level dispatching device of framework
Existing internal fine-grained scheduling of resource.Realize the resource of shared cluster, and resource is uniformly used
Described above is only the preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, without departing from the technical principles of the invention, some improvement and deformation can also be made, these improve and deformed
Also it should be regarded as protection scope of the present invention.
Claims (5)
1. a kind of distributed resource scheduling method of resource United Dispatching, it is characterised in that including：
Determine the content of available resources；
The available resources being collected into are packaged into list of available resources；
Determine the priority of each distributed computing framework；
According to the order from high to low of the priority of distributed computing framework, list of available resources is selected by distributed computing framework
In resource, after the selection of each distributed computing framework is finished, list of available resources is updated, until all Distributed Calculation frames
Frame all select to finish or list of available resources in there is no available resources；
Restart this method from the content for determining available resources.
2. the distributed resource scheduling method of resource United Dispatching as claimed in claim 1, it is characterised in that available resources bag
Include available CPU and free memory of the scheduling of resource from node.
3. the distributed resource scheduling method of resource United Dispatching as claimed in claim 1, it is characterised in that the Resources list bag
Include：Available scheduling of resource is from node set, and each scheduling of resource includes from node set：Scheduling of resource from the numbering of node,
CPU and free memory information can be used.
4. the distributed resource scheduling method of resource United Dispatching as claimed in claim 1, it is characterised in that each distributed
After Computational frame selection is finished, list of available resources is updated, and the list of available resources after renewal is sent to a time priority
Distributed computing framework.
5. the distributed resource scheduling method of resource United Dispatching as claimed in claim 1, it is characterised in that Distributed Calculation
Framework is according to its internal calculating demand, the resource information in list of available resources, the available resources needed for selection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710374158.9A CN107220114A (en) | 2017-05-24 | 2017-05-24 | Distributed resource scheduling method based on resource United Dispatching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710374158.9A CN107220114A (en) | 2017-05-24 | 2017-05-24 | Distributed resource scheduling method based on resource United Dispatching |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107220114A true CN107220114A (en) | 2017-09-29 |
Family
ID=59944793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710374158.9A Pending CN107220114A (en) | 2017-05-24 | 2017-05-24 | Distributed resource scheduling method based on resource United Dispatching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107220114A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108170417A (en) * | 2017-12-29 | 2018-06-15 | 曙光信息产业(北京)有限公司 | A kind of method and apparatus that high performance job scheduling frame is integrated in MESOS clusters |
| CN109634749A (en) * | 2018-12-20 | 2019-04-16 | 厦门商集网络科技有限责任公司 | A kind of distribution United Dispatching method and apparatus |
| CN111459678A (en) * | 2020-04-02 | 2020-07-28 | 上海极链网络科技有限公司 | Resource scheduling method and device, storage medium and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722576A (en) * | 2012-06-05 | 2012-10-10 | 西安未来国际信息股份有限公司 | Encipherment protection system and encipherment protection method for database in cloud computing environment |
| CN102722413A (en) * | 2012-05-16 | 2012-10-10 | 上海兆民云计算科技有限公司 | Distributed resource scheduling method for desktop cloud cluster |
-
2017
- 2017-05-24 CN CN201710374158.9A patent/CN107220114A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722413A (en) * | 2012-05-16 | 2012-10-10 | 上海兆民云计算科技有限公司 | Distributed resource scheduling method for desktop cloud cluster |
| CN102722576A (en) * | 2012-06-05 | 2012-10-10 | 西安未来国际信息股份有限公司 | Encipherment protection system and encipherment protection method for database in cloud computing environment |
Non-Patent Citations (2)
| Title |
|---|
| 柯尊旺: "一种Mesos平台下多资源调度算法的研究与改进", 《中国优秀硕士学位论文全文数据库信息科技辑（月刊）》 * |
| 胡俊: "集群环境下聚类算法的并行化研究与实现", 《中国优秀硕士学位论文全文数据库信息科技辑（月刊）》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108170417A (en) * | 2017-12-29 | 2018-06-15 | 曙光信息产业(北京)有限公司 | A kind of method and apparatus that high performance job scheduling frame is integrated in MESOS clusters |
| CN108170417B (en) * | 2017-12-29 | 2022-02-11 | 曙光信息产业(北京)有限公司 | Method and device for integrating high-performance job scheduling framework in MESOS cluster |
| CN109634749A (en) * | 2018-12-20 | 2019-04-16 | 厦门商集网络科技有限责任公司 | A kind of distribution United Dispatching method and apparatus |
| CN109634749B (en) * | 2018-12-20 | 2020-10-27 | 厦门商集网络科技有限责任公司 | Distributed unified scheduling method and device |
| CN111459678A (en) * | 2020-04-02 | 2020-07-28 | 上海极链网络科技有限公司 | Resource scheduling method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105808328B (en) | The methods, devices and systems of task schedule | |
| CN106776005B (en) | Resource management system and method for containerized application | |
| CN107038069B (en) | Dynamic label matching DLMS scheduling method under Hadoop platform | |
| US6732139B1 (en) | Method to distribute programs using remote java objects | |
| CN108762896A (en) | One kind being based on Hadoop cluster tasks dispatching method and computer equipment | |
| WO2019001092A1 (en) | Load balancing engine, client, distributed computing system, and load balancing method | |
| CN104375882B (en) | The multistage nested data being matched with high-performance computer structure drives method of calculation | |
| JP2010122758A (en) | Job managing device, job managing method and job managing program | |
| CN104298550A (en) | Hadoop-oriented dynamic scheduling method | |
| CN104239144A (en) | Multilevel distributed task processing system | |
| CN109783225B (en) | Tenant priority management method and system of multi-tenant big data platform | |
| CN113886034A (en) | Task scheduling method, system, electronic device and storage medium | |
| CN103365729A (en) | Dynamic MapReduce dispatching method and system based on task type | |
| CN112596902A (en) | Task scheduling method and device based on CPU-GPU cooperative computing | |
| CN103503412B (en) | For the method and device of scheduling resource | |
| CN104123182A (en) | Map Reduce task data-center-across scheduling system and method based on master-slave framework | |
| CN107220114A (en) | Distributed resource scheduling method based on resource United Dispatching | |
| WO2021126272A1 (en) | Machine learning workload orchestration in heterogeneous clusters | |
| CN110308984A (en) | It is a kind of for handle geographically distributed data across cluster computing system | |
| CN115292016A (en) | Task scheduling method based on artificial intelligence and related equipment | |
| CN105553732B (en) | A kind of distributed network analogy method and system | |
| CN103595654A (en) | HQoS implementation method, device and network equipment based on multi-core CPUs | |
| CN104965762B (en) | A kind of scheduling system towards hybrid task | |
| CN109976873B (en) | Scheduling scheme obtaining method and scheduling method of containerized distributed computing framework | |
| CN103269431B (en) | A kind of cloud transcoding implementation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170929 |
|
| RJ01 | Rejection of invention patent application after publication |