CN109445931A - A kind of big data resource scheduling system and method - Google Patents
A kind of big data resource scheduling system and method Download PDFInfo
- Publication number
- CN109445931A CN109445931A CN201811009398.XA CN201811009398A CN109445931A CN 109445931 A CN109445931 A CN 109445931A CN 201811009398 A CN201811009398 A CN 201811009398A CN 109445931 A CN109445931 A CN 109445931A
- Authority
- CN
- China
- Prior art keywords
- resource
- scheduling
- layer
- module
- load
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5022—Workload threshold
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of big data resource scheduling systems, comprising: acquisition layer, the acquisition layer are acquired and integrate to the state and load information of the resource in system, including data acquisition module and resource consolidation module；Dispatch layer, the dispatch layer realize monitoring, coordination, the distribution of resource, including monitoring resource module, resource coordination module, resource distribution module according to the state and load information of the resource in the acquisition layer and by cloud computing platform；Service layer, the service layer provides strategy setting to the dispatch layer, and provides a user personalized service, to realize the rational management of big data resource, including strategy setting module and personalized service module.The present invention realizes the load balancing of resource, improves resource utilization, and reduces operation cost, further ensures the stability of whole system, improves the service level of cloud computing platform.
Description
Technical field
The present invention relates to computer big data applied technical field, especially a kind of big data resource scheduling system and side
Method.
Background technique
Scheduling of resource refers under specific resource environment, according to certain resource using rule, makes in different resources
The process of resource adjustment is carried out between user.In big data era instantly, the calculating cost of picture, video and artificial intelligence AI
Weight can not be born by being increasingly becoming.Due to the diversification of calculating task and the finiteness of resource, causes resource to be difficult to equilibrium and be used for
Each calculating task；Due to the purchase of equipment, dissolve, circulating forms a large amount of short-term free device, lead to whole resource benefit
With and it is insufficient, how to excavate the existing idling-resource of multiplexing to meet the demand of the calculating business of current magnanimity becomes big number
According to application field problem.
Currently, common resource regulating method has the following problems: one, all kinds of resources cannot unify acquisition, shared and prison
Control, the basis for the scheduling that is deficient in resources；Two, resource cannot be easy to cause frequently to dispatch and be from global angle cooperative scheduling resource
It unites unstable；Three, it is difficult to meet ultra-large scheduling of resource, is finally also difficult to realize the load balancing of resource.
Summary of the invention
In order to overcome the defects of the prior art described above, the present invention provides a kind of big data resource scheduling system, meets super
Extensive scheduling of resource realizes the load balancing of resource, improves resource utilization, and reduces operation cost, further
The stability for ensuring whole system improves the service level of cloud computing platform.
To achieve the above object, the present invention uses following technical scheme, comprising:
A kind of big data resource scheduling system, including consisting of part:
Acquisition layer, the acquisition layer is acquired and integrates to the state and load information of the resource in system, including number
According to acquisition module and resource consolidation module；
The data acquisition module is acquired the state and load information of the resource in system, will collect resource
State and load information are sent to the resource consolidation module；
The resource consolidation module receives and the state and load information of unified managing resource；
Dispatch layer, the dispatch layer are according to the state and load information of the resource in the acquisition layer and flat by cloud computing
Platform realizes monitoring, coordination, the distribution of resource, including monitoring resource module, resource coordination module, resource distribution module；
The monitoring resource module obtains the state and load information of the resource after the resource consolidation module management, right
The state and load information of resource are monitored, and judge the negative of resource according to the pre-set load threshold of strategy setting module
The load condition of resource is sent to the resource coordination module by load state；
The resource coordination module according to the pre-set scheduling strategy of load condition and strategy setting module of resource into
The scheduling request that resource coordinating is dispatched is sent to the resource distribution module by the scheduling of row resource coordinating；
The resource distribution module receives the scheduling request of the resource coordination module, and carries out Resource dynamic allocation；
Service layer, the service layer provide the load threshold and strategy of resource to the dispatch layer, to realize that big data provides
The rational management in source, including strategy setting module；
The strategy setting module is used to that the load threshold of resource to be arranged and the scheduling plan of resource when beyond load threshold
Slightly, the strategy setting module provides the load threshold of resource to the monitoring resource module, and the strategy setting module is to institute
It states resource coordination module and the scheduling strategy of resource is provided.
The collection of resources module is respectively acquired the load of different resource and status information；The type of resource includes
Virtual resource and physical resource, and physical resource includes CPU, memory, network；The state and load information of the resource include money
Total capacity, the usage amount of resource, the utilization rate of resource in source；The resource consolidation module respectively to the total capacity of different resource,
Usage amount, utilization rate are counted.
The strategy setting module is respectively set under the upper loading limit threshold value and load of CPU, memory, network, virtual resource
Threshold value is limited, the load that CPU, memory, network, virtual resource is respectively set is higher than the expanding policy of upper loading limit threshold value, sets respectively
The load of CPU, memory, network, virtual resource is set lower than the reduced technique for loading lower threshold.
If the load of some resource is higher than the upper loading limit threshold value of the resource, the monitoring resource module judges the resource
Load condition be high load；If the load of some resource is lower than the load lower threshold of the resource, the monitoring resource mould
Block judges the load condition of the resource for low-load.
If the load condition of some resource is high load, the resource coordination module calls the expanding policy of the resource,
And resource coordinating scheduling is carried out according to dispatching priority, the scheduling request of resource expansion is sent to resource distribution module, request increases
Add virtual machine；If the load condition of some resource is low-load, the resource coordination module calls the reduced technique of the resource,
And resource coordinating scheduling is carried out according to dispatching priority, the scheduling request that resource is shunk is sent to resource distribution module, request moves
Except virtual machine；
The dispatching priority: the scheduling request in different time, the scheduling request of priority processing request time morning；It is same
Scheduling request in time, the priority that resource is shunk are higher than the priority of resource expansion, the i.e. tune of priority processing resource contraction
Degree request.
The resource distribution module is the scheduling request for receiving the resource coordination module, and according to the scheduling mode of setting
Resource dynamic allocation is carried out, resource expansion is completed or resource is shunk, that is, the increase or removal of virtual machine are completed, to realize resource
Load balancing.
The scheduling mode set is dispatched as multi-layer classification, comprising: scheduling virtual machine layer, server scheduling layer, cluster
Dispatch layer, total activation layer；Described only one data center of total activation layer is responsible for the scheduling that processing colony dispatching layer is sent and asks
It asks；The colony dispatching layer includes all clusters under total activation layer, is responsible for the scheduling request that processing server dispatch layer is sent；
The server scheduling layer includes the Servers-all under each cluster, is responsible for the scheduling that processing scheduling virtual machine layer is sent and asks
It asks；The scheduling virtual machine layer includes all virtual machines under each server, and the scheduling for being responsible for process resource Coordination module is asked
It asks, i.e. the request of increase virtual machine or removal virtual machine；
Shown in the mode of the multi-layer classification scheduling is specific as follows: scheduling virtual machine layer first sends to server scheduling layer and adjusts
Degree request, request are increased or are removed to other virtual machines under same server；If other under same server are virtual
The unscheduled success of machine, then server scheduling layer sends scheduling request to colony dispatching layer again, requests to other under same cluster
Virtual machine under server is increased or is removed；If the unscheduled success of virtual machine under other servers under same cluster,
Then colony dispatching layer finally sends scheduling request to total activation layer, and request is increased or moved to the virtual machine under other clusters
It removes.
The service layer further includes personalized service module, and the personalized service module provides a user personalized clothes
Business requests to carry out Resources Customization scheduling according to user；User's request includes that the scheduling request of resource expansion and resource are shunk
Scheduling request；
The scheduling mode of the personalized service module are as follows: the personalized service module is requested according to user to the money
Source Coordination module sends resource scheduling request, and the resource coordination module sends scheduling of resource to the resource distribution module and asks
It asks, the resource distribution module carries out Resource dynamic allocation according to resource scheduling request.
The present invention also provides a kind of big data resource regulating methods, which comprises the following steps:
S1, acquisition module are acquired the state and load information of the different resource in system；The different resource packet
Include CPU, memory, network, virtual resource；The state and load information of the resource include the use of the total capacity of resource, resource
Amount, the utilization rate of resource；
S2, resource consolidation module are respectively managed collectively the state and load information of the different resource in system, point
Other total capacity, usage amount, utilization rate to different resource counts；
S3, monitoring resource module are monitored the load condition of the resource in system, and pre- according to strategy setting module
The upper loading limit threshold value and load lower threshold for the resource being first arranged judge the load condition of resource for high load or low-load；
S4, according to the load condition of resource, the resource coordination module strategically pre-set scheduling strategy of setup module
Resource coordinating scheduling is carried out with dispatching priority；
S5, resource distribution module according to the scheduling request of resource coordination module, and in the way of multi-layer classification scheduling into
The dynamic allocation of row resource.
In step S5, multi-layer classification scheduling, comprising: scheduling virtual machine layer, server scheduling layer, colony dispatching layer,
Total activation layer；Described only one data center of total activation layer is responsible for the scheduling request that processing colony dispatching layer is sent；The collection
Group's dispatch layer includes all clusters under total activation layer, is responsible for the scheduling request that processing server dispatch layer is sent；The service
Device dispatch layer includes the Servers-all under each cluster, is responsible for the scheduling request that processing scheduling virtual machine layer is sent；The void
Quasi- machine dispatch layer includes all virtual machines under each server, is responsible for the scheduling request of process resource Coordination module, that is, increases
Virtual machine or the request for removing virtual machine；
Shown in the mode of the multi-layer classification scheduling is specific as follows: scheduling virtual machine layer first sends to server scheduling layer and adjusts
Degree request, request are increased or are removed to other virtual machines under same server；If other under same server are virtual
The unscheduled success of machine, then server scheduling layer sends scheduling request to colony dispatching layer again, requests to other under same cluster
Virtual machine under server is increased or is removed；If the unscheduled success of virtual machine under other servers under same cluster,
Then colony dispatching layer finally sends scheduling request to total activation layer, and request is increased or moved to the virtual machine under other clusters
It removes.
The present invention has the advantages that
(1) resource scheduling system of the invention, realize to the acquisition, integration, monitoring, coordination of different resource in system,
The overall process of distribution and personalized service is supervised, and is effectively improved the whole efficiency of scheduling of resource, is further improved money
The dynamic dispatching ability in source.
(2) present invention combine resource coordinating scheduling and multi-layer classification scheduling, to resources all in system carry out unified monitoring,
Cooperative scheduling, dynamic allocation avoid the occurrence of the wasting of resources and the phenomenon that resource is excessively dispatched to realize the load balancing of resource,
Maximally utilise existing resource, improve resource utilization, reduces operation cost, further ensure whole system
Stability, improve the service level of cloud computing platform.
Detailed description of the invention
Fig. 1 is a kind of structural schematic diagram of big data resource scheduling system of the invention.
Fig. 2 is a kind of method flow diagram of big data resource regulating method of the invention.
Fig. 3 is the schematic diagram of multi-layer classification scheduling.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
As shown in Figure 1, a kind of big data resource scheduling system, including consisting of part:
Acquisition layer, the acquisition layer is acquired and integrates to the state and load information of the resource in system, including number
According to acquisition module and resource consolidation module；
The data acquisition module is acquired the state and load information of the resource in system, will collect resource
State and load information are sent to the resource consolidation module；
The resource consolidation module receives and the state and load information of unified managing resource；
Dispatch layer, the dispatch layer are according to the state and load information of the resource in the acquisition layer and flat by cloud computing
Platform realizes monitoring, coordination, the distribution of resource, including monitoring resource module, resource coordination module, resource distribution module；
The monitoring resource module obtains the state and load information of the resource after the resource consolidation module management, right
The state and load information of resource are monitored, and judge the negative of resource according to the pre-set load threshold of strategy setting module
The load condition of resource is sent to the resource coordination module by load state；
The resource coordination module according to the pre-set scheduling strategy of load condition and strategy setting module of resource into
The scheduling request that resource coordinating is dispatched is sent to the resource distribution module by the scheduling of row resource coordinating；
The resource distribution module receives the scheduling request of the resource coordination module, and carries out Resource dynamic allocation；
Service layer, the service layer provide the load threshold and strategy of resource to the dispatch layer, to realize that big data provides
The rational management in source, including strategy setting module；
The strategy setting module is used to that the load threshold of resource to be arranged and the scheduling plan of resource when beyond load threshold
Slightly, the strategy setting module provides the load threshold of resource to the monitoring resource module, and the strategy setting module is to institute
It states resource coordination module and the scheduling strategy of resource is provided.
Wherein, the collection of resources module is respectively acquired the load of different resource and status information；The difference
Resource includes CPU, memory, network, virtual resource；Wherein, the type of resource includes virtual resource and physical resource, and physics provides
Source includes CPU, memory, network；The state and load information of the resource include the total capacity of resource, the usage amount of resource, money
The utilization rate in source；The resource consolidation module respectively counts the total capacity of different resource, usage amount, utilization rate.
The upper loading limit of CPU, memory, network, virtual resource is respectively set according to historical data in the strategy setting module
Threshold value and load lower threshold, the load that CPU, memory, network, virtual resource is respectively set are higher than the extension of upper loading limit threshold value
The load of CPU, memory, network, virtual resource is respectively set lower than the reduced technique for loading lower threshold in strategy.It is with CPU
Example, is configured in advance in strategy setting module: cpu load range is f1~f2, i.e., upper loading limit threshold value is f1, loads lower limit
Threshold value is f2；If cpu load is less than f1, resource reduced technique is triggered；If cpu load is greater than f2, money is triggered
Source expanding policy.
The monitoring resource module is monitored the state and load information of resource, and according to the strategy setting module
Pre-set load threshold judges the load condition of resource, if the load of some resource is higher than the upper loading limit threshold of the resource
Value, then the monitoring resource module judges the load condition of the resource for high load；If the load of some resource is lower than the resource
Load lower threshold, then the monitoring resource module judges the load condition of the resource for low-load.By taking CPU as an example, application
Cluster k is in the cpu load of t moment
Wherein, N indicates the number that monitoring data is collected in the unit time, cpukiIndicate i-th virtual machine of cluster k
The total capacity of CPU, cpuusekitmIndicate that i-th virtual machine of cluster k monitors the usage amount of the CPU at moment in tm.It presets
Loading range be f1~f2, if fzcpukiWhen < f1, then the CPU is low-load；If fzcpukiWhen > f2, then the CPU is height
Load.
The resource coordination module is according to the load condition of resource and the pre-set scheduling plan of the strategy setting module
Resource coordinating scheduling is slightly carried out, if the load condition of some resource is high load, the resource coordination module calls the resource
Expanding policy, and according to dispatching priority carry out resource coordinating scheduling, to resource distribution module send resource expansion scheduling
Request, request increase virtual machine；If the load condition of some resource is low-load, the resource coordination module calls the resource
Reduced technique, and according to dispatching priority carry out resource coordinating scheduling, to resource distribution module send resource shrink scheduling
Request, request remove virtual machine.
The dispatching priority: the scheduling request in different time, the scheduling request of priority processing request time morning；It is same
Scheduling request in time, the priority that resource is shunk are higher than the priority of resource expansion, the i.e. tune of priority processing resource contraction
Degree request.
The resource distribution module is the scheduling request for receiving the resource coordination module, and according to multi-layer classification scheduling
Mode carries out Resource dynamic allocation, completes resource expansion or resource is shunk, that is, complete the increase or removal of virtual machine, to realize money
The load balancing in source.
As shown in Figure 3, multi-layer classification scheduling, comprising: scheduling virtual machine layer, server scheduling layer, colony dispatching layer,
Total activation layer；Described only one data center of total activation layer is responsible for the scheduling request that processing colony dispatching layer is sent；The collection
Group's dispatch layer includes all clusters under total activation layer, is responsible for the scheduling request that processing server dispatch layer is sent；The service
Device dispatch layer includes the Servers-all under each cluster, is responsible for the scheduling request that processing scheduling virtual machine layer is sent；The void
Quasi- machine dispatch layer includes all virtual machines under each server, is responsible for the scheduling request of process resource Coordination module, that is, increases
Virtual machine or the request for removing virtual machine.
Shown in the mode of the multi-layer classification scheduling is specific as follows: scheduling virtual machine layer first sends to server scheduling layer and adjusts
Degree request, request are increased or are removed to other virtual machines under same server；If other under same server are virtual
The unscheduled success of machine, then server scheduling layer sends scheduling request to colony dispatching layer again, requests to other under same cluster
Virtual machine under server is increased or is removed；If the unscheduled success of virtual machine under other servers under same cluster,
Then colony dispatching layer finally sends scheduling request to total activation layer, and request is increased or moved to the virtual machine under other clusters
It removes.
The service layer further includes personalized service module, and the personalized service module provides a user personalized clothes
Business requests to carry out Resources Customization scheduling according to user；User's request includes that the scheduling request of resource expansion and resource are shunk
Scheduling request.If business demand is in the peak rise period, user is asked by personalized service module to resource coordination module
Resource expansion is sought, if business demand decline phase at low ebb, user are asked by personalized service module to resource coordination module
It asks resource to shrink, makes maximum resource utilization.
The scheduling mode of the personalized service module are as follows: the personalized service module is requested according to user to the money
Source Coordination module sends resource scheduling request, and the resource coordination module sends scheduling of resource to the resource distribution module and asks
It asks, the resource distribution module carries out Resource dynamic allocation according to resource scheduling request.
As shown in Figure 2, a kind of big data resource regulating method, comprising the following steps:
S1, acquisition module are acquired the state and load information of the different resource in system；The different resource packet
Include CPU, memory, network, virtual resource；The state and load information of the resource include the use of the total capacity of resource, resource
Amount, the utilization rate of resource；
S2, resource consolidation module are respectively managed collectively the state and load information of the different resource in system, point
Other total capacity, usage amount, utilization rate to different resource counts；
S3, monitoring resource module are monitored the load condition of the resource in system, and pre- according to strategy setting module
The upper loading limit threshold value and load lower threshold for the resource being first arranged judge the load condition of resource for high load or low-load；
S4, according to the load condition of resource, the resource coordination module strategically pre-set scheduling strategy of setup module
Resource coordinating scheduling is carried out with dispatching priority；
S5, resource distribution module according to the scheduling request of resource coordination module, and in the way of multi-layer classification scheduling into
The dynamic allocation of row resource.
The present invention is to provide a kind of big data resource scheduling system and methods, meet ultra-large scheduling of resource, realize
The load balancing of resource improves resource utilization, cuts operating costs, further ensure the stability of whole system, mention
The high service level of cloud computing platform, is suitable for big data monitoring resource, coordination and distribution.
The above is only the preferred embodiments of the invention, are not intended to limit the invention creation, all in the present invention
Made any modifications, equivalent replacements, and improvements etc., should be included in the guarantor of the invention within the spirit and principle of creation
Within the scope of shield.
Claims (10)
1. a kind of big data resource scheduling system, which is characterized in that including consisting of part:
Acquisition layer, the acquisition layer is acquired and integrates to the state and load information of the resource in system, including data are adopted
Collect module and resource consolidation module；
The data acquisition module is acquired the state and load information of the resource in system, will collect the state of resource
And load information is sent to the resource consolidation module；
The resource consolidation module receives and the state and load information of unified managing resource；
Dispatch layer, the dispatch layer is according to the state and load information of the resource in the acquisition layer and to pass through cloud computing platform real
Monitoring, coordination, the distribution of existing resource, including monitoring resource module, resource coordination module, resource distribution module；
The monitoring resource module obtains the state and load information of the resource after the resource consolidation module management, to resource
State and load information be monitored, and the load shape of resource is judged according to the pre-set load threshold of strategy setting module
The load condition of resource is sent to the resource coordination module by state；
The resource coordination module is provided according to the pre-set scheduling strategy of load condition and strategy setting module of resource
The scheduling request that resource coordinating is dispatched is sent to the resource distribution module by source cooperative scheduling；
The resource distribution module receives the scheduling request of the resource coordination module, and carries out Resource dynamic allocation；
Service layer, the service layer provides the load threshold and strategy of resource to the dispatch layer, to realize big data resource
Rational management, including strategy setting module；
The strategy setting module is used to that the load threshold of resource to be arranged and the scheduling strategy of resource when beyond load threshold, institute
It states strategy setting module and provides the load threshold of resource to the monitoring resource module, the strategy setting module is to the resource
The scheduling strategy of Coordination module offer resource.
2. a kind of big data resource scheduling system according to claim 1, which is characterized in that the collection of resources module point
It is other that the load of different resource and status information are acquired；The type of resource includes virtual resource and physical resource, and physics
Resource includes CPU, memory, network；The state and load information of the resource include the total capacity of resource, the usage amount of resource,
The utilization rate of resource；The resource consolidation module respectively counts the total capacity of different resource, usage amount, utilization rate.
3. a kind of big data resource scheduling system according to claim 2, which is characterized in that the strategy setting module point
Not She Zhi CPU, memory, network, virtual resource upper loading limit threshold value and load lower threshold, CPU, memory, net is respectively set
The load of network, virtual resource is higher than the expanding policy of upper loading limit threshold value, and CPU, memory, network, virtual resource is respectively set
Reduced technique of the load lower than load lower threshold.
4. a kind of big data resource scheduling system according to claim 3, which is characterized in that if the load of some resource is high
In the upper loading limit threshold value of the resource, then the monitoring resource module judges the load condition of the resource for high load；If some
The load of resource is lower than the load lower threshold of the resource, then the monitoring resource module judges the load condition of the resource to be low
Load.
5. a kind of big data resource scheduling system according to claim 4, which is characterized in that if the load shape of some resource
State is high load, then the resource coordination module calls the expanding policy of the resource, and carries out resource association according to dispatching priority
With scheduling, the scheduling request of resource expansion is sent to resource distribution module, request increases virtual machine；If the load shape of some resource
State is low-load, then the resource coordination module calls the reduced technique of the resource, and carries out resource association according to dispatching priority
With scheduling, the scheduling request that resource is shunk is sent to resource distribution module, request removes virtual machine；
The dispatching priority: the scheduling request in different time, the scheduling request of priority processing request time morning；The same time
Interior scheduling request, the priority that resource is shunk are higher than the priority of resource expansion, i.e., the scheduling that priority processing resource is shunk is asked
It asks.
6. a kind of big data resource scheduling system according to claim 5, which is characterized in that the resource distribution module is
The scheduling request of the resource coordination module is received, and carries out Resource dynamic allocation according to the scheduling mode of setting, completes resource
Extension or resource are shunk, that is, the increase or removal of virtual machine are completed, to realize the load balancing of resource.
7. a kind of big data resource scheduling system according to claim 6, which is characterized in that the scheduling mode of the setting
For multi-layer classification scheduling, comprising: scheduling virtual machine layer, server scheduling layer, colony dispatching layer, total activation layer；The total activation
Only one data center of layer is responsible for the scheduling request that processing colony dispatching layer is sent；The colony dispatching layer includes total activation
All clusters under layer are responsible for the scheduling request that processing server dispatch layer is sent；The server scheduling layer includes each collection
Servers-all under group is responsible for the scheduling request that processing scheduling virtual machine layer is sent；The scheduling virtual machine layer includes each
All virtual machines under server are responsible for the scheduling request of process resource Coordination module, i.e. increase virtual machine or removal virtual machine
Request；
Shown in the mode of the multi-layer classification scheduling is specific as follows: scheduling virtual machine layer first sends scheduling to server scheduling layer and asks
It asks, request is increased or removed to other virtual machines under same server；If other virtual machines under same server are not
It dispatches successfully, then server scheduling layer sends scheduling request to colony dispatching layer again, requests to other services under same cluster
Virtual machine under device is increased or is removed；If the unscheduled success of virtual machine under other servers under same cluster, collects
Group's dispatch layer finally sends scheduling request to total activation layer, and request is increased or removed to the virtual machine under other clusters.
8. a kind of big data resource scheduling system according to claim 1, which is characterized in that the service layer further includes
Property service module, the personalized service module provides a user personalized service, is requested to carry out Resources Customization according to user
Scheduling；User's request includes the scheduling request of resource expansion and the scheduling request that resource is shunk；
The scheduling mode of the personalized service module are as follows: the personalized service module requests to assist to the resource according to user
Mode transfer block sends resource scheduling request, and the resource coordination module sends resource scheduling request, institute to the resource distribution module
It states resource distribution module and Resource dynamic allocation is carried out according to resource scheduling request.
9. a kind of big data resource regulating method, which comprises the following steps:
S1, acquisition module are acquired the state and load information of the different resource in system；The different resource includes
CPU, memory, network, virtual resource；The state and load information of the resource include the total capacity of resource, the usage amount of resource,
The utilization rate of resource；
S2, resource consolidation module is respectively managed collectively the state and load information of the different resource in system, right respectively
Total capacity, usage amount, the utilization rate of different resource are counted；
S3, monitoring resource module are monitored the load condition of the resource in system, and are set in advance according to strategy setting module
The upper loading limit threshold value and load lower threshold for the resource set judge the load condition of resource for high load or low-load；
S4, according to the load condition of resource, resource coordination the module strategically pre-set scheduling strategy of setup module and tune
It spends priority and carries out resource coordinating scheduling；
S5, resource distribution module are provided according to the scheduling request of resource coordination module, and in the way of multi-layer classification scheduling
The dynamic allocation in source.
10. a kind of big data resource regulating method according to claim 9, which is characterized in that in step S5, the multilayer
Graded dispatching, comprising: scheduling virtual machine layer, server scheduling layer, colony dispatching layer, total activation layer；The total activation layer only has
One data center is responsible for the scheduling request that processing colony dispatching layer is sent；The colony dispatching layer includes under total activation layer
All clusters are responsible for the scheduling request that processing server dispatch layer is sent；The server scheduling layer includes under each cluster
Servers-all is responsible for the scheduling request that processing scheduling virtual machine layer is sent；The scheduling virtual machine layer includes each server
Under all virtual machines, be responsible for process resource Coordination module scheduling request, i.e., increase virtual machine or remove virtual machine request；
Shown in the mode of the multi-layer classification scheduling is specific as follows: scheduling virtual machine layer first sends scheduling to server scheduling layer and asks
It asks, request is increased or removed to other virtual machines under same server；If other virtual machines under same server are not
It dispatches successfully, then server scheduling layer sends scheduling request to colony dispatching layer again, requests to other services under same cluster
Virtual machine under device is increased or is removed；If the unscheduled success of virtual machine under other servers under same cluster, collects
Group's dispatch layer finally sends scheduling request to total activation layer, and request is increased or removed to the virtual machine under other clusters.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811009398.XA CN109445931A (en) | 2018-08-31 | 2018-08-31 | A kind of big data resource scheduling system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811009398.XA CN109445931A (en) | 2018-08-31 | 2018-08-31 | A kind of big data resource scheduling system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109445931A true CN109445931A (en) | 2019-03-08 |
Family
ID=65533074
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811009398.XA Pending CN109445931A (en) | 2018-08-31 | 2018-08-31 | A kind of big data resource scheduling system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109445931A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111897636A (en) * | 2020-07-14 | 2020-11-06 | 广东科徕尼智能科技有限公司 | Scheduling method, device and storage medium based on data calculation and analysis |
| CN112148439A (en) * | 2019-06-28 | 2020-12-29 | 浙江宇视科技有限公司 | Task processing method, device, equipment and storage medium |
| WO2021046775A1 (en) * | 2019-09-11 | 2021-03-18 | 阿里巴巴集团控股有限公司 | Resource scheduling, application, and control methods, apparatus, system, and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104009862A (en) * | 2013-02-27 | 2014-08-27 | 腾讯科技（深圳）有限公司 | Equipment scheduling method and system |
| CN104010028A (en) * | 2014-05-04 | 2014-08-27 | 华南理工大学 | Dynamic virtual resource management strategy method for performance weighting under cloud platform |
| CN104836819A (en) * | 2014-02-10 | 2015-08-12 | 阿里巴巴集团控股有限公司 | Dynamic load balancing method and system, and monitoring and dispatching device |
| US20160203026A1 (en) * | 2013-09-27 | 2016-07-14 | Hewlett Packard Enterprise Development Lp | Processing a hybrid flow associated with a service class |
| CN106686136A (en) * | 2017-02-24 | 2017-05-17 | 郑州云海信息技术有限公司 | Cloud resource scheduling method and device |
| CN106874099A (en) * | 2015-12-10 | 2017-06-20 | 中国电信股份有限公司 | Load dispatching method, device and cloud computing system based on traffic handing capacity |
| CN107066319A (en) * | 2017-01-17 | 2017-08-18 | 北京国电通网络技术有限公司 | A kind of multidimensional towards heterogeneous resource dispatches system |
| CN107689892A (en) * | 2017-09-12 | 2018-02-13 | 中国人民解放军信息工程大学 | Virtual machine Placement Strategy selecting device, attack defense method and virtual machine deployment method, device coexists |
| CN107943559A (en) * | 2017-11-21 | 2018-04-20 | 广东奥飞数据科技股份有限公司 | A kind of big data resource scheduling system and its method |
| CN108023958A (en) * | 2017-12-08 | 2018-05-11 | 中国电子科技集团公司第二十八研究所 | A kind of resource scheduling system based on cloud platform resource monitoring |
| CN108429815A (en) * | 2018-03-23 | 2018-08-21 | 重庆邮电大学 | Dynamic resource scheduling method based on OpenStack |
-
2018
- 2018-08-31 CN CN201811009398.XA patent/CN109445931A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104009862A (en) * | 2013-02-27 | 2014-08-27 | 腾讯科技（深圳）有限公司 | Equipment scheduling method and system |
| US20160203026A1 (en) * | 2013-09-27 | 2016-07-14 | Hewlett Packard Enterprise Development Lp | Processing a hybrid flow associated with a service class |
| CN104836819A (en) * | 2014-02-10 | 2015-08-12 | 阿里巴巴集团控股有限公司 | Dynamic load balancing method and system, and monitoring and dispatching device |
| CN104010028A (en) * | 2014-05-04 | 2014-08-27 | 华南理工大学 | Dynamic virtual resource management strategy method for performance weighting under cloud platform |
| CN106874099A (en) * | 2015-12-10 | 2017-06-20 | 中国电信股份有限公司 | Load dispatching method, device and cloud computing system based on traffic handing capacity |
| CN107066319A (en) * | 2017-01-17 | 2017-08-18 | 北京国电通网络技术有限公司 | A kind of multidimensional towards heterogeneous resource dispatches system |
| CN106686136A (en) * | 2017-02-24 | 2017-05-17 | 郑州云海信息技术有限公司 | Cloud resource scheduling method and device |
| CN107689892A (en) * | 2017-09-12 | 2018-02-13 | 中国人民解放军信息工程大学 | Virtual machine Placement Strategy selecting device, attack defense method and virtual machine deployment method, device coexists |
| CN107943559A (en) * | 2017-11-21 | 2018-04-20 | 广东奥飞数据科技股份有限公司 | A kind of big data resource scheduling system and its method |
| CN108023958A (en) * | 2017-12-08 | 2018-05-11 | 中国电子科技集团公司第二十八研究所 | A kind of resource scheduling system based on cloud platform resource monitoring |
| CN108429815A (en) * | 2018-03-23 | 2018-08-21 | 重庆邮电大学 | Dynamic resource scheduling method based on OpenStack |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148439A (en) * | 2019-06-28 | 2020-12-29 | 浙江宇视科技有限公司 | Task processing method, device, equipment and storage medium |
| CN112148439B (en) * | 2019-06-28 | 2024-03-08 | 浙江宇视科技有限公司 | Task processing method, device, equipment and storage medium |
| WO2021046775A1 (en) * | 2019-09-11 | 2021-03-18 | 阿里巴巴集团控股有限公司 | Resource scheduling, application, and control methods, apparatus, system, and storage medium |
| CN114008996A (en) * | 2019-09-11 | 2022-02-01 | 阿里巴巴集团控股有限公司 | Resource scheduling, applying and regulating method, device, system and storage medium |
| CN114008996B (en) * | 2019-09-11 | 2024-05-07 | 阿里巴巴集团控股有限公司 | Resource scheduling, application and regulation method, equipment, system and storage medium |
| CN111897636A (en) * | 2020-07-14 | 2020-11-06 | 广东科徕尼智能科技有限公司 | Scheduling method, device and storage medium based on data calculation and analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111953526B (en) | Hierarchical computational power network arrangement method, device and storage medium | |
| CN102111337B (en) | Method and system for task scheduling | |
| CN109672627A (en) | Method for processing business, platform, equipment and storage medium based on cluster server | |
| CN112583861A (en) | Service deployment method, resource configuration method, system, device and server | |
| CN105245617A (en) | Container-based server resource supply method | |
| CN105159775A (en) | Load balancer based management system and management method for cloud computing data center | |
| CN104836819A (en) | Dynamic load balancing method and system, and monitoring and dispatching device | |
| CN103559072A (en) | Method and system for implementing bidirectional auto scaling service of virtual machines | |
| CN109445931A (en) | A kind of big data resource scheduling system and method | |
| CN110888732B (en) | Resource allocation method, equipment, device and computer readable storage medium | |
| CN104917805A (en) | Load sharing method and equipment | |
| CN106790636A (en) | A kind of equally loaded system and method for cloud computing server cluster | |
| CN107943559A (en) | A kind of big data resource scheduling system and its method | |
| CN103428008A (en) | Big data distribution strategy oriented to multiple user groups | |
| CN101227416A (en) | Method for distributing link bandwidth in communication network | |
| CN114389955B (en) | Method for managing heterogeneous resource pool of embedded platform | |
| CN116662020B (en) | Dynamic management method and system for application service, electronic equipment and storage medium | |
| CN106131227A (en) | Balancing method of loads, meta data server system and load balance system | |
| CN111459641A (en) | Cross-machine-room task scheduling and task processing method and device | |
| CN112463395A (en) | Resource allocation method, device, equipment and readable storage medium | |
| CN117971499B (en) | Resource allocation method, device, electronic equipment and storage medium | |
| CN117370138A (en) | High capacity distributed storage system | |
| CN103297511A (en) | Client/ server scheduling method and system under highly dynamic environment | |
| CN107203256A (en) | Energy-conservation distribution method and device under a kind of network function virtualization scene | |
| CN112565388A (en) | Distributed acquisition service scheduling system and method based on scoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190308 |
|
| RJ01 | Rejection of invention patent application after publication |