CN113204433B - Dynamic allocation method, device, equipment and storage medium for cluster resources - Google Patents
Dynamic allocation method, device, equipment and storage medium for cluster resources Download PDFInfo
- Publication number
- CN113204433B CN113204433B CN202110747732.7A CN202110747732A CN113204433B CN 113204433 B CN113204433 B CN 113204433B CN 202110747732 A CN202110747732 A CN 202110747732A CN 113204433 B CN113204433 B CN 113204433B
- Authority
- CN
- China
- Prior art keywords
- resource
- queue
- queues
- resources
- configuration data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
Abstract
The invention discloses a dynamic allocation method, a device, equipment and a storage medium of cluster resources. The method comprises the following steps: acquiring resource demand information and resource configuration data of a plurality of queues, wherein the resource configuration data comprises minimum queue resources; determining the states of the queues according to the resource demand information and the resource configuration data; when the state of at least one queue in the plurality of queues is a starvation mode, preferentially allocating resources to the at least one queue according to the resource configuration data, wherein the starvation mode is that the available resources of the queue are smaller than the minimum resources of the queue. The method provided by the embodiment of the invention can dynamically adjust the distribution of the cluster resources, thereby improving the utilization rate of the cluster resources.
Description
Technical Field
The present invention relates to the field of resource management technologies, and in particular, to a method, an apparatus, a device, and a storage medium for dynamically scheduling cluster resources.
Background
With the continuous expansion and diversification of enterprise business, data is also growing exponentially. More and more companies choose to ecologically build a data warehouse by a distributed system infrastructure (Hadoop), more and more requirements for complex application office scenes are built on a large data cluster, and server resources are increasingly tense.
At present, in order to prevent a disordered state of data and cluster resources, a multi-tenant registration Resource management mode of a cluster is widely adopted by related enterprises, and multi-tenant Resource management is mainly realized by Another Resource coordinator (Yarn) of an adjustment framework.
The current cluster resource management method cannot dynamically schedule cluster resources, so that the utilization rate of the cluster resources is low.
Disclosure of Invention
Embodiments of the present invention provide a method, an apparatus, a device, and a storage medium for dynamically allocating cluster resources, which can dynamically adjust allocation of cluster resources, thereby improving utilization rate of cluster resources.
In a first aspect, an embodiment of the present invention provides a method for dynamically allocating cluster resources, where the method includes:
acquiring resource demand information and resource configuration data of a plurality of queues, wherein the resource configuration data comprises minimum queue resources; determining the states of a plurality of queues according to the resource demand information and the resource configuration data; when the state of at least one queue in the plurality of queues is a starvation mode, preferentially allocating resources to the at least one queue according to the resource configuration data, wherein the starvation mode is that the available resources of the queue are smaller than the minimum resources of the queue.
In a possible implementation manner, before preferentially allocating resources to at least one queue according to the resource configuration data when the status of the at least one queue in the plurality of queues is in the starvation mode, the method further includes: calculating the usable resource according to the resource configuration data.
In one possible implementation manner, when the states of at least two queues in the plurality of queues are in the starvation mode, preferentially allocating resources to at least one queue according to the resource configuration data includes: and allocating resources for the at least two queues according to the resource configuration data and the preset priorities of the at least two queues.
In a possible implementation manner, allocating resources to the at least two queues according to the resource configuration data and the preset priorities of the at least two queues includes: determining the used resource proportion of each queue in the at least two queues according to the resource configuration data of the at least two queues; when the used resource proportion of each queue in the at least two queues is equal, acquiring the resource usage amount of each queue in the at least two queues; calculating the ratio of the resource usage amount to the preset priority to obtain the weights of at least two queues; and sequentially distributing resources for the corresponding queues according to the sequence of the weights of the at least two queues from small to large.
In a second aspect, an embodiment of the present invention provides an apparatus for dynamically allocating cluster resources, where the apparatus includes: the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring resource demand information and resource configuration data of a plurality of queues, and the resource configuration data comprises minimum queue resources; the determining module is used for determining the states of the queues according to the resource demand information and the resource configuration data; and the allocation module is used for preferentially allocating resources to at least one queue according to the resource configuration data when the state of at least one queue in the queues is in a starvation mode, wherein the starvation mode is that the available resources of the queue are smaller than the minimum resources of the queue.
In a possible implementation manner, the allocation module is further configured to obtain resource demand information and resource configuration data of the plurality of queues; and also for calculating available resources from the resource configuration data.
In a possible implementation manner, the allocation module is configured to allocate resources to at least two queues according to the resource configuration data and the preset priorities of the at least two queues when the states of the at least two queues in the plurality of queues are in the starvation mode.
In a possible implementation manner, the allocation module is configured to determine, according to the resource configuration data of the at least two queues, a used resource proportion of each queue of the at least two queues; the method is also used for acquiring the resource usage amount of each queue in the at least two queues when the used resource proportion of each queue in the at least two queues is equal; the method is also used for calculating the ratio of the resource usage amount to the preset priority to obtain at least two weights; and the resource allocation module is also used for sequentially allocating resources to the corresponding queues according to the sequence of the at least two weights from small to large.
In a third aspect, an embodiment of the present invention provides an electronic device, including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, performs the method as in the first aspect or any possible implementation of the first aspect.
In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the method as in the first aspect or any possible implementation manner of the first aspect.
The technical scheme provided by the embodiment of the invention at least has the following beneficial effects: the method determines the states of the queues according to the resource demand information and the resource configuration data of the queues, and preferentially allocates resources to at least one queue according to the resource configuration data of the at least one queue when the state of the at least one queue is in a starvation mode. The allocation of the cluster resources can be dynamically adjusted according to the priority of the queue, so that the utilization rate of the cluster resources is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a dynamic allocation architecture of cluster resources according to an embodiment of the present invention.
Fig. 2 is a method for dynamically allocating cluster resources according to an embodiment of the present invention.
Fig. 3 is another method for dynamically allocating cluster resources according to an embodiment of the present invention.
Fig. 4 is another method for dynamically allocating cluster resources according to an embodiment of the present invention.
Fig. 5 is a device for dynamically allocating cluster resources according to an embodiment of the present invention.
Fig. 6 is an electronic device according to an embodiment of the present invention.
Detailed Description
Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the present invention.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
With the continuous expansion and diversification of enterprise business, data is also growing exponentially. More and more companies choose to build a data warehouse with Hadoop ecology, more and more requirements for complex application office scenes are built on a big data cluster, and server resources are more and more tense.
At present, in order to prevent the occurrence of a disordered state of data and cluster resources, related enterprises widely adopt a cluster multi-tenant registration resource management mode, and realize multi-tenant resource management mainly through an adjustment framework Yarn.
The current cluster resource management method cannot dynamically schedule cluster resources, so that the utilization rate of the cluster resources is low.
Embodiments of the present invention provide a method, an apparatus, a device, and a storage medium for dynamically allocating cluster resources, where states of a plurality of queues are determined according to resource demand information and resource configuration data of the plurality of queues, and when a state of at least one queue is a starvation mode, resources are preferentially allocated to the at least one queue according to the resource configuration data of the at least one queue. The allocation of the cluster resources can be dynamically adjusted according to the priority of the queue, so that the utilization rate of the cluster resources is improved.
In an embodiment provided by the present invention, a Fair resource Scheduler (FS) of Yarn is used to schedule cluster resources. The method comprises the steps that firstly, an FS divides total cluster resources into queues (also called resource pools), a tenant (also called a registration system) corresponds to one queue in the cluster, when each queue is created, demand information of the tenant is evaluated according to actual development demand of the tenant, resource configuration data are configured for the queue corresponding to the tenant according to the demand information of the tenant, resource scheduling algorithm operation is carried out through the resource configuration data, and all tasks need to run in the queue and are managed and limited by the queue resource configuration data.
In an embodiment provided by the present invention, the FS adopts a three-layer scheduling manner, and the FS adopts a Fair Share Policy (FSP) in the scheduling of each layer, and the three-layer scheduling includes:
1. and the FS submits the tasks to the queue corresponding to the tenant. All tasks of the tenant need to be submitted to the queue via the FS. For example: and the tenant A corresponds to the queue A, and the FS submits the tasks submitted by the tenant A to the queue A.
And 2, the FS allocates resources for the application programs in the queue according to the dynamic allocation method of the cluster resources.
And 3, the FS allocates resources for the plurality of tasks of the application program according to the dynamic allocation method of the cluster resources.
The following describes, with reference to fig. 1, a dynamic allocation architecture of cluster resources according to an embodiment of the present invention, where the architecture includes a resource scheduler 101, a parameter configuration module 102, and a task scheduling module 103.
The resource scheduler 101 is configured to allocate resources to the queues and the applications in the queues and the multiple tasks of the applications according to a three-tier scheduling manner.
The parameter configuration module 102 is configured to configure resource configuration data for the queue and the applications in the queue according to the tenant demand information.
The resource configuration data may include at least one of: queue maximum resource, queue minimum resource, reserved resource, queue used resource, application manager maximum resource, application manager minimum resource, application manager used resource, maximum number of applications that can be run, number of applications that are running, number of applications that are piled, fixed Fair Share resource (Steady Fair Share), whether occupied resources are allowed.
Queue maximum resources: the platform is self-adjusted for improving the resource utilization rate of the cluster, is non-guaranteed resource and is set for improving the resource utilization rate of the cluster. The queue may use more than the queue minimum resource but no more than the queue maximum resource when the cluster has free resources. When the resource used by a certain queue exceeds the minimum resource, the part of resource exceeding the minimum resource is non-guaranteed resource, which is equivalent to temporarily occupying the minimum resource of other queues, and if other queues have resource requirements, the part of resource exceeding the minimum resource can be reclaimed, so that the part of resource exceeding the minimum resource is unstable, and is possibly influenced under the condition that tasks are on-line in other queues, and task delay is caused. When the resource occupied by the queue reaches the maximum resource of the queue, the resource can not be allocated to the resource, even if the task with resource demand in the queue can only wait for other tasks in the queue to release the resource, the released resource can be reallocated by the resource scheduler.
Queue minimum resources: the resources which are prepared and applied by the user during registration are guaranteed resources.
The reserved resources are: typically the current queue resources reserved to cope with special cases, which do not participate in the dynamic allocation of cluster resources.
Each application submitted by a user comprises an application manager, and the main functions comprise: negotiating with a resource scheduler to obtain resources; further distributing the obtained application program tasks to internal application program tasks (namely secondary distribution of resources); communicating with a resource and task manager on each node to start/stop application tasks; and monitoring the running states of all application program tasks, and re-applying resources for the application program tasks to restart the application program tasks when the running of the application program tasks fails.
Application manager maximum resources: the platform allocates the maximum resource available for the application.
Application manager minimum resources: the resource is a resource which is reported and applied by the user during registration and is a guaranteed resource.
Resources used by the application manager: the resources that are already in use by the applications in the queue.
Maximum number of applications that can run: the maximum number of applications that can be run in each queue.
Number of applications running: the number of applications running in each queue.
Number of applications piled up: the number of applications waiting to run in each queue.
Fixed fair sharing of resources: and the Yarn calculates the maximum available resource which can be allocated to each queue according to the weight of each queue and the maximum and minimum operable resources. However, the fixed fair sharing resource belongs to a theoretical value, as long as the cluster total resource and the queue configuration are unchanged, the fixed fair sharing resource value of each queue is unchanged, and the instantaneous fair sharing resource is a dynamic change value, which represents the maximum resource that can be used by each queue in operation.
Whether the occupied resources are allowed: when the multi-tenant exists, resource preemption can not be started for the tenants in a specific scene, such as users with low priority.
The task scheduling module 103 is used for scheduling and executing the tasks of the tenants according to the FSP policy.
The dynamic allocation architecture of the cluster resources provided by the embodiment of the invention can dynamically adjust the allocation of the cluster resources according to the priority of the queue and the application program, thereby improving the utilization rate of the cluster resources.
A method for dynamically allocating cluster resources according to an embodiment of the present invention will be described in detail below with reference to fig. 2.
As shown in fig. 2, the method may include the steps of:
s210, acquiring resource demand information and resource configuration data of a plurality of queues, wherein the resource configuration data comprises minimum queue resources.
When a cluster multi-tenant registration management mode is adopted, the queues correspond to tenants one by one, when the queues are established for each tenant, the requirement information of the tenant is evaluated according to the actual development requirement of the tenant, and resource configuration data is configured according to the queues corresponding to the tenants according to the requirement information of the tenant.
And traversing all queues through the resource scheduler, checking whether each queue has a resource requirement, determining a plurality of queues with the resource requirement, and acquiring resource requirement information and resource configuration data of the plurality of queues with the resource requirement by the resource scheduler.
The resource scheduler may traverse the states of all queues according to a preset period.
S220, determining the states of the queues according to the resource demand information and the resource configuration data.
And the resource scheduler determines the states of the queues according to the queues corresponding to the resource demand information and the resource configuration data of the queues through the actual use condition of the multi-tenant resources.
The status of each queue may be one of a minimum resource pattern, a starvation pattern, a demand resource pattern, a maximum resource pattern.
The minimum resource mode is that the queue usable resource is equal to the queue minimum resource; starvation mode is that the available resources of the queue are less than the queue minimum resources; the demand resource mode is that the resource required by the queue is greater than the minimum resource of the queue and less than the maximum resource of the queue; the maximum resource mode is that the resource required by the queue is equal to the queue maximum resource.
S230, when the status of at least one queue in the plurality of queues is in the starvation mode, preferentially allocating resources to the at least one queue according to the resource configuration data.
When the state of at least one queue in the plurality of queues with resource demands is in a starvation mode, resources are preferentially allocated to the at least one queue with resource demands according to resource configuration data of queues without resource demands in all the queues and the queue in the starvation mode.
The resource scheduler may invoke a queue minimum resource of the resource demand free queue for the queue that is ordered top.
In an embodiment of the present invention, when the status of one of the queues is starvation mode, resources are preferentially allocated to the queue with resource demand according to resource configuration data of queues without resource demand and the queue with starvation mode.
The resource scheduler may invoke the queue minimum resource of the resource demand free queue for one queue with resource demand.
The method provided by the embodiment of the invention can preferentially allocate resources to one queue when the state of the queue is in the starvation mode.
In an embodiment of the present invention, when the states of at least two of the plurality of queues are in the starvation mode, the preferentially allocating resources to the at least one queue according to the resource configuration data includes: and allocating resources for the at least two queues according to the resource configuration data and the preset priority of the at least two queues.
The user can preset priority for the queue, the preset priority can be 1-7, and the higher the number is, the higher the priority is. And when the states of at least two queues in the plurality of queues are in a starvation mode, allocating resources for the at least two queues according to the resource configuration data and the preset priorities of the at least two queues.
The method provided by the embodiment of the invention can allocate resources for at least two queues according to the priorities of the at least two queues when the states of the at least two queues are in the starvation mode.
The method provided by the embodiment of the invention can determine the states of the queues according to the resource demand information and the resource configuration data of the queues, and preferentially allocate resources to at least one queue according to the resource configuration data of the at least one queue when the state of the at least one queue is in the starvation mode. The resource scheduler can call the queue minimum resource of the queue without resource demand for the queue in the front sequence, and dynamically adjust the distribution of the cluster resources according to the priority of the queue, thereby improving the utilization rate of the cluster resources.
In an embodiment of the present invention, before preferentially allocating resources to at least one queue according to the resource configuration data when the status of the at least one queue in the plurality of queues is in the starvation mode, the method further includes: calculating the usable resource according to the resource configuration data.
The resource requirement information may include the resources required by the queue.
The resource scheduler dynamically schedules resources, when the queue A has no resource demand or the required resources are smaller than the queue minimum resources of the queue A, and the resources required by the queue B are larger than the queue minimum resources of the queue B, the queue B can temporarily occupy part of the queue minimum resources of the queue A, when the queue A has more resource demands, the resource scheduler can recover the resources occupied by the queue B in the queue A, the available resources of the queue A comprise the part of the resources recovered by the queue A, in addition, the queue A can also occupy the resources of other queues when the required resources are larger than the queue minimum resources, and the queue minimum resources of the queue A subtract the used resources of the queue to obtain the available resources of the queue A.
The method provided by the embodiment of the invention can determine a plurality of queues with resource requirements by traversing the states of the queues, and obtain the available resources according to the resource configuration data of the plurality of queues, thereby providing a basis for determining the states of the queues, so as to dynamically allocate resources to the queues and improve the utilization rate of cluster resources.
In an embodiment provided by the present invention, as shown in fig. 3, the allocating resources to the at least two queues according to the resource configuration data and the preset priorities of the at least two queues includes the following steps:
s310, determining the used resource proportion of each queue in the at least two queues according to the resource configuration data of the at least two queues.
In one example, the resource required by each of the at least two queues needs to be acquired, the used resource ratio = used resource/max { min { queue minimum resource, required resource },1}, and the resource scheduler allocates resources to the queues in order of the used resource ratio from small to large.
S320, when the used resource proportion of each queue in the at least two queues is equal, acquiring the resource usage amount of each queue in the at least two queues.
The resource usage is a size of used resources of each of the at least two queues.
S330, calculating the ratio of the resource usage amount to the preset priority to obtain the weights of the at least two queues.
The resource usage of the queue can reflect the usage of the queue resources, and the ratio of the resource usage to the preset priority is used as the weight of the queue, that is, the weight of the queue is determined from the usage of the queue resources and the dimension of the preset priority of the queue, so that the weight can be changed according to the change of the usage of the queue resources, and the resources allocated to the queue can be dynamically changed according to the weight of the queue.
S340, sequentially distributing resources to the corresponding queues according to the sequence of the weights of the at least two queues from small to large.
The resource scheduler may invoke a queue minimum resource of the resource demand free queue for the queue that is ordered top.
The method provided by the embodiment of the invention can determine the sequence of the allocated resources of the at least two queues according to the resource configuration data and the preset priority of the at least two queues when the at least two queues are in the starvation mode, the queues in the front of the sequence can occupy the minimum queue resources of the queues without resource demands, and the utilization rate of the cluster resources is improved.
In an embodiment provided by the present invention, the dynamic allocation method for cluster resources provided by the embodiment of the present invention may further allocate resources for the application programs in the queue, and the method for allocating resources includes:
1. the method comprises the steps of obtaining resource demand information and resource configuration data of a plurality of application programs in the same queue, wherein the resource configuration data comprise the minimum resource which can be used by an application program manager.
The resource scheduler traverses the states of all the application programs, checks whether each application program has a resource requirement, and determines a plurality of application programs with the resource requirement.
2. And determining the states of the plurality of application programs according to the resource demand information and the resource configuration data.
The resource requirement information may include resources required by the application.
Because the resource scheduler dynamically schedules the resources, in the same queue, when the application program C has no resource demand or the required resources are less than the application program manager minimum resources of the application program C, and the resources required by the application program D are greater than the application program manager minimum resources of the application program D, the application program D may temporarily occupy part of the application program manager minimum resources of the application program C, when the application program C has more resource demands, the resource scheduler will reclaim the resources occupied by the application program D by the application program C, and the resources that the application program C can use include the part of the resources reclaimed by the application program C. And the application program C can also occupy the resources of other application programs when the required resources are greater than the minimum resources of the application program manager, and the available resources of the application program C are obtained by subtracting the used resources of the application program C from the minimum resources of the application program sequence C.
3. When the state of at least one application program in the plurality of application programs is a starvation mode, preferentially allocating resources for the at least one application program according to the resource configuration data, wherein the starvation mode is that the available resources of the application program are smaller than the minimum resources of the application program manager.
The state of each application may be one of a minimum resource pattern, a starvation pattern, a demand resource pattern, and a maximum resource pattern.
The minimum resource mode is that the application usable resource is equal to the application manager minimum resource; starvation mode is that the available resources of the application are less than the application manager minimum resources; the demand resource mode is that the resource demanded by the application program is larger than the minimum resource of the application program manager and smaller than the maximum resource of the application program manager; the maximum resource mode is that the resource required by the application is equal to the application manager maximum resource.
When the state of at least one application program in the plurality of application programs with resource requirements is in a starvation mode, resources are preferentially allocated to the at least one application program with resource requirements according to resource configuration data of the application programs without resource requirements and the application programs in the starvation mode in all the application programs in the same queue.
And when the states of at least two application programs in the plurality of application programs are in a starvation mode, allocating resources for the at least two application programs according to the resource configuration data and the preset priority of the at least two application programs.
The user can preset priority for own application program, the priority can be 1-7, and the priority is higher if the number is larger. And when the states of at least two application programs in the plurality of application programs are in a starvation mode, allocating resources for the at least two application programs according to the resource configuration data and the preset priority of the at least two application programs.
And determining the used resource proportion of each queue in the at least two queues according to the resource configuration data of the at least two queues.
In one example, allocating resources for at least two applications according to resource configuration data and a preset priority of the at least two applications comprises the steps of:
1. the resource required by the at least two applications needs to be acquired, the used resource ratio = used resource/max { min { application manager minimum resource, required resource },1}, and the resource scheduler preferentially allocates the resource to the queue with the smaller used resource ratio.
2. And when the used resource proportions of the at least two application programs are equal, acquiring the resource usage amount of each of the at least two application programs.
The resource usage is a size of a used resource of each of the at least two applications.
3. And calculating the ratio of the resource usage to the preset priority to obtain the weight of at least two application programs.
4. And sequentially allocating resources for the corresponding application programs according to the sequence of the weights of the at least two application programs from small to large.
The resource scheduler may invoke application manager minimum resources for resource-demanding applications for heavily weighted applications.
The method provided by the embodiment of the invention can determine the states of the plurality of application programs according to the resource demand information and the resource configuration data of the plurality of application programs, and preferentially allocate resources to at least one application program according to the resource configuration data of the at least one application program when the state of the at least one application program is in the starvation mode. The resource scheduler can call the minimum resource of the application manager of the application without resource demand for the application with large weight value, and dynamically adjust the distribution of the cluster resources according to the priority of the application, thereby improving the utilization rate of the cluster resources.
In an example, a method for dynamically allocating cluster resources according to an embodiment of the present invention is shown in fig. 4. The method comprises the following steps:
the resource scheduler traverses the tenant's task, which may be a queue or an application in a queue, and determines whether the task has a resource requirement 410.
And traversing the tasks of the tenants by the resource scheduler according to a preset period.
If only one task has a resource requirement, step 411 is executed to preferentially allocate the resource for the task.
If there are at least two tasks that have resource requirements, step 420 is performed.
At 420, it is determined whether both tasks are starving.
If only one task is in starvation mode, step 421 is executed to preferentially allocate resources for the task.
If at least two tasks are in starvation mode, step 430 is performed.
430, the used resource ratios of the two tasks are compared.
If the used resource proportion of each task in the at least two tasks is not equal, step 431 is executed, and the resources are sequentially allocated to the at least two tasks according to the sequence from small to large of the used resource proportion.
If the used resource ratio of each of the at least two tasks is equal, step S440 is executed.
The weights of at least two tasks are compared 440.
If the weights of the at least two tasks are not equal, step 441 is executed, and resources are sequentially allocated to the at least two tasks according to the order of the weights from small to large.
If the weights of at least two tasks are equal, step 450 is executed.
And 450, judging the creation time or the starting time of at least two tasks.
If the creation time or the start time of the at least two tasks are different, step 451 is executed, and resources are sequentially allocated to the at least two tasks according to the order of the creation time or the start time from morning to evening.
If the creation time or the start time of at least two tasks are the same, step 460 is executed.
The lexicographic ordering of the names of the at least two tasks is compared 460.
And comparing the dictionary ordering of the names of the at least two tasks, wherein each task has a unique name, and allocating resources to the at least two tasks in sequence from the front to the back according to the names.
The resource scheduler can call a queue without resource requirements or a queue minimum resource of an application program or an application program manager minimum resource for the task which is ranked at the top, and the task which is ranked at the top can be rapidly processed.
The method provided by the embodiment of the invention can determine the states of a plurality of queues or applications according to the resource demand information and the resource configuration data of the queues or applications, and preferentially allocate resources to at least one queue or application according to the resource configuration data of the at least one queue or application when the state of the at least one queue or application is in the starvation mode. The allocation of the cluster resources can be dynamically adjusted according to the priority of the queue or the application program, so that the utilization rate of the cluster resources is improved.
The embodiment of the present invention further provides a device 500 for dynamically allocating cluster resources, which includes an obtaining module 510, a determining module 520, and an allocating module 530. Wherein:
an obtaining module 510, configured to obtain resource requirement information of multiple queues and the resource configuration data, where the resource configuration data includes a queue minimum resource.
The obtaining module 510 may determine a plurality of queues with resource requirements by traversing all the queues, checking whether each queue has a resource requirement, and obtaining resource requirement information and resource configuration data of the plurality of queues with resource requirements by the resource scheduler.
A determining module 520, configured to determine states of the queues according to the resource demand information and the resource configuration data.
The determining module 520 may determine the states of the queues by checking the actual usage of the multi-tenant resources according to the queues corresponding to the resource demand information and the resource configuration data of the queues.
The status of each queue may be one of a minimum resource pattern, a starvation pattern, a demand resource pattern, a maximum resource pattern
An allocating module 530, configured to preferentially allocate resources to at least one queue according to the resource configuration data when a status of the at least one queue in the plurality of queues is a starvation mode, where the starvation mode is that available resources of the queue are smaller than the minimum resource of the queue.
The queues are in one-to-one correspondence with tenants, and the resource configuration data is configured for the queue corresponding to each tenant according to the requirement information of each tenant when the queue is created for each tenant.
The dynamic allocation device for cluster resources provided in the embodiments of the present invention determines the states of the plurality of queues according to the resource demand information and the resource configuration data of the plurality of queues, and preferentially allocates resources to at least one queue according to the resource configuration data of the at least one queue when the state of the at least one queue is in the starvation mode. The allocation of the cluster resources can be dynamically adjusted according to the priority of the queue, so that the utilization rate of the cluster resources is improved.
In an embodiment of the present invention, the allocating module 530 is configured to allocate resources to at least two queues according to the resource configuration data and the preset priorities of the at least two queues when the status of the at least two queues is in the starvation mode.
The device provided by the embodiment of the invention can allocate resources for at least two queues according to the priorities of the at least two queues when the states of the at least two queues are in the starvation mode.
In an embodiment of the present invention, when the status of one queue in the plurality of queues is starvation mode, resources are preferentially allocated to the queue with resource demand according to resource configuration data of queues without resource demand and the queue with starvation mode.
The allocation module 530 may invoke a queue minimum resource of the resource demand free queue for one queue with resource demand.
The device provided by the embodiment of the invention can preferentially allocate resources to one queue when the state of the queue is in the starvation mode.
In an embodiment of the present invention, when the states of at least two of the plurality of queues are in the starvation mode, the preferentially allocating resources to the at least one queue according to the resource configuration data includes:
and allocating resources for the at least two queues according to the resource configuration data and the preset priority of the at least two queues.
The user can preset priority for the queue, the preset priority can be 1-7, and the higher the number is, the higher the priority is. And when the states of at least two queues in the plurality of queues are in a starvation mode, allocating resources for the at least two queues according to the resource configuration data and the preset priorities of the at least two queues.
The device provided by the embodiment of the invention can allocate resources for at least two queues according to the priorities of the at least two queues when the states of the at least two queues are in the starvation mode. In one embodiment of the present invention, the allocation module 530 is further configured to calculate the available resources according to the resource configuration data.
In an embodiment of the present invention, the allocating module 530 is configured to determine a used resource proportion of each queue of the at least two queues according to the resource configuration data of the at least two queues; the queue management module is further configured to obtain a resource usage amount of each queue of the at least two queues when the used resource proportion of each queue of the at least two queues is equal; the system is also used for calculating the ratio of the resource usage amount to the preset priority to obtain at least two weights; and the resource allocation module is further used for sequentially allocating resources to the corresponding queues according to the sequence of the at least two weights from small to large.
The device provided by the embodiment of the invention can determine the sequence of the distributed resources of the at least two queues according to the resource configuration data and the preset priority of the at least two queues when the at least two queues are in the starvation mode, thereby improving the utilization rate of the cluster resources.
The dynamic allocation apparatus for cluster resources provided in the embodiment of the present invention executes each step in the method shown in fig. 2, and can achieve the technical effect of dynamically adjusting allocation of cluster resources according to the priority of the queue, thereby improving the utilization rate of the cluster resources, which is not described in detail herein for brevity.
Fig. 6 is a schematic diagram illustrating a hardware structure of an electronic device according to an embodiment of the present invention.
The electronic device may comprise a processor 601 and a memory 602 in which computer program instructions are stored.
Specifically, the processor 601 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured as one or more Integrated circuits implementing embodiments of the present invention.
The processor 601 may implement any one of the methods for dynamically allocating cluster resources in the embodiments shown in fig. 2 to 4 by reading and executing computer program instructions stored in the memory 602.
In one example, the electronic device may also include a communication interface 603 and a bus 610. As shown in fig. 6, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 to complete communication therebetween.
The communication interface 603 is mainly used for implementing communication between modules, apparatuses, units and/or devices in the embodiments of the present invention.
The bus 610 includes hardware, software, or both to couple the components of the electronic device to one another. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 610 may include one or more buses, where appropriate. Although specific buses have been described and shown in the embodiments of the invention, any suitable buses or interconnects are contemplated by the invention.
The electronic device may execute the dynamic allocation method for cluster resources in the embodiment of the present invention, so as to implement the dynamic allocation method for cluster resources described in conjunction with fig. 2.
In addition, in combination with the dynamic allocation method of cluster resources in the foregoing embodiments, embodiments of the present invention may provide a computer storage medium to implement. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement a method for dynamic allocation of cluster resources as in any of the above embodiments.
It is to be understood that the invention is not limited to the specific arrangements and instrumentality described above and shown in the drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present invention.
The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.
It should also be noted that the exemplary embodiments mentioned in this patent describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.
As described above, only the specific embodiments of the present invention are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.
Claims (6)
1. A method for dynamic allocation of cluster resources, the method comprising:
acquiring resource demand information and resource configuration data of a plurality of queues, wherein the resource configuration data comprises minimum queue resources;
determining the states of the queues according to the resource demand information and the resource configuration data;
when the state of at least one queue in the plurality of queues is a starvation mode, preferentially allocating resources to the at least one queue according to the resource configuration data, wherein the starvation mode is that the available resources of the queue are smaller than the minimum resources of the queue;
when the status of at least two of the plurality of queues is starvation mode, the preferentially allocating resources to the at least one queue according to the resource configuration data includes:
determining the used resource proportion of each queue in the at least two queues according to the resource configuration data of the at least two queues;
when the used resource proportion of each queue in the at least two queues is equal, acquiring the resource usage amount of each queue in the at least two queues;
calculating the ratio of the resource usage amount to a preset priority to obtain the weights of the at least two queues;
sequentially distributing resources for the corresponding queues according to the sequence of the weights of the at least two queues from small to large;
determining a used resource proportion of each queue of the at least two queues according to the resource configuration data of the at least two queues, including:
acquiring resources required by each queue in the at least two queues, wherein the used resource ratio = used resource/max { min { minimum queue resource, required resource },1 };
the queues are in one-to-one correspondence with the tenants, and when the queues are created for each tenant, the resource configuration data is configured for the queues corresponding to each tenant according to the demand information of each tenant.
2. The method of claim 1, wherein before preferentially allocating resources to at least one queue of the plurality of queues according to the resource configuration data when the status of the at least one queue is starved mode, the method further comprises:
calculating the usable resource according to the resource configuration data.
3. An apparatus for dynamic allocation of cluster resources, the apparatus comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring resource demand information and resource configuration data of a plurality of queues, and the resource configuration data comprises minimum queue resources;
a determining module, configured to determine states of the queues according to the resource demand information and the resource configuration data;
an allocation module, configured to preferentially allocate resources to at least one queue according to the resource configuration data when a state of the at least one queue in the plurality of queues is a starvation mode, where the starvation mode is that available resources of the queue are smaller than the minimum resource of the queue;
the queues correspond to the tenants one by one, and when the queues are created for each tenant, the resource configuration data is configured for the queues corresponding to each tenant according to the demand information of each tenant;
when the states of at least two of the plurality of queues are in the starvation mode, the allocating module is specifically configured to:
determining the used resource proportion of each queue in the at least two queues according to the resource configuration data of the at least two queues;
when the used resource proportion of each queue in the at least two queues is equal, acquiring the resource usage amount of each queue in the at least two queues;
calculating the ratio of the resource usage amount to a preset priority to obtain at least two weights;
sequentially distributing resources for the corresponding queues according to the sequence of the at least two weights from small to large;
the allocation module is further specifically configured to:
obtaining the resource required by each queue of the at least two queues, the used resource ratio = used resource/max { min { minimum resource of queue, required resource },1 }.
4. The apparatus of claim 3, wherein the allocation module is further configured to calculate the available resources according to the resource configuration data.
5. An electronic device, characterized in that the device comprises: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements a method for dynamic allocation of cluster resources as recited in any of claims 1-2.
6. A computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method for dynamic allocation of cluster resources of any of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110747732.7A CN113204433B (en) | 2021-07-02 | 2021-07-02 | Dynamic allocation method, device, equipment and storage medium for cluster resources |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110747732.7A CN113204433B (en) | 2021-07-02 | 2021-07-02 | Dynamic allocation method, device, equipment and storage medium for cluster resources |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113204433A CN113204433A (en) | 2021-08-03 |
| CN113204433B true CN113204433B (en) | 2021-10-22 |
Family
ID=77022688
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110747732.7A Active CN113204433B (en) | 2021-07-02 | 2021-07-02 | Dynamic allocation method, device, equipment and storage medium for cluster resources |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113204433B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113986497B (en) * | 2021-10-27 | 2022-11-22 | 北京百度网讯科技有限公司 | Queue scheduling method, device and system based on multi-tenant technology |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104991830A (en) * | 2015-07-10 | 2015-10-21 | 山东大学 | YARN resource allocation and energy-saving scheduling method and system based on service level agreement |
| CN106201681A (en) * | 2016-06-30 | 2016-12-07 | 湖南大学 | Task scheduling algorithm based on pre-release the Resources list under Hadoop platform |
| CN106445675A (en) * | 2016-10-20 | 2017-02-22 | 焦点科技股份有限公司 | B2B platform distributed application scheduling and resource allocation method |
| CN111679900A (en) * | 2020-06-15 | 2020-09-18 | 杭州海康威视数字技术股份有限公司 | Task processing method and device |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105718316A (en) * | 2014-12-01 | 2016-06-29 | 中国移动通信集团公司 | Job scheduling method and apparatus |
| US10412024B2 (en) * | 2016-06-08 | 2019-09-10 | Accenture Global Solutions Limited | Resource evaluation for complex task execution |
| US11010193B2 (en) * | 2017-04-17 | 2021-05-18 | Microsoft Technology Licensing, Llc | Efficient queue management for cluster scheduling |
| US11144325B2 (en) * | 2017-12-15 | 2021-10-12 | Jpmorgan Chase Bank, N.A. | Systems and methods for optimized cluster resource utilization |
| CN110609742B (en) * | 2019-09-25 | 2023-01-06 | 苏州浪潮智能科技有限公司 | Method and device for configuring queues of Kubernetes scheduler |
-
2021
- 2021-07-02 CN CN202110747732.7A patent/CN113204433B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104991830A (en) * | 2015-07-10 | 2015-10-21 | 山东大学 | YARN resource allocation and energy-saving scheduling method and system based on service level agreement |
| CN106201681A (en) * | 2016-06-30 | 2016-12-07 | 湖南大学 | Task scheduling algorithm based on pre-release the Resources list under Hadoop platform |
| CN106445675A (en) * | 2016-10-20 | 2017-02-22 | 焦点科技股份有限公司 | B2B platform distributed application scheduling and resource allocation method |
| CN111679900A (en) * | 2020-06-15 | 2020-09-18 | 杭州海康威视数字技术股份有限公司 | Task processing method and device |
Non-Patent Citations (4)
| Title |
|---|
| "Hadoop YARN资源分配与调度的研究";李媛祯;《中国优秀硕士学位论文全文数据库 信息科技辑》;20160715(第07期);第I138-697页 * |
| "Job Scheduling Optimization for Multi-user MapReduce Clusters";Yongcai Tao等;《2011 Fourth International Symposium on Parallel Architectures, Algorithms and Programming》;20120112;第213-217页 * |
| "基于Hadoop平台的作业调度算法研究与改进";安思华;《中国优秀硕士学位论文全文数据库 信息科技辑》;20160315;第38-39页 * |
| "基于Yarn架构的动态资源感知调度算法的设计与实现";王琪;《中国优秀硕士学位论文全文数据库 信息科技辑》;20180315(第03期);第I138-1168页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113204433A (en) | 2021-08-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10772115B2 (en) | Resource scheduling method and server | |
| CN108965485B (en) | Container resource management method and device and cloud platform | |
| CN109992403B (en) | Optimization method and device for multi-tenant resource scheduling, terminal equipment and storage medium | |
| CN112363813A (en) | Resource scheduling method and device, electronic equipment and computer readable medium | |
| US10117103B2 (en) | Method for sharing network and network element | |
| CN109783225B (en) | Tenant priority management method and system of multi-tenant big data platform | |
| CN112996116B (en) | Resource allocation method and system for guaranteeing quality of power time delay sensitive service | |
| CN113938435B (en) | Data transmission method, device, electronic equipment, storage medium and program product | |
| CN110971623A (en) | Micro-service instance elastic scaling method and device and storage medium | |
| CN113204433B (en) | Dynamic allocation method, device, equipment and storage medium for cluster resources | |
| CN103595654A (en) | HQoS implementation method, device and network equipment based on multi-core CPUs | |
| CN112214288B (en) | Pod scheduling method, device, equipment and medium based on Kubernetes cluster | |
| WO2020108337A1 (en) | Cpu resource scheduling method and electronic equipment | |
| CN114138428A (en) | SLO (Simultaneous task oriented) guaranteeing method, device, node and storage medium for multi-priority tasks | |
| CN113051064A (en) | Task scheduling method, device, equipment and storage medium | |
| CN115766582A (en) | Flow control method, device and system, medium and computer equipment | |
| CN115562841A (en) | Cloud video service self-adaptive resource scheduling system and method | |
| CN114489970A (en) | Method and system for realizing queue sequencing by using scheduling plug-in Kubernetes | |
| CN114138427A (en) | SLO guarantee method, SLO guarantee device, node, and storage medium | |
| CN117056064A (en) | Resource allocation method, device, server, storage medium and program product | |
| CN113672347A (en) | Container group scheduling method and device | |
| CN112631766B (en) | Dynamic adjustment method and device for project environment resources | |
| CN114020433A (en) | Instance resource scheduling method, device, storage medium and device | |
| CN111597034B (en) | Processor resource scheduling method and device, terminal equipment and computer storage medium | |
| CN114077493A (en) | Resource allocation method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211213 Address after: 200233 4th floor, South B3, Putian Industrial Park, No. 700, Yishan Road, Xuhui District, Shanghai Patentee after: Shanghai Youkun Information Technology Co.,Ltd. Address before: 201201 room 81001, 1st floor, building 6, No. 633, Shangfeng Road, Pudong New Area, Shanghai Patentee before: Shanghai SMK Network Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right |