CN113535398B - Resource allocation adjustment method, device, electronic equipment and readable storage medium - Google Patents
Resource allocation adjustment method, device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN113535398B CN113535398B CN202110796301.XA CN202110796301A CN113535398B CN 113535398 B CN113535398 B CN 113535398B CN 202110796301 A CN202110796301 A CN 202110796301A CN 113535398 B CN113535398 B CN 113535398B
- Authority
- CN
- China
- Prior art keywords
- resource
- amount
- adjustment
- proxy node
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013468 resource allocation Methods 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000007726 management method Methods 0.000 claims description 21
- 238000004891 communication Methods 0.000 claims description 7
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 230000000875 corresponding effect Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 101100264195 Caenorhabditis elegans app-1 gene Proteins 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Abstract
The application provides a resource allocation adjustment method, a device, an electronic device and a readable storage medium, wherein a first resource adjustment amount and a second resource adjustment amount are determined according to resource application amounts and resource actual use amounts by obtaining the resource application amounts and the resource actual use amounts of each task instance executed by each proxy node in a system, resource allocation adjustment is performed for the proxy node based on the first resource adjustment amount, and resource allocation adjustment is performed for each task instance executed by the proxy node based on the second resource adjustment amount. In the scheme, the resources of the whole agent node and the task instance inside the agent node are respectively adjusted, so that reasonable resource allocation among the agent nodes and reasonable resource allocation among the task instances inside the agent nodes are realized, the resource adjustment amount is determined by integrating the resource application amount and the resource actual use amount, the problem that the application of the resources is inconsistent with the actual use of the resources is solved, and the resource utilization rate can be improved.
Description
Technical Field
The present invention relates to the field of distributed system resource management technologies, and in particular, to a method and apparatus for adjusting resource allocation, an electronic device, and a readable storage medium.
Background
Hadoop is a distributed system architecture designed to fit a distributed file system running on general purpose hardware. Hadoop can provide high throughput data access, and is very suitable for application on a large-scale data set. A typical Hadoop cluster is composed of proxy node heterogeneous, with system task instances being distributed to each proxy node for execution. The configuration of resources such as CPU, memory, etc. on each proxy node will typically vary.
Currently, for resource allocation of proxy nodes and resource allocation of task instances, it is common for users to specify the number of task instances and individual task instance resources required by communicating with a resource manager. The resource manager distributes the task instances to the proxy nodes for execution according to the number of task instances specified by the user and the single task instance resources. An administrator may increase or decrease the total amount of resources of a proxy node by communicating with a resource manager, as shown in fig. 1. The adjustment mode in the prior art is limited to manual intervention adjustment, and only takes the whole resource of the proxy node as an adjustment object, and does not reasonably adjust the task instance resource in the proxy node. In addition, in the actual situation, there may be a case where the actual use resource of the task instance does not coincide with the application resource (display resource), and in the existing adjustment manner, there may be a phenomenon that the use rate of the task instance resource is very low.
Disclosure of Invention
The invention aims at providing a resource allocation adjustment method, a device, electronic equipment and a readable storage medium, which can realize resource adjustment of the whole and internal tasks of a proxy node and improve the utilization rate of resources.
Embodiments of the invention may be implemented as follows:
in a first aspect, the present invention provides a resource allocation adjustment method applied to a management node in a distributed system, where the distributed system further includes a plurality of proxy nodes, each of the proxy nodes executing at least one task instance, the method including:
for each proxy node, obtaining the resource application amount and the actual resource usage amount of each task instance executed by the proxy node;
determining a first resource adjustment amount and a second resource adjustment amount according to the resource application amount and the actual resource usage amount;
and performing resource allocation adjustment for the proxy node based on the first resource adjustment amount, and performing resource allocation adjustment for each task instance executed by the proxy node based on the second resource adjustment amount.
In an alternative embodiment, the step of determining the first resource adjustment amount according to the resource application amount and the actual resource usage amount includes:
Acquiring the display occupied resource quantity and the actual occupied resource quantity of the proxy node according to the resource application quantity and the actual resource usage quantity of all task instances executed by the proxy node;
obtaining the operation busyness of the proxy node;
and determining a first resource adjustment amount according to the display occupied resource amount, the actual occupied resource amount and the operation busyness.
In an optional embodiment, the step of determining the first resource adjustment amount according to the display occupied resource amount, the actual occupied resource amount and the operation busyness includes:
when the display occupied resource amount is larger than the actual occupied resource amount and the operation busyness is lower than a preset value, calculating to obtain a difference value between the display occupied resource amount and the actual occupied resource amount;
a first resource adjustment amount is determined based on the difference.
In an alternative embodiment, the step of determining the first resource adjustment amount based on the difference value includes:
obtaining the total resource quantity of the proxy node;
and determining a first resource adjustment amount according to the difference value and the total resource amount, so that corresponding residual available resources after the first resource adjustment amount is added to the total resource amount of the proxy node meet the set requirement, wherein the residual available resources are equal to the updated total resource amount minus the display occupied resource amount.
In an alternative embodiment, the step of determining the second resource adjustment amount according to the resource application amount and the actual resource usage amount includes:
for each task instance executed by the proxy node, obtaining a current adjustable rate carried by the task instance when submitted;
and when the relation among the resource application amount, the resource actual use amount and the current adjustable rate of the task instance meets a preset relation, determining a second resource adjustment amount according to the current adjustable rate.
In an alternative embodiment, the method further comprises the step of obtaining a current adjustable rate of submitted task instances, the step comprising:
when the currently submitted task instance is not submitted for the first time, obtaining historical resource information of the task instance after historical submission and execution;
calculating to obtain the historical adjustable rate of the task instance according to the historical resource information;
a current adjustable rate of the task instance is obtained based on the historical adjustable rate of the task instance.
In an alternative embodiment, the step of obtaining the current adjustable rate of the task instance based on the historical adjustable rate of the task instance includes:
Acquiring the historical scanning data volume carried by the task instance during historical submission and the current scanning data volume carried by the task instance during current submission;
calculating according to the historical scanning data quantity and the current scanning data quantity to obtain a scanning data quantity difference rate;
a current adjustable rate of the task instance is obtained based on the scan data volume difference rate and the historical adjustable rate.
In an alternative embodiment, each of the proxy nodes has a plurality of computing resources;
the step of obtaining the resource application amount and the actual resource usage amount of each task instance executed by the proxy node includes:
acquiring resource application amounts of the task instances executed by the proxy node to the computing resources;
each time interval is preset, the actual usage amount of the resources of each computing resource by each task instance executed by the agent node is collected, and the time stamp during collection is recorded;
and packaging the recorded time stamp, the actual resource usage amount of each computing resource and the resource application amount according to a preset format, and then storing the packaged time stamp, the actual resource usage amount and the resource application amount of each computing resource.
In a second aspect, the present invention provides a resource allocation adjustment device applied to a management node in a distributed system, where the distributed system further includes a plurality of proxy nodes, each of the proxy nodes executing at least one task instance, the device includes:
The obtaining module is used for obtaining the resource application amount and the actual resource usage amount of each task instance executed by the proxy node for each proxy node;
the determining module is used for determining a first resource adjustment amount and a second resource adjustment amount according to the resource application amount and the actual resource usage amount;
and the allocation adjustment module is used for carrying out resource allocation adjustment on the basis of the first resource adjustment quantity for the proxy node and carrying out resource allocation adjustment on the basis of the second resource adjustment quantity for each task instance executed by the proxy node.
In a third aspect, the present invention provides an electronic device comprising one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing machine-executable instructions that are executable by the processor to perform the method steps recited in any one of the preceding embodiments when the electronic device is operated.
In a fourth aspect, the present invention provides a computer-readable storage medium storing machine-executable instructions which, when executed, implement the method steps of any of the preceding embodiments.
The beneficial effects of the embodiment of the invention include, for example:
according to the resource allocation adjustment method, the device, the electronic equipment and the readable storage medium, the resource application amount and the resource actual use amount of each task instance executed by each proxy node in the system are obtained, the first resource adjustment amount and the second resource adjustment amount are determined according to the resource application amount and the resource actual use amount, the resource allocation adjustment is carried out for the proxy node based on the first resource adjustment amount, and the resource allocation adjustment is carried out for each task instance executed by the proxy node based on the second resource adjustment amount. In the scheme, the resources of the whole agent node and the task instance inside the agent node are respectively adjusted, so that reasonable resource allocation among the agent nodes and reasonable resource allocation among the task instances inside the agent nodes are realized, the resource adjustment amount is determined by integrating the resource application amount and the resource actual use amount, the problem that the application of the resources is inconsistent with the actual use of the resources is solved, and the resource utilization rate can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a block diagram of a conventional resource management system;
fig. 2 is an application scenario schematic diagram of a resource allocation adjustment method provided in an embodiment of the present application;
fig. 3 is a flowchart of a method for adjusting resource allocation according to an embodiment of the present application;
FIG. 4 is a flowchart of sub-steps included in step S120 of FIG. 3;
FIG. 5 is another flowchart of sub-steps included in step S120 of FIG. 3;
fig. 6 is a flowchart of a current adjustable rate obtaining method in the resource allocation adjustment method according to the embodiment of the present application;
FIG. 7 is a flowchart of sub-steps included in step S230 of FIG. 6;
FIG. 8 is a block diagram of a resource management system according to an embodiment of the present disclosure;
fig. 9 is a block diagram of an electronic device according to an embodiment of the present application;
fig. 10 is a functional block diagram of a resource allocation adjustment device according to an embodiment of the present application.
Icon: 110-a storage medium; a 120-processor; 130-resource allocation adjustment means; 131-obtaining a module; 132-a determination module; 133-an allocation adjustment module; 140-communication interface.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
In the description of the present invention, it should be noted that, if the terms "first," "second," and the like are used merely to distinguish the descriptions, they are not to be construed as indicating or implying relative importance.
It should be noted that the features of the embodiments of the present invention may be combined with each other without conflict.
Fig. 2 is a schematic application scenario diagram of a resource allocation adjustment method according to an embodiment of the present application. In this embodiment, the application scenario of the allocation adjustment method may be a resource management system of a Hadoop cluster, such as Hadoop Yarn. The application scenario includes a management node and a plurality of proxy nodes in communication with the management node. The management node may be a cluster of multiple devices in the system, for example, the management node includes a Resource collector, a machine Resource decision maker, a task Resource collector and a task submitter in addition to a Resource Manager in a conventional Hadoop cluster.
It should be noted that, in this embodiment, the resource manager, the resource collector, the machine resource decision maker, the task resource collector, and the task submitter may be disposed on the same device, for example, disposed in the same server, and serve as a plurality of components that may be communicated in the server to implement resource allocation adjustment.
Each agent Node Manager is a machine for actually executing tasks in the system, and the total amount of resources of the cluster is equal to the sum of the total amounts of resources of all agent nodes. In the system, the content is an abstraction of computer computing resources by Yarn, which is essentially a set of CPU and memory resources, and all applications run in the content. Application Master is an abstraction of a certain application running in Yarn, which is in fact an instance of a certain type of application. Application Master is application-level, its main function being to apply for computing resources (Containers) to the resource manager and interact with proxy nodes to perform and monitor specific tasks. The Scheduler is a component specially used for resource management by the resource manager and is responsible for distributing the content resources on the proxy node, and the proxy node can also continuously send the own resource use condition to the resource manager.
In this embodiment, a resource collector, a machine resource decision maker, a task resource collector and a task presenter are added based on the original architecture of the system. Each proxy node is respectively configured with a corresponding resource collector, and the resource collectors are used for collecting resource information of the corresponding proxy nodes and sending the resource information to a machine resource decision maker. And the machine resource decision device counts the resource condition of the corresponding proxy node according to the resource information acquired by the resource collector, and externally provides the resource statistics condition of the proxy node. The task resource collector can collect resource information of the completed task instance through communication with the resource manager, including such as operation time length, actual use resource amount, application resource amount and the like.
And the task submitter can be responsible for submitting specific tasks which the user needs to run and providing relevant adjustment information of the tasks for assisting in subsequent task resource adjustment.
With reference to fig. 3, the embodiment of the application further provides a resource allocation adjustment method applicable to the electronic device, which is used for adjusting the proxy node resources and the task instance resources in the proxy node. The electronic device may be the management node, and the resource allocation adjustment is implemented by a plurality of devices included in the management node. The method steps defined by the flow related to the resource allocation adjustment method may be implemented by the electronic device. The specific flow shown in fig. 3 will be described in detail.
Step S110, for each proxy node, obtaining a resource application amount and a resource actual usage amount of each task instance executed by the proxy node.
And step S120, determining a first resource adjustment amount and a second resource adjustment amount according to the resource application amount and the actual resource usage amount.
Step S130, performing resource allocation adjustment for the proxy node based on the first resource adjustment amount, and performing resource allocation adjustment for each task instance executed by the proxy node based on the second resource adjustment amount.
In this embodiment, the resource management system includes a plurality of proxy nodes, each proxy node is a machine that actually performs a specific task in the system, and the plurality of proxy nodes form a cluster. Task submissions may be submitted by a task submitter in communication with a resource manager. The resource manager is responsible for all proxy nodes and may decide to which proxy node to distribute submitted tasks for execution.
In this embodiment, the task submitted by the task presenter may be a task repeatedly executed in the system for a long time, that is, a task that has been executed historically. Or may be a task submitted by the user, for example, a task for which no history execution information is acquired. Because the tasks executed in the system are often repeatedly executed for a long time, in this embodiment, the resource allocation adjustment is mainly performed for the tasks repeatedly executed in the system.
The resources allocated by each proxy node to perform tasks include a variety of computing resources including, for example, CPU, memory, etc. When the resource application amount and the resource actual use amount of each task instance executed by the proxy node are obtained, the resource application amount of each task instance executed by the proxy node to each computing resource can be obtained, the resource actual use amount of each task instance executed by the proxy node to each computing resource is acquired every preset time, the time stamp during acquisition is recorded, and then the recorded time stamp, the resource actual use amount and the resource application amount of each computing resource are packaged according to a preset format and then stored.
Each proxy node is provided with a resource collector, and the resource information of the proxy node is collected by the corresponding resource collector. The resource collector sends the collected resource information to the machine resource decision maker. The resource application amount of each task instance is the amount applied when the task is submitted, and the actual use amount of the resource is the amount used by the task instance which is actually collected by the resource collector every preset time period.
The plurality of proxy nodes contained in the resource management system form a cluster, and the total resource amount of the cluster is equal to the sum of the total resource amounts of all proxy nodes. The resource application amount and the actual resource usage amount of each task instance executed by the proxy node can determine the overall resource condition of the proxy node. Thus, based on the resource application amounts and the actual resource usage amounts of all task instances executed on the proxy node, a first resource adjustment amount for making resource allocation adjustments to the proxy node may be determined.
The first resource adjustment amount includes adjustment amounts for various computing resources, for example, adjustment amounts for cpu resources, adjustment amounts for memory resources, and the like. The adjustment is performed with the agent node as a whole, i.e., the resource adjustment is performed only to the agent node regardless of the specific situation of the task instance performed by the agent node.
In addition, for each task instance executed by the proxy node, based on the resource application amount and the actual usage amount of each task instance, a second resource adjustment amount may be obtained, where the second resource adjustment amount may be used to perform resource allocation adjustment on the corresponding task instance. Likewise, the second resource adjustment amount includes adjustment amounts for various computing resources for each task instance, i.e., reducing computing resources or increasing computing resources for a task instance.
In the embodiment, the resources of the whole proxy node and the task instance inside the proxy node are respectively adjusted, so that reasonable resource allocation among the proxy nodes and reasonable resource allocation among the task instances inside the proxy nodes are realized. And the resource adjustment amount is determined by integrating the resource application amount and the actual resource use amount, so that the problem that the application of the resource is inconsistent with the actual resource use is solved, and the resource use rate can be improved.
In this embodiment, when the overall resource of the proxy node is adjusted, the adjustment amount may be determined from the resource usage of the proxy node and the operation of the proxy node. Optionally, referring to fig. 4, the step of determining the first resource adjustment amount may include the following process:
step S121, obtaining the display occupied resource amount and the actual occupied resource amount of the proxy node according to the resource application amount and the actual resource usage amount of all task instances executed by the proxy node.
Step S122, obtaining the operation busyness of the proxy node.
Step S123, determining a first resource adjustment amount according to the display occupied resource amount, the actual occupied resource amount and the operation busyness.
From the above, the sum of the resource application amounts of all task instances on the proxy node is equal to the resource application amount of the proxy node, and the sum of the actual resource usage amounts of all task instances on the proxy node is equal to the actual resource usage amount of the proxy node. For convenience of distinction, in this embodiment, the resource application amount of the proxy node is represented by the display occupied resource amount, and the actual resource usage amount is represented by the actual occupied resource amount.
In this embodiment, the resource collector may further collect, for each preset time period, a running busyness of the corresponding proxy node, where the running busyness may be represented by using a system load average value (systemload averages), and represents a requirement of a running thread (task) on the system in terms of an average running number and a waiting thread number.
In this embodiment, the proxy node is taken as a whole, the display occupied resource amount and the actual occupied resource amount of the proxy node are combined, the operation busyness is combined to determine the adjustment amount, the actual operation condition of the proxy node can be synthesized to adjust the resource, and the self condition of the proxy node is met.
In this embodiment, in one implementation manner, when the display occupied resource amount is greater than the actual occupied resource amount and the operation busyness is lower than the preset value, the difference between the display occupied resource amount and the actual occupied resource amount is calculated. In this case, the resource represented by the difference is a resource that is not actually used by the proxy node. The first resource adjustment amount may be determined based on the calculated difference. The machine resource decider may submit an application carrying the first resource adjustment amount to a resource manager, which may perform resource adjustment for the proxy node based on the first resource adjustment amount in the application.
Since tasks continue to arrive in the system, the aim of resource adjustment for the proxy node is to enable the proxy node to leave a certain amount of resources to cope with the subsequently arriving tasks. The resource collector of each proxy node can also collect the total resource amount of the proxy node at each interval for a preset time length, so that when the first resource adjustment amount is determined based on the difference value, the total resource amount of the proxy node can be obtained, the first resource adjustment amount is determined according to the difference value and the total resource amount, so that the corresponding residual available resources after the first resource adjustment amount is added to the total resource amount of the proxy node meet the set requirement, and the residual available resources are equal to the updated total resource amount minus the display occupied resource amount.
For example, when the amount of display occupied resources of the proxy node is greater than the amount of actual occupied resources and the operation busyness is less than 50, it indicates that the amount of display occupied resources is virtually high. And the proxy node remaining available resource amount=total resource amount-display occupied resource amount, in order to increase the remaining available resource amount to meet the set requirement, the total resource amount needs to be increased. Alternatively, the first resource adjustment amount may obtain a 30% difference. The machine resource decider may send the first resource adjustment amount to the resource manager, and the resource manager may add resources allocated 30% by difference to the proxy node, so that the available resources remaining after the new allocation may become larger, so as to be convenient for coping with a subsequent arriving task.
In this embodiment, in the above case, the display occupied resource amount may be reduced by, for example, 30%, and the remaining available resource amount may be increased if the total resource amount is unchanged. However, since the display occupied resource of the proxy node is the sum of the resource application amounts of all task instances executed on the proxy node, if the display occupied resource amount is directly adjusted, the resource application amounts of a plurality of task instances need to be adjusted, and such adjustment manner is complex and many factors need to be considered. Therefore, in this embodiment, the method of directly adjusting the total resource amount is more convenient.
After the total resource amount is increased, the remaining available resource amount is correspondingly increased, and the proxy node can process more task instances. After being allocated with more tasks, the display occupied resource quantity of the proxy node can be consistent with the actual occupied resource quantity as much as possible, so that the phenomenon of high resource deficiency is avoided.
In addition to the above adjustment of the resources of the entire proxy node, this embodiment may also perform resource adjustment for each task instance executed on the proxy node.
Referring to fig. 5, in one possible implementation, the second resource adjustment amount may be determined according to the resource application amount and the actual resource usage amount.
Step S124, for each task instance executed by the proxy node, obtaining a current adjustable rate carried by the task instance when submitted.
Step S125, when the relation among the resource application amount, the actual resource usage amount and the current adjustable rate of the task instance satisfies a preset relation, determining a second resource adjustment amount according to the current adjustable rate.
In this embodiment, when a task is submitted, the task presenter specifies the current adjustable rate of the task in addition to the resource application amount of the task, where the current adjustable rate has different values for different task instances. The current adjustable rate can be used as an adjustment basis for task resource adjustment to help each proxy node to adaptively adjust task instance resources.
In this embodiment, the resource collector may collect the actual usage of the resources of the task instance after the task instance runs for several minutes, for example, after 3 minutes, for each preset time period. When the relation among the resource application amount, the resource actual use amount and the current adjustable rate satisfies a preset relation, a second resource adjustment amount can be determined.
The preset relation may be, for example, that the actual usage amount of the resource is smaller than the application amount of the resource, and the actual usage amount of the resource (1—the current adjustable rate/2) is smaller than the application amount of the resource. Wherein, the second resource adjustment amount may be (current adjustable rate/2) or (current adjustable rate/3) or the like without limitation. In this case, the proxy node may apply for modifications to the resource manager to reduce the amount of resource application by the proxy node. While the reduced resources may be allocated to other tasks performed on the proxy node.
In addition, when the resource usage amount of the task instance acquired by the resource acquirer reaches the resource application amount thereof, the proxy node can apply a certain resource adjustment amount to the resource manager so as to increase the resource application amount of the task instance.
In this embodiment, the task presenter specifies the current adjustable rate of the task when presenting the task. Each task will correspond to an application type on the resource manager, identified by $ { appname_appid }. While tasks are typically repeated in the system, in many cases the tasks are not submitted for the first time, i.e. the same tasks have historically been submitted and executed.
In this embodiment, for a task that is not submitted for the first time, the current adjustable rate of the task can be obtained by the following method, please refer to fig. 6 in combination:
step S210, when the currently submitted task instance is not submitted for the first time, obtaining historical resource information of the task instance after historical submission and execution.
Step S220, calculating the historical adjustable rate of the task instance according to the historical resource information.
Step S230, obtaining the current adjustable rate of the task instance based on the historical adjustable rate of the task instance.
In this embodiment, the task resource collector may acquire and store resource information of the completed task instance in the system at intervals, so as to perform adjustable rate calculation for the next task submission. That is, the task instance may obtain its historical resource information after execution is complete. In the case that the task instance is completed, a historical adjustable rate of the completed task may be obtained based on the historical resource information, and then the current adjustable rate may be determined in conjunction with the current information.
Referring to fig. 7, as one possible implementation, the current adjustable rate may be obtained by:
Step S231, obtaining the historical scanning data volume carried by the task instance in historical submission and the current scanning data volume carried by the task instance in current submission.
And step S232, calculating the scanning data quantity difference rate according to the historical scanning data quantity and the current scanning data quantity.
Step S233, obtaining a current adjustable rate of the task instance based on the scan data volume difference rate and the historical adjustable rate.
The historical resource information of the task instance may include the historical resource application amount LA, the historical resource actual usage amount LU. In addition, the task submitter can also specify the scanning data volume required by the task when submitting the task, wherein the scanning data volume is positively correlated with the actual use amount of the resource, namely, the larger the required scanning data volume is, the larger the actual use amount of the resource is. The historical resource information of the task instance also comprises a historical scanning data quantity LD. The historical adjustable rate of the task instance may be calculated by, for example, the historical adjustable rate ly= (LA-LU)/LA.
The current scan data size CD of the task is also specified when the task instance is submitted. The scan data amount difference CDY can be obtained based on the historical scan data amount and the current scan data amount in the following manner.
CDY＝MAX{0,CD-LD/LD}
On this basis, the current adjustable rate CY can be determined based on the scan data amount difference rate and the historical adjustable rate as follows:
CY＝CDY>0.30:LY*(1-CDY)
wherein, the above formula indicates that if the scan data amount difference rate is greater than 30%, the current adjustable rate is 0, and if the scan data amount difference rate is less than or equal to 30%, the current adjustable rate is equal to the historical adjustable rate multiplied by (1-scan data amount difference rate).
It should be noted that the above specific values are only examples, and the application amount and the actual use amount of the task instance can be adjusted as required based on the above values during implementation, so as to be consistent as much as possible.
In order to provide a clearer understanding of the process of resource allocation adjustment provided in the present application, the process of resource allocation adjustment will be described with reference to fig. 8.
In terms of system architecture, the resource management system of the present embodiment includes a management node and a plurality of proxy nodes, where the management node includes a resource manager, a task submitter, a task resource collector, a resource collector, and a machine resource decider.
When the task is submitted, the task submitter can submit the task in an App form according to the requirement of a user, for example, the task submission App1 and the task submission Appn in FIG. 8. In addition, the submission of the task may also be submitted by the user communicating directly with the resource manager, such as submitting task App2 in fig. 8.
The task submitter may also specify an adjustable rate of tasks when submitting the tasks. The task resource collector can collect resource information of completed tasks, including actual usage amount, application amount, scanning data amount and the like. The adjustable rate designated by the task presenter is calculated according to the resource information of the completed task acquired by the task resource acquirer, and the calculation mode can be seen from the above.
After obtaining the submitted task, the resource manager may decide to which proxy node to assign the task for processing, such as proxy node 1, proxy node 2, and proxy node N in fig. 8. Each proxy node is deployed with resource collectors, such as resource collector 1, resource collector 2, resource collector N. In the process of executing task examples by the proxy node, the resource collector collects information such as total resource quantity, resource application quantity of each task example, actual use quantity of resources and the like of the proxy node. For example, resource information of a task App1 instance C1 on the proxy node 1, resource information of a task App2 instance C2 on the proxy node 2, resource information of a task AppN instance CN on the proxy node N, and the like are collected. And the acquired information is packaged and then sent to a machine resource decision-making device.
The machine resource decider may determine whether to adjust the resources of the proxy node or whether to adjust the resources of a task instance executed on the proxy node according to the obtained resource-related information of each proxy node. If the adjustment is needed, the first resource adjustment amount for the proxy node and the second resource adjustment amount for the task instance can be calculated, and the specific calculation method can be seen from the above.
The machine resource decision maker can send an application carrying the first resource adjustment amount and/or the second resource adjustment amount to the resource manager, and after receiving the application, the resource manager can adjust the resources of the proxy node based on the first resource adjustment amount, and can also adjust the resources of each task instance based on the second resource adjustment amount, for example, the task App1 instance C1 resource amount adjustment, the task App2 instance C2 resource amount adjustment, and the task AppN instance CN resource amount adjustment.
In this embodiment, by combining the task submitter, the task resource collector, the machine resource decision maker and the resource manager in the above manner, resource allocation adjustment of the whole agent node and each task instance executed by the agent node is realized. The resource reasonable distribution among the proxy nodes and the resource reasonable distribution among the task instances inside the proxy nodes are realized, the resource adjustment quantity is determined by integrating the resource application quantity and the resource actual use quantity, the problem that the application of the resource is inconsistent with the resource actual use is solved, and the resource utilization rate can be improved.
Referring to fig. 9, a schematic diagram of exemplary components of an electronic device according to an embodiment of the present application is provided, where the electronic device may be the management node shown in fig. 2, that is, a server configured with a resource manager, a resource collector, a machine resource decision maker, a task resource collector, and a task submitter. The electronic device may comprise a storage medium 110, a processor 120, a resource allocation adjustment means 130, a communication interface 140. In this embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and are separately disposed. However, it should be understood that the storage medium 110 may also be separate from the electronic device and accessible to the processor 120 through a bus interface. Alternatively, the storage medium 110 may be integrated into the processor 120, for example, as a cache and/or general purpose registers.
The resource allocation adjustment device 130 may be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, or may be understood as a software functional module that implements the above-mentioned resource allocation adjustment method under the control of the electronic device, independently of the above-mentioned electronic device or the processor 120.
As shown in fig. 10, the resource allocation adjustment device 130 may include an obtaining module 131, a determining module 132, and an allocation adjustment module 133, and the functions of the respective functional modules of the resource allocation adjustment device 130 are described in detail below.
The obtaining module 131 is configured to obtain, for each proxy node, a resource application amount and a resource actual usage amount of each task instance executed by the proxy node.
It will be appreciated that the obtaining module 131 may be used to perform the step S110 described above, and reference may be made to the details of the implementation of the obtaining module 131 regarding the step S110 described above.
A determining module 132, configured to determine a first resource adjustment amount and a second resource adjustment amount according to the resource application amount and the actual resource usage amount.
It will be appreciated that the determination module 132 may be used to perform step S120 described above, and reference may be made to the details of implementation of the determination module 132 as described above with respect to step S120.
And the allocation adjustment module 133 is configured to perform resource allocation adjustment for the proxy node based on the first resource adjustment amount, and perform resource allocation adjustment for each task instance executed by the proxy node based on the second resource adjustment amount.
It is understood that the allocation adjustment module 133 may be used to perform the step S130 described above, and reference may be made to the details of the implementation of the allocation adjustment module 133 with respect to the step S130 described above.
In one possible implementation, the determining module 132 may specifically be configured to determine the first resource adjustment amount by:
Acquiring the display occupied resource quantity and the actual occupied resource quantity of the proxy node according to the resource application quantity and the actual resource usage quantity of all task instances executed by the proxy node;
obtaining the operation busyness of the proxy node;
and determining a first resource adjustment amount according to the display occupied resource amount, the actual occupied resource amount and the operation busyness.
In one possible implementation, the determining module 132 may specifically be configured to:
when the display occupied resource amount is larger than the actual occupied resource amount and the operation busyness is lower than a preset value, calculating to obtain a difference value between the display occupied resource amount and the actual occupied resource amount;
a first resource adjustment amount is determined based on the difference.
In one possible implementation, the determining module 132 may be configured to:
obtaining the total resource quantity of the proxy node;
and determining a first resource adjustment amount according to the difference value and the total resource amount, so that corresponding residual available resources after the first resource adjustment amount is added to the total resource amount of the proxy node meet the set requirement, wherein the residual available resources are equal to the updated total resource amount minus the display occupied resource amount.
In one possible implementation, the determining module 132 may specifically be configured to determine the second resource adjustment amount by:
for each task instance executed by the proxy node, obtaining a current adjustable rate carried by the task instance when submitted;
and when the relation among the resource application amount, the resource actual use amount and the current adjustable rate of the task instance meets a preset relation, determining a second resource adjustment amount according to the current adjustable rate.
In one possible implementation, the obtaining module 131 may be further configured to:
when the currently submitted task instance is not submitted for the first time, obtaining historical resource information of the task instance after historical submission and execution;
calculating to obtain the historical adjustable rate of the task instance according to the historical resource information;
a current adjustable rate of the task instance is obtained based on the historical adjustable rate of the task instance.
In one possible implementation manner, the obtaining module 131 may specifically be configured to:
acquiring the historical scanning data volume carried by the task instance during historical submission and the current scanning data volume carried by the task instance during current submission;
Calculating according to the historical scanning data quantity and the current scanning data quantity to obtain a scanning data quantity difference rate;
a current adjustable rate of the task instance is obtained based on the scan data volume difference rate and the historical adjustable rate.
In one possible implementation manner, each proxy node has multiple computing resources, and the obtaining module 131 may specifically be configured to:
acquiring resource application amounts of the task instances executed by the proxy node to the computing resources;
each time interval is preset, the actual usage amount of the resources of each computing resource by each task instance executed by the agent node is collected, and the time stamp during collection is recorded;
and packaging the recorded time stamp, the actual resource usage amount of each computing resource and the resource application amount according to a preset format, and then storing the packaged time stamp, the actual resource usage amount and the resource application amount of each computing resource.
The process flow of each module in the apparatus and the interaction flow between the modules may be described with reference to the related descriptions in the above method embodiments, which are not described in detail herein.
Further, the embodiment of the present application further provides a computer readable storage medium, where machine executable instructions are stored, where the machine executable instructions when executed implement the resource allocation adjustment method provided in the foregoing embodiment.
In particular, the computer readable storage medium can be a general purpose storage medium, such as a removable disk, a hard disk, etc., and the computer program on the computer readable storage medium can execute the above-described resource allocation adjustment method when executed. With respect to the processes involved in the computer readable storage medium and when executed as executable instructions thereof, reference is made to the relevant descriptions of the method embodiments described above and will not be described in detail herein.
In summary, the method, the device, the electronic device and the readable storage medium for adjusting resource allocation provided by the present application determine a first resource adjustment amount and a second resource adjustment amount according to a resource application amount and a resource actual usage amount by obtaining the resource application amount and the resource actual usage amount of each task instance executed by each proxy node in a system, perform resource allocation adjustment for the proxy node based on the first resource adjustment amount, and perform resource allocation adjustment for each task instance executed by the proxy node based on the second resource adjustment amount. In the scheme, the resources of the whole agent node and the task instance inside the agent node are respectively adjusted, so that reasonable resource allocation among the agent nodes and reasonable resource allocation among the task instances inside the agent nodes are realized, the resource adjustment amount is determined by integrating the resource application amount and the resource actual use amount, the problem that the application of the resources is inconsistent with the actual use of the resources is solved, and the resource utilization rate can be improved.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for adjusting resource allocation, applied to a management node in a distributed system, the distributed system further comprising a plurality of proxy nodes, each proxy node executing at least one task instance, the method comprising:
for each proxy node, obtaining the resource application amount and the actual resource usage amount of each task instance executed by the proxy node;
determining a first resource adjustment amount and a second resource adjustment amount according to the resource application amount and the actual resource usage amount;
performing resource allocation adjustment for the proxy node based on the first resource adjustment amount, and performing resource allocation adjustment for each task instance executed by the proxy node based on the second resource adjustment amount;
the first resource adjustment amount is a difference value between the display occupied resource amount and the actual occupied resource amount under all task examples when the operation busyness of the proxy node is lower than a preset value, and is calculated according to the following mode:
And obtaining the total resource quantity of the proxy node, and determining a first resource adjustment quantity according to the difference value and the total resource quantity, so that corresponding residual available resources after the first resource adjustment quantity is added to the total resource quantity of the proxy node meet the set requirement, wherein the residual available resources are equal to the updated total resource quantity minus the display occupied resource quantity.
2. The resource allocation adjustment method according to claim 1, wherein the step of determining the first resource adjustment amount based on the resource application amount and the actual resource usage amount comprises:
acquiring the display occupied resource quantity and the actual occupied resource quantity of the proxy node according to the resource application quantity and the actual resource usage quantity of all task instances executed by the proxy node;
obtaining the operation busyness of the proxy node;
and determining a first resource adjustment amount according to the display occupied resource amount, the actual occupied resource amount and the operation busyness.
3. The resource allocation adjustment method according to claim 2, wherein the step of determining the first resource adjustment amount according to the display occupied resource amount, the actual occupied resource amount, and the operation busyness comprises:
When the display occupied resource amount is larger than the actual occupied resource amount and the operation busyness is lower than a preset value, calculating to obtain a difference value between the display occupied resource amount and the actual occupied resource amount;
a first resource adjustment amount is determined based on the difference.
4. The resource allocation adjustment method according to claim 1, wherein the step of determining the second resource adjustment amount based on the resource application amount and the actual resource usage amount comprises:
for each task instance executed by the proxy node, obtaining a current adjustable rate carried by the task instance when submitted;
and when the relation among the resource application amount, the resource actual use amount and the current adjustable rate of the task instance meets a preset relation, determining a second resource adjustment amount according to the current adjustable rate.
5. The resource allocation adjustment method according to claim 4, further comprising the step of obtaining a current adjustable rate of submitted task instances, the step comprising:
when the currently submitted task instance is not submitted for the first time, obtaining historical resource information of the task instance after historical submission and execution;
Calculating to obtain the historical adjustable rate of the task instance according to the historical resource information;
a current adjustable rate of the task instance is obtained based on the historical adjustable rate of the task instance.
6. The resource allocation adjustment method according to claim 5, wherein the step of obtaining the current adjustable rate of the task instance based on the historical adjustable rate of the task instance comprises:
acquiring the historical scanning data volume carried by the task instance during historical submission and the current scanning data volume carried by the task instance during current submission;
calculating according to the historical scanning data quantity and the current scanning data quantity to obtain a scanning data quantity difference rate;
a current adjustable rate of the task instance is obtained based on the scan data volume difference rate and the historical adjustable rate.
7. The resource allocation adjustment method according to any one of claims 1 to 6, wherein each of the proxy nodes has a plurality of computing resources;
the step of obtaining the resource application amount and the actual resource usage amount of each task instance executed by the proxy node includes:
acquiring resource application amounts of the task instances executed by the proxy node to the computing resources;
Each time interval is preset, the actual usage amount of the resources of each computing resource by each task instance executed by the agent node is collected, and the time stamp during collection is recorded;
and packaging the recorded time stamp, the actual resource usage amount of each computing resource and the resource application amount according to a preset format, and then storing the packaged time stamp, the actual resource usage amount and the resource application amount of each computing resource.
8. A resource allocation adjustment device for use in a management node in a distributed system, the distributed system further comprising a plurality of agent nodes, each agent node executing at least one task instance, the device comprising:
the obtaining module is used for obtaining the resource application amount and the actual resource usage amount of each task instance executed by the proxy node for each proxy node;
the determining module is used for determining a first resource adjustment amount and a second resource adjustment amount according to the resource application amount and the actual resource usage amount;
the allocation adjustment module is used for carrying out resource allocation adjustment on the basis of the first resource adjustment quantity for the proxy node and carrying out resource allocation adjustment on the basis of the second resource adjustment quantity for each task instance executed by the proxy node;
the first resource adjustment amount is a difference value between the display occupied resource amount and the actual occupied resource amount under all task examples when the operation busyness of the proxy node is lower than a preset value, and is calculated according to the following mode:
And obtaining the total resource quantity of the proxy node, and determining a first resource adjustment quantity according to the difference value and the total resource quantity, so that corresponding residual available resources after the first resource adjustment quantity is added to the total resource quantity of the proxy node meet the set requirement, wherein the residual available resources are equal to the updated total resource quantity minus the display occupied resource quantity.
9. An electronic device comprising one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing processor-executable machine-executable instructions that, when the electronic device is run, are executed by the processor to perform the method steps recited in any of claims 1-7.
10. A computer readable storage medium storing machine executable instructions which when executed implement the method steps of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110796301.XA CN113535398B (en) | 2021-07-14 | 2021-07-14 | Resource allocation adjustment method, device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110796301.XA CN113535398B (en) | 2021-07-14 | 2021-07-14 | Resource allocation adjustment method, device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113535398A CN113535398A (en) | 2021-10-22 |
| CN113535398B true CN113535398B (en) | 2024-02-27 |
Family
ID=78127978
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110796301.XA Active CN113535398B (en) | 2021-07-14 | 2021-07-14 | Resource allocation adjustment method, device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113535398B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114327918B (en) * | 2022-03-11 | 2022-06-10 | 北京百度网讯科技有限公司 | Method and device for adjusting resource amount, electronic equipment and storage medium |
| CN115037752B (en) * | 2022-04-22 | 2024-03-22 | 网易（杭州）网络有限公司 | Resource allocation method and device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932281A (en) * | 2012-10-31 | 2013-02-13 | 华为技术有限公司 | Method and equipment for dynamic allocation of resources |
| CN104158841A (en) * | 2014-07-09 | 2014-11-19 | 中电科华云信息技术有限公司 | Computing resource allocation method |
| CN107179945A (en) * | 2017-03-31 | 2017-09-19 | 北京奇艺世纪科技有限公司 | A kind of resource allocation methods and device |
| CN107908479A (en) * | 2017-12-11 | 2018-04-13 | 北京奇艺世纪科技有限公司 | A kind of node resource distribution method and device |
| CN112789599A (en) * | 2018-11-30 | 2021-05-11 | 北京比特大陆科技有限公司 | Information recommendation method, device, equipment and readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9871741B2 (en) * | 2014-03-10 | 2018-01-16 | Microsoft Technology Licensing, Llc | Resource management based on device-specific or user-specific resource usage profiles |
-
2021
- 2021-07-14 CN CN202110796301.XA patent/CN113535398B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932281A (en) * | 2012-10-31 | 2013-02-13 | 华为技术有限公司 | Method and equipment for dynamic allocation of resources |
| CN104158841A (en) * | 2014-07-09 | 2014-11-19 | 中电科华云信息技术有限公司 | Computing resource allocation method |
| CN107179945A (en) * | 2017-03-31 | 2017-09-19 | 北京奇艺世纪科技有限公司 | A kind of resource allocation methods and device |
| CN107908479A (en) * | 2017-12-11 | 2018-04-13 | 北京奇艺世纪科技有限公司 | A kind of node resource distribution method and device |
| CN112789599A (en) * | 2018-11-30 | 2021-05-11 | 北京比特大陆科技有限公司 | Information recommendation method, device, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113535398A (en) | 2021-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8510741B2 (en) | Computing the processor desires of jobs in an adaptively parallel scheduling environment | |
| CN113535398B (en) | Resource allocation adjustment method, device, electronic equipment and readable storage medium | |
| US8849891B1 (en) | Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system | |
| CN105159782A (en) | Cloud host based method and apparatus for allocating resources to orders | |
| CN112162865A (en) | Server scheduling method and device and server | |
| CN108989238A (en) | A kind of method and relevant device for distributing service bandwidth | |
| CN107851039A (en) | System and method for resource management | |
| CN111722928A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| US11201824B2 (en) | Method, electronic device and computer program product of load balancing for resource usage management | |
| CN110914805A (en) | Computing system for hierarchical task scheduling | |
| CN113032102A (en) | Resource rescheduling method, device, equipment and medium | |
| WO2020108337A1 (en) | Cpu resource scheduling method and electronic equipment | |
| CN111813543A (en) | Adaptive scheduling method, device and computer readable storage medium | |
| CN111813558B (en) | Resource scheduling method for hybrid cloud | |
| WO2024088078A1 (en) | Bandwidth adjustment method, system and device, and storage medium | |
| CN110413393B (en) | Cluster resource management method and device, computer cluster and readable storage medium | |
| US20180309686A1 (en) | Reducing rate limits of rate limiters | |
| CN112256436B (en) | Resource allocation method, device, equipment and computer storage medium | |
| CN112433838A (en) | Batch scheduling method, device, equipment and computer storage medium | |
| US20070011682A1 (en) | Affinization of transaction types | |
| CN111538575A (en) | Resource scheduling system, method, device, equipment and medium | |
| US11409570B2 (en) | Dynamic management of system computing resources | |
| CN111736981A (en) | Container resource allocation method, device, equipment and storage medium | |
| CN115391042B (en) | Resource allocation method and device, electronic equipment and storage medium | |
| Huang et al. | Hestia: A Cost-Effective Multi-dimensional Resource Utilization for Microservices Execution in the Cloud |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |