TW202338651A - Method for optimizing resource allocation based on prediction with reinforcement learning - Google Patents

Abstract

A method for optimizing resource allocation based on prediction with reinforcement learning is disclosed, comprising the steps of: (a) providing a prediction on the number of units of the resource for a workload in more than N timepoints after a 0-th timepoint to the processor; (b) calculating at least one 0-th possible operation cost (POC) based on at least one possible provisioned number (PPN) at 1-th timepoint; (c) repeating the following sub-steps for the i-th timepoint with i from 1 to N by the processor: (c1) calculating at least one i-th possible operation cost (POCi); (c2) finding out the smallest and the second small POCi; and (c3) setting PPNi used to calculate the smallest POCi as an i-th assigned number; and (d) provisioning 1 unit of the resource at the 0-th timepoint and i-th assigned number of units of the resource at the i-th timepoint for the workload by the processor.

Description

Methods to optimize resource allocation based on reinforcement learning predictions

The present invention relates to methods of optimizing configurations. In particular, the present invention relates to a method for optimizing resource allocation based on reinforcement learning predictions.

In a computer cluster, hardware resources are dynamically allocated to workloads. Because the resource requirements utilized by workloads, such as RAM modules or CPUs, change over time, many forecasting methods provide rough estimates of the future. Although the construction cost of a computer cluster is generally fixed, the computing cost including lifetime amortization and power consumption will increase with the number of hardware resources. Significant changes in hardware resources based on any forecast workload may not be economical from a total cost perspective. When there is no prediction, the adjustment of the number of hardware resources is passive, resulting in a large response delay.

Therefore, to reduce the impact of the above problems, more accurate prediction of workloads is needed. Reinforcement learning (RL) methods introduced in the past use the learning process to explore action possibilities at any future point in time and make optimization decisions to meet the goals of certain tasks or games. When RL methods are used to predict workloads, it leads to all possible paths problems. That is, if the workload has a choice of utilizing n resources at a time and needs to predict m time points in the future, the RL method must consider the resource deployment of n ^m paths based on the long-term learning journey. Clearly, the computational complexity of the decision-making process required to compute the optimization results is exponential. Calculating costs is another hassle for managers because it requires more software development manpower.

From a balance point of view, if the adjustment of the number of hardware resources is as small as possible and the computational cost of the RL method is limited, the total cost of the workload can be minimized. However, the prior art does not provide a method for this function of a computer cluster.

This paragraph excerpts and compiles certain features of the invention; other features will be disclosed in subsequent paragraphs. It is intended to cover various modifications and similar arrangements included within the spirit and scope of the claimed patent.

According to one aspect of the present invention, a method for optimizing resource allocation based on predictions of reinforcement learning is disclosed. The processor determines the number of resource units in the computer cluster to be utilized. The method includes the following steps: a) Provide the processor with a prediction of the number of resource units required for the workload exceeding N time points after the 0th time point, where a maximum of M unit sources can be provided, U _i is based on Predict the number of units required at the i-th time point, N, M, and i are all positive integers; b) The processor calculates at least one 0th possible operation cost (POC ₀ ), which is based on U ₁ to At least one possible offering quantity (PPN) at the 1st time point (PPN ₁ ) in M, where POC ₀ is derived from POC ₀ = K + RF x | PPN ₁ – K | + PPN ₁ , where RF is between The rebalancing factor between 0 and 1, and K is a real number; c) The processor repeats the following sub-steps in sequence for the i-th time point, i is from 1 to N: c1) Calculate at least one i-th possible The operational cost of (POC _i ), where POC _i is derived from POC _i = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) , where POC _{(i-1 )} is the possible operation cost calculated for the (i-1)th time point, PPN _(i+1) is the PPN at the _{(i+1)th time point in U (i+1)} to M, PPN _i is the PPN _at the i-th time point in U _i to M, and is used to calculate the PPN i of POC _i and POC _(i-1) with the same value; c2) Find the smallest and second smallest POC _i ; and c3) If the smallest and second smallest POC _i are calculated from the same PPN _i , then the PPN _i used to calculate the smallest POC _i is set to the i-th specified quantity, calculated from the i-th specified quantity at the next time point Remove the uncalculated POC _i ; and d) provide 1 unit of resources by the processor at the 0th time point and provide the i-th specified unit amount of resources to the workload at the i-th time point.

The method may further include the following sub-steps: c4) If the smallest and second smallest POC _i are not calculated from the same PPN _i , then individually use PPN _i to calculate POC _(i+1) , which will be used to calculate the smallest POC _{(i+1 )} of PPN _i is set to the i-th specified quantity, and the uncalculated POC _i is removed from the calculated i-th specified quantity at the next time point, where POC _(i+1) is the calculated i-th specified quantity at the (i+1)th time point The calculated possible operational cost.

Preferably, the resource is memory modules, CPU, I/O throughput, response time, requests per second, or latency.

The present invention also discloses another method of optimizing resource allocation based on predictions of reinforcement learning. The processor determines the number of resource units in the computer cluster to be utilized. The method includes the following steps: a) Provide the processor with a prediction of the number of resource units required for the workload exceeding N time points after the 0th time point, where a maximum of M unit sources can be provided, U _i is based on Predict the number of units required at the i-th time point, N, M, and i are all positive integers; b) The processor calculates at least one 0th possible operation cost (POC ₀ ), which is based on U ₁ to U ₁ + at least one possible supply quantity (PPN) at the first time point (PPN ₁ ) in the smallest of A and M, where POC ₀ is derived from POC ₀ = K + RF x | PPN ₁ – K | + PPN ₁ , where RF is a rebalancing factor between 0 and 1, A is an integer, and K is a real number; c) The processor repeats the following sub-steps in sequence for the i-th time point, i is from 1 to N: c1) Calculate at least one i-th possible operation cost (POC _i ), where POC _i is derived from POC _i = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) , where POC _(i-1) is the possible operation cost calculated for the (i-1)th time point, and PPN _(i+1) is between U _(i+1) and The PPN at the _(i+1) th time point of U (i+1) + the smallest of A and M, PPN _i is the smallest of U _i to U _i +A and M The PPN at the i-th time point is used to calculate the PPN _i of POC _i and POC _(i-1) with the same value; c2) find the smallest and second smallest POC _i ; and c3) if the smallest and second smallest POC _i is calculated from the same PPN _i , then the PPN _i used to calculate the minimum POC _i is set to the i-th specified quantity, and the uncalculated POC _i is removed from the i-th specified quantity calculated at the next time point; and d) The processor provides 1 unit of resources at the 0th time point and provides the ith specified unit amount of resources to the workload at the ith time point.

The method may further include the following sub-steps: c4) If the smallest and second smallest POC _i are not calculated from the same PPN _i , then individually use PPN _i to calculate POC _(i+1) , which will be used to calculate the smallest POC _{(i+1 )} of PPN _i is set to the i-th specified quantity, and the uncalculated POC _i is removed from the calculated i-th specified quantity at the next time point, where POC _(i+1) is the calculated i-th specified quantity at the (i+1)th time point The calculated possible operational cost.

Preferably, the resource is memory modules, CPU, I/O throughput, response time, requests per second, or latency.

Another method of optimizing resource allocation based on reinforcement learning predictions is implemented by the processor determining the number of resource units in the computer cluster to be used, including the following steps: a) Provide more than N after the 0th time point The prediction of the number of resource units required by the workload at a time point is given to the processor, where a maximum of M units can be provided. U _i is the number of units required at the i-th time point according to the prediction, N, M, i are all positive integers; b) the processor calculates at least one possible operation cost (POC ₁ ), which is based on at least one possible provision at the first time point (PPN ₁ ) in U ₁ to M Amount (PPN) and at least one PPN at the 2nd time point (PPN ₂ ) in U ₂ to M, where POC ₀ is derived from POC ₁ = K + RF x |PPN ₁ - K| + PPN ₁ + RF x |PPN ₂ – PPN ₁ | + PPN ₂ , where RF is a rebalancing factor between 0 and 1, and K is a real number; c) Set PPN _{1 used to calculate the minimum POC 1 as} the first specified Quantity; d) The processor repeats the following sub-steps sequentially for time points, i is an even number from 2 to 2 x [N/2]: d1) Calculate at least one (i+1)th possible operation cost POC _{( i+1)} , where POC _(i+1) is derived from POC _(i+1) = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) + Wi , where Wi is RF x | PPN _(i+2) – PPN _(i+1) | + PPN _(i+2) , POC _(i-1) is the possibility calculated for the (i-1)th time point The operation cost of , PPN _(i+2) is the PPN at the ( _i+2 )th time point in U _{(i+2) to} PPN at the (i+1)th time point in M, PPN _i is the PPN at the i-th time point in U _i to M, used to calculate POC _(i+1) and POC _(i-1) PPN _i has the same value; where if (i+2) is greater than N, then Wi is omitted from the calculation; d2) find the smallest and second smallest POC _i ; and d3) if the smallest and second smallest POC _i are calculated from the same PPN _i , then the PPN _i used to calculate the minimum POC _(i+1) is set to the i-th specified number and the PPN (i _{+1) used to calculate the minimum POC (i+ 1)} is set to the (i+1)-th Specify the quantity, remove the uncalculated POC _(i+1) from the calculated ith specified quantity at the next time point; and e) provide 1 unit of resources by the processor at the 0th time point and at the jth time Point provides the jth specified unit amount of resources to the workload, where j is in the range 1 to N.

Preferably, the resource is memory modules, CPU, I/O throughput, response time, requests per second, or latency.

The invention will now be described in more detail with reference to the following examples.

Please refer to Figure 1. It shows the hardware architecture to which the invention is applied. The computer cluster 10 (eg, data center) has many computing units 100, each with multiple CPUs 101 and memory modules 102 (such as SDRAM), sharing data stored in the storage device 103 (such as a hard disk). The computing units 100 may cooperate via the internal data network 110 to support the computing of workloads. The computer cluster 10 further connects the client 30 to the Internet 20 through the network communication interface 120 . For example, the computing unit 100 provides streaming services to the client 30 as a workload. The number of CPUs 101 and memory modules 102 provided to a workload determines how quickly the workload's requests can be responded to. CPU and memory modules are called resources in the embodiment. While more resources available to a workload improves the performance of the workload, it also places a load on the entire system. The life of the resource is consumed and other workloads cannot share the same resource. Therefore, the present invention is a method for optimizing resource allocation. Configurations are based on predictions of resource requirements but modified by reinforcement learning. The method is implemented by a processor 101a (ASIC or one of the CPUs 100) in one of the computing units 100. The processor 101a determines the number of units of resources in the computer cluster 10 to be utilized, the utilization being a result of the method. Predictions may be calculated and provided by another CPU 101b, or input from an external system outside of the computer cluster 10.

Please refer to Figure 2. It is a flow chart of the method disclosed in the first embodiment of the present invention. The first step of the method is to provide the processor with a prediction of the number of resource units required for the workload exceeding N time points after the 0th time point, where a maximum of M units can be provided, U _i is based on the prediction The number of units required at the i-th time point, N, M, and i are all positive integers (S01). The prediction can be any method based on algorithms, formulas, experience, or even artificial intelligence, as long as there are unit quantities of resources required for future workloads. According to the present invention, not all predicted results are used in the calculation. Only a limited amount N is used. N is a number defined by the user of this invention. In this example, N is set to 9. Forecasts can provide resource requirements for more than N points in time. The additional information can be used in another stage of resource allocation calculations. In this embodiment, M is 6 as an example. Given the values of M and N, for the next step of the operation, the prediction results are shown in Figure 3. The solid dots represent the predicted number of units of the resource (for example, CPU) at different points in time. For example, according to predictions, 6 units of resources will be required at the third time point. Broadly speaking, all controllable computing results of the system that can improve workload performance are considered a resource, such as but not limited to I/O throughput, response time, requests per second, and latency.

The second step of the method is for the processor to calculate at least one 0th possible operation cost (POC ₀ ) based on at least one possible 1st time point (PPN ₁ ) in U ₁ to M Provides the quantity (PPN) where POC ₀ is obtained from POC ₀ = K + RF x | PPN ₁ – K | + PPN ₁ , where RF is a rebalancing factor between 0 and 1 and K is a real number (S02 ). This step is performed by the processor 101a. For better clarity, please refer to Figure 5. It lists calculations for a subset of time points. Each calculation is divided into three matrices. The left matrix lists all PPNs at previous time points. The middle matrix is compared to Figure 4, showing part of the calculation on transition costs. The right matrix adds the POC to each transition cost in the same column, showing the results of finding the specified number for the current point in time. The basic idea of the present invention is to determine the appropriate unit number of resources for a point in time. It finds the minimum cumulative cost by considering the cumulative computational cost from all kinds of combinations of resources provided to the workload for computing at previous points in time, using It is possible to provide a quantity as a specified amount of resources to be utilized by the workload. If necessary, consider calculations at the next point in time. The entire process extends from the logic of reinforcement learning (dynamic programming), using similar concepts such as cost function, maximum reward (minimum cost), and iteration. However, to reduce lengthy calculation procedures, reasonable and unnecessary calculations are omitted. PPN represents the total number of candidates at a point in time. PPN _i is any number from the predicted quantity U _i to the maximum quantity M that the system can provide. For example, at the second time point, the predicted number of units of required resources is 3, and up to 6 units of resources can be utilized. The PPN ₂ used for the calculations at the first and second time points are 3 and 4. ,5,6. If U _i and M are the same, only one quantity is used. At any point in time, when a unit of a resource is utilized, the life of the resource is reduced and power is consumed. To simplify and quantify the results, all subsequent calculations use the minimum cost, 1, as K. For convenience of explanation, K is set to 1 in all embodiments. Of course, the number of resource changes between two subsequent time points is also expensive, but not as much as using just one. Therefore, rebalancing factors are used to reduce changes. RF varies with different kinds of resources. The selection of RF is based on the characteristics of the resource. In this embodiment, 0.6 is taken as an example.

This is an exception for calculations at time point 0. Since there is no previous time point, a starting value should be specified as the cumulative cost. According to the present invention, the number is "1". In the above calculation formula, "|PPN ₁ - 1|" gets the number of resource changes between the 0th and 1st time points, "RF x |PPN ₁ - 1|" reduces |PPN ₁ - 1| by 40 % (RF = 0.6), the calculation of the first time point resulted in two POC ₀ : 8.4, 10. It should be noted that in other instances, the number of POC ₀ can be greater than 2, and can also be 1.

The third step of the method is for the processor to sequentially repeat the following sub-steps for the i-th time point, where i is from 1 to N (S03). This step is the calculation of all N time points. This sub-step is detailed below.

The first sub-step is to calculate at least one ith possible operation cost (POC _i ), where POC _i is derived from POC _i = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) , where POC _(i-1) is the possible operation cost calculated for the (i-1)th time point, and PPN _(i+1) is among U _(i+1) to M PPN at the (i+1)th time point, PPN _i is the PPN at the i-th time point in U _i to M, used to calculate the PPN i of POC _i and POC _(i-1) with the same _value (S03 -1). For example, calculate the 1st time point. In Figure 5, two POC ₀ are used as "POC _(i-1) " of the above calculation formula in the first sub-step. Here, PPN _(i+1) is PPN ₂ and includes 3, 4, 5, and 6. PPN _i is PPN ₁ , including 5 and 6 as above. To simplify the operation of "RF x | PPN _(i+1) - PPN _i |", the lookup table in Figure 4 shows the calculation considering U _i and U _{(i+1) from} 1 to 6, ₊₁₎ and PPN _i both remain at 6, and RF is equal to 0.6. Since the right matrix only adds PPN to each transition cost in the same column, 8.4 is added to 4.2, 4.6, 5, 6.6, but not to 4.8, 5.2, 5.6, 6. Therefore, the resulting column of POC ₁ (POC _i ) is 12.6, 13, 13.4, and 15. Likewise, 10 adds to 4.8, 5.2, 5.6, and 6, but not to 4.2, 4.6, 5, and 6.6. The other column of POC ₁ obtained is 14.8, 15.2, 15.6, 16. "5" in PPN ₁ is used to calculate "8.4" in POC ₀ and 12.6, 13, 13.4, and 15 in POC ₁ .

The second sub-step is to find the smallest and second smallest POC _i (S03-2). In Figure 5, the smallest and second smallest POC ₁ are 12.6 and 13 respectively. Both come from the same PPN ₁ , but PPN ₂ is different.

The third sub-step is that if the smallest and second smallest POC _i are calculated from the same PPN _i , then the PPN _{i used to calculate the smallest POC i is} set to the i-th specified quantity, from the i-th calculation at the next time point The specified number is removed from the uncalculated POC _i (S03-3). As shown above, the smallest and second smallest POC ₁ come from the same PPN ₁ , so the premise is met. The PPN ₁ used to calculate the smallest POC _i (12.6) is 5 (connected to 12.6 by an arrow) and is designated as the first specified quantity. . The calculation of the next time point excludes POC ₁ that is not calculated from the first specified quantity 5 (that is, 14.8, 15.2, 15.6, 16). Only 12.6, 13, 13.4, and 15 are left for the second time point. See Figures 5 to 7 for the same calculations.

It should be noted that the calculations from steps S01 to S03 are only used for resource allocation when the first point in time comes, and nothing is used at this time. The calculation is prepared before time point 1, even before time point 0.

The final step of the method is for the processor to provide 1 unit of resources at the 0th time point and provide the ith specified unit amount of resources to the workload at the ith time point (S04). This step is a resource configuration step. Based on the relevant data calculated in the previous steps, the unit quantities of resources are 5, 5, 6, 5, 5, 5, 5, 5, and 6 for the first time point to the ninth time point respectively. The result is different from the prediction that requires 5, 2, 6, 1, 5, 6, 5, 2, and 9 respectively from the 1st time point to the 9th time point. The number of units provided by the resource minimizes the cost of the operation.

Although the same calculation is performed for time points 2 to 9, there are two exceptions in different situations.

The first exception is shown in Figure 6. The prerequisite of the third sub-step requires that the smallest and second smallest POC _i should be calculated from the same PPN _i . However, when processing calculations for the fourth time point, the smallest and second smallest POC ₄ are calculated from different PPN ₄ s. That is, the smallest POC ₄ (30.6) is calculated from 5, and the second smallest POC ₄ (32) is calculated from 6. This means that it is difficult to know which PPN ₄ has the lowest computational cost for the 5th time point. According to the present invention, the solution to this situation is to use the two individually calculated PPN _4s for the fifth time point, and determine the fourth designated quantity based on the results of the two calculations. Therefore, a fourth sub-step is required, if the smallest and second smallest POC _i are not calculated from the same PPN _i , then individually use PPN _i to calculate POC _(i+1) , which will be used to calculate the PPN of the smallest POC _(i+1) Set _i to the i-th specified quantity, and remove the uncalculated POC _i from the i-th specified quantity calculated at the next time point, where POC _(i+1) is calculated for the (i+1)-th time point Possible operational costs. In the example of Figure 6, 5 of PPN ₄ and related POC ₄ (30.6 and 32.2) are used for the calculation of the 5th time point, version 1; 6 of PPN ₄ and related POC ₄ (32.2 and 32) are used for the 5th time point. Calculation of time points, version 2. In this case, the two versions showing 5 are PPN ₄ to calculate the minimum POC ₅ (34.4 and 35.4). Then 5 is set back to the 4th specified quantity. The rest is the same as the third sub-step and will not be repeated.

The second exception occurs in the calculation at the 9th time point. Because this method only accepts predictions for 9 time points, even if the 10th time point can be predicted, it will not be used. Therefore, PPN ₁₀ is set to "0". The calculation is shown in Figure 7.

From a traditional reinforcement learning point of view, if you do not lose any change in finding the minimum cost, you should calculate all the information of the right matrix in Figure 5 to Figure 7. For example, the right matrix calculated at the first time point only contains 8 data, but 36 calculations should be done to obtain 36 data. At this time point, 28 calculations are saved; from the 0th time point to the 9th time point, the entire process saves 257 calculations; the actual situation has thousands of resource units and hundreds of time points; millions can be saved calculation. With the help of this method, exponential complexity can be reduced. Close to optimal and acceptable results can be achieved within a limited time, greatly increasing availability and reducing computing costs.

According to the spirit of the present invention, the calculation amount can be further reduced in another improved method. An improved method is shown in the second example.

Please refer to Figure 8. It is a flow chart of an improved method disclosed in the second embodiment of the present invention. To simplify the description of the second embodiment, the resource demand prediction of the first embodiment is still applied here. In addition, the second embodiment is also applied to the architecture of FIG. 1 . The maximum unit M of available sources remains 6. The only difference between the method of the first embodiment and the improved method is the definition of PPN. The amount of PPN at each time point for this example was further reduced. PPN contains the number of units U _i required at the i-th time point and other larger quantities. Larger quantities may not contain M. The improved method is shown in Figures 10 and 11, which list the calculations at all time points.

The first step of the improved method is to provide the processor with a prediction of the number of resource units required for the workload beyond N time points after the 0th time point, where a maximum of M units can be provided, U _i is based on the prediction The number of units required at the i-th time point, N, M, and i are all positive integers (S11). This step is the same as step S01. As mentioned above, N and M remain the same to illustrate the second embodiment.

The second step of the improved method is for the processor to calculate at least a 0th possible operation cost (POC ₀ ) based on the 1st of the smallest of U ₁ to U ₁ +A and M At least one possible offering quantity (PPN) at a point in time (PPN ₁ ), where POC ₀ is derived from POC ₀ = K + RF x | PPN ₁ – K | + PPN ₁ , where RF is between 0 and 1 Rebalancing factor, A is an integer, and K can be any real number, here K is set to 1 (S12). RF is also set to 0.6. The difference between step S02 and step S12 is that PPN ₁ is limited to its upper limit. To better understand the differences, please refer to Figure 9. As in Figure 3, with the same prediction results, but different numbers of possible offers. In this example, A is 2. For each time point calculation, the bottom line is U _i and the upper limit is the smaller of U ₁ +A or M. For the calculation at the 0th time point, U ₁ +A is equal to 7 and M is 6. Therefore, since the maximum available resource is 6 units, PPN ₁ is 6. It cannot be 7. Since 1 is given as the minimum cost at the beginning, the calculation results of POC ₀ at the 0th time point are also 8.4 and 10. Please return to Figure 7. If for each i-th time point, the upper limit of PPN _i is connected by a dotted line and the lower limit of PPN _i is connected by a solid line, a "tunnel" is formed. A specified number of resource units fall into the tunnel at all points in time.

The third step of the improved method is for the processor to sequentially repeat the following sub-steps for the i-th time point, where i is from 1 to N (S13). Seems to be the same as step S03. However, the sub-steps are different. Below are descriptions of these sub-steps.

The first sub-step is to calculate at least one ith possible operation cost (POC _i ), where POC _i is derived from POC _i = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) , where POC _(i-1) is the possible operation cost calculated for the (i-1)th time point, and PPN _(i+1) is between U _(i+1) and U _{(i +1)} +The PPN at the (i+1)th time point of the smallest of A and M, PPN _i is the i-th of the smallest of A _and M from U _i to U i The PPN at the time point used to calculate the _PPN i of POC _i and POC _(i-1) has the same value (S13-1). Obviously, the upper limits of PPN _(i+1) and PPN _i have changed. For the calculation of the first time point, PPN ₂ includes 3, 4, and 5. U ₂ is predicted to be 3. Since the smallest of U ₂ +2 and 6 is U ₂ +2 (5), 6 is no longer used. Likewise, PPN ₃ only contains 6. PPN ₄ contains 1, 2, and 3. PPN ₅ contains 5 and 6. PPN ₆ includes 2, 3, and 4. PPN ₇ contains 5 and 6. PPN ₈ includes 2, 3, and 4. PPN ₉ contains 6. Since PPN has different definitions, the calculation results of POC _i change accordingly. For example, in the calculation at the first time point, 8.4 is added to 4.2, 4.6, and 5, but not to 4.8, 5.2, and 5.6. Therefore, the resulting POC ₁ (POC _i ) of a column is 12.6, 13, 13.4. Likewise, 10 will add to 4.8, 5.2, 5.6, but not 4.2, 4.6, 5. The resulting POC ₁ of the other column is 14.8, 15.2, 15.6. "5" in PPN ₁ is used to calculate "8.4" in POC ₀ and 12.6, 13, and 13.4 in POC ₁ .

The second sub-step is to find the smallest and second smallest POC _i (S13-2). This step is the same as sub-step S03-2. In Figure 10, the smallest and second smallest POC ₁ are 12.6 and 13 respectively. They are from the same PPN ₁ , but PPN ₂ is different.

The third sub-step is that if the smallest and second smallest POC _i are calculated from the same PPN _i , then the PPN _{i used to calculate the smallest POC i is} set to the i-th specified quantity, from the i-th calculation at the next time point The uncalculated POC _i is removed by a specified number (S13-3). This step is the same as sub-step S03-3. Since the second step and first sub-step of the two methods are different, the subsequent results are also different. For example, using the same prediction, the 4th designated quantity for the second embodiment is 3, while the 4th designated quantity for the first embodiment is 5.

Similarly, the final step of the improved method is for the processor to provide 1 unit of resources at the 0th time point and provide the ith specified unit amount of resources to the workload at the ith time point (S14). It is the same as step S04 and will not be described in detail.

Like the first embodiment, the improved method encounters two exceptions. In the second embodiment, the first exception is not seen. However, it may occur in other instances. In this case, the following substep is performed: If the smallest and second smallest POC _i are not calculated from the same PPN _i , then individually use PPN _i to calculate POC _(i+1) , which will be used to calculate the smallest POC _(i+1) The PPN _i is set to the i-th specified quantity, and the uncalculated POC _i is removed from the calculated i-th specified quantity at the next time point, where POC _(i+1) is the calculated i-th specified quantity at the (i+1)th time point. The possible operational cost of the calculation.

The second exception also occurs in the calculation of the 9th time point in the second embodiment. The same means as in the first embodiment are used to process PPN ₁₀ here. PPN ₁₀ is set to "0". The calculation is shown in Figure 11.

The total number of calculations for the method of the first embodiment is 67 (irrelevant to the calculation at the 5th time point, version 2 will not occur in the second embodiment), while the improved method of the second embodiment is 41. Compared with the first embodiment, the improvement method of the second embodiment requires less calculation. Save 26 calculations. Although the amount of resources provided at each time point is different, there is no significant difference and it is valid for the same prediction.

In the above embodiment, a specified quantity is derived from calculations at two points in time. According to the present invention, more than a specified amount can be obtained from the calculation of two points in time. From another perspective, the previous embodiment collects data to determine the resources to provide during a "window" between two points in time. More empty windows can be used to collect information. The number of calculations increases, but time is saved. Another method of optimizing resource allocation based on reinforcement learning predictions is shown in the third embodiment.

Please refer to Figure 12. It is a flow chart of the method disclosed in the third embodiment of the present invention. In this embodiment, the resource demand prediction in the previous embodiment is still applied. Likewise, the third embodiment can also be used in the architecture of FIG. 1 . The maximum unit M of available sources remains 6. The method is illustrated in Figures 15 to 19, which list the calculations for all time points.

The first step of the method is to provide the processor with a prediction of the number of resource units required for the workload exceeding N time points after the 0th time point, where a maximum of M units can be provided, U _i is based on the prediction The number of units required at the i-th time point, N, M, and i are all positive integers (S21). This step is the same as steps S01 and S11. As mentioned above, N and M remain the same to illustrate this embodiment.

The second step of the method is for the processor to calculate at least a first possible operation cost (POC ₁ ) based on at least a possible first time point (PPN ₁ ) in U ₁ to M Provide the quantity (PPN) and at least one PPN at the 2nd time point (PPN ₂ ) in U ₂ to M, where POC ₀ is derived from POC ₁ = K + RF x |PPN ₁ - K| + PPN ₁ + RF x |PPN ₂ – PPN ₁ | + PPN ₂ , where RF is a rebalancing factor between 0 and 1, and K is a real number (S22). RF is also set to 0.6 and K is 1. To better understand the differences, please refer to Figure 13 and Figure 15. Figure 13 has the same prediction results as Figure 3. Figure 14 contains two calculation tables. The above table lists all calculations for RF x |PPN ₁ - 1| + PPN ₁ + RF x |PPN ₂ – PPN ₁ | + PPN ₂ , while the below table shows POC ₁ = 1 + RF x |PPN ₁ - 1| + All calculations of PPN ₁ + RF x |PPN ₂ – PPN ₁ | + PPN ₂ . "1" (not in PPN ₁ and POC ₁ ) is the number specified in the table below. The initial operation cost is set to provide only one resource. The formula for calculating POC ₁ is similar to the formula for POC _i , but the calculation spans three time points (two empty windows).

The third step of the method is to set the PPN ₁ used to calculate the minimum POC ₁ as the first specified quantity (S23). The specified number of functions is the same as in the first embodiment. Since POC ₁ is the smallest, PPN ₁ can lead the results and be selected as the 1st specified quantity.

The fourth step of the method is for the processor to sequentially repeat the following sub-steps for time points, i is an even number from 2 to 2 x [N/2] (S24). The definition of "i" is different from the previous embodiment. First, "i" is an even number, such as 2, 4, 6...etc. [N/2] Calculated in Gaussian notation. In this example, N is 9, so [N/2] is 8. In other words, 2, 4, 6, and 8 are taken as "i" of calculations in different iterations. These sub-steps are described below.

The first sub-step is to calculate at least one (i+1)th possible operation cost POC _(i+1) , where POC _(i+1) is obtained from POC _(i+1) = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) + Wi, where Wi is RF x | PPN _(i+2) – PPN _(i+1) | + PPN _(i+2) , POC _(i-1) is the possible operation cost calculated for the (i-1)th point in time, PPN _(i+2) is the (i+2)th one from U _(i+2) to M PPN at time point, PPN _(i+1) is the PPN at the _{(i+1)th time point in U (i+1)} to M, PPN _i is the i-th time in U _i to M The PPN of the point, used to calculate the PPN _i of POC _(i+1) and POC _(i-1), has the same value; where if (i+2) is greater than N, Wi (S24-1) is omitted from the calculation. For convenience, POC _(i+1) = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) + RF x | PPN _(i+2) – The results for PPN _(i+1) | + PPN _(i+2) are shown in Figure 14, while RF=0.6. For example, let i = 2. See Figure 16. Figure 16 lists the calculations for all time points. Here, POC ₃ is calculated from POC ₁ . In Figure 15, POC ₁ is 12.6, 13, 13.4, and 15. The formula becomes POC ₃ = POC ₁ + RF x | PPN ₃ - PPN ₂ | + PPN ₃ + RF x | PPN ₄ - PPN ₃ | + PPN ₄ . The table above shows the results for RF x | PPN ₃ – PPN ₂ | + PPN ₃ + RF x | PPN ₄ – PPN ₃ | + PPN ₄ . In Figure 16, PPN ₂ is 3 to 6, PPN ₃ is only 6, and PPN ₄ is 1 to 6. Therefore, only 24 quantities are calculated and shown in the table above. The results for POC ₃ are at the bottom right of the table below.

The second sub-step is to find the smallest and second smallest POC _i (S24-2). This step is the same as sub-steps S03-2 and S13-2. In Figure 16, the smallest and second smallest POC ₃ are 24 and 24.2 respectively. They are from different PPN ₂ .

The third sub-step is that if the smallest and second smallest POC _i are calculated from the same PPN _i , then the PPN _i used to calculate the smallest POC _(i+1) is set to the i-th specified number and used to calculate the smallest POC _(i+ The PPN _(i+1 ) of ₁₎ is set to the (i+1)-th specified quantity, and the uncalculated POC _(i+1) is removed from the calculated i-th specified quantity at the next time point (S24-3). In this embodiment, although the smallest and second smallest POC ₃ are calculated from the same PPN ₂ , the PPN ₂ used can be found by the same procedure of the second embodiment. It is omitted here and will not be repeated. 24 is PPN ₂ (point background). Therefore, the 2nd specified quantity is 5 and the 3rd specified quantity is 6 (dot background). The POC ₃ used for the calculation of the next iteration is 24, 24.4, 24.8, 25.2, 25.6, 26. The calculation results for i=4 and i=6 are shown in Figure 17 and Figure 18.

A different calculation occurs when i is 8. There is no PPN ₁₀ . It matches the case where _(i+2) is greater than N (10>9). Omit the part of Wi from the calculation. The calculation results are shown in Figure 19. The simplified formula becomes POC ₉ = POC ₇ + RF x | PPN ₉ - PPN ₈ | + PPN ₉ . This formula is the same as the previous embodiment. However, the 8th and 9th specified quantities are selected according to the same method. The 8th specified quantity is 5, and the 9th specified quantity is 6.

The final step of the method is for the processor to provide 1 unit of resources at the 0th time point and provide the jth specified unit amount of resources to the workload at the jth time point, where j is among 1 to N (S25 ). It is the same as steps S04 and S14, but the variable signs are different.

The calculated total number for the method of the third embodiment is 137. Compared with the first and second embodiments, the method of the third embodiment uses more calculations. Although the number of resources provided at each point in time may not be the same, it is not much different, and the time to produce results can be reduced.

It should be noted that all the above mathematical formulas are only for illustration and do not limit the application of the present invention. Any other mathematical formula that expresses the same calculus logic is also within the scope of the invention.

While the invention has been described in terms of what are presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not necessarily limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements included within the spirit and scope of the appended claims, and they are to be accorded the broadest interpretation so as to cover all such modifications and similar arrangements.

10:Computer cluster 20:Internet 30: Client 100:Computing unit 101:CPU 101a: Processor 101b:CPU 102:Memory module 103:Storage device 120:Network communication interface

Figure 1 shows the hardware architecture of the application of the present invention. FIG. 2 is a flowchart of the method disclosed in the first embodiment of the present invention. Figure 3 shows the prediction results. Figure 4 is a lookup table. Figures 5 to 7 list the calculations for all time points. FIG. 8 is a flow chart of an improved method disclosed in the second embodiment of the present invention. Figure 9 shows the same prediction results as Figure 3, with a different number of possible offers. Figures 10 and 11 list the calculations for all time points. FIG. 12 is a flowchart of a method disclosed in the third embodiment of the present invention. Figure 13 shows the prediction results. Figure 14 is a lookup table. Figures 15 to 19 list the calculations for all time points.

10:Computer cluster

20:Internet

30: Client

100:Computing unit

101:CPU

101a: Processor

101b:CPU

102:Memory module

103:Storage device

120:Network communication interface

Claims (8)

A method to optimize resource allocation based on reinforcement learning predictions is implemented by the processor determining the number of resource units in the computer cluster to be used, including the following steps: a) Provide more than N units after the 0th time point The prediction of the number of resource units required by the workload at a point in time is given to the processor, which can provide a source of up to M units. U _i is the number of units required at the i-th point in time based on the prediction. N, M, and i are all is a positive integer; b) The processor calculates at least one 0th possible operation cost (POC ₀ ), which is based on at least one possible supply quantity at the 1st time point (PPN ₁ ) in U ₁ to M (PPN), where POC ₀ is obtained from POC ₀ = K + RF x | PPN ₁ – K | + PPN ₁ , where RF is a rebalancing factor between 0 and 1, and K is a real number; c) by The processor sequentially repeats the following sub-steps for the i-th time point, i is from 1 to N: c1) Calculate at least one i-th possible operation cost (POC _i ), where POC _i is obtained from POC _i = POC _{(i -1)} + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) , where POC _(i-1) is the possible operation cost calculated for the (i-1)th time point , PPN _(i+1) is the PPN at the _{(i+1)-th time point in U (i+1)} to M, PPN _i is the PPN at the i-th time point in U _(i+ 1) to M, use to calculate POC _i and POC _(i-1) whose PPN _i have the same value; c2) find the smallest and second smallest POC _i ; and c3) if the smallest and second smallest POC _i are calculated from the same PPN _i , then Set the PPN _i used to calculate the minimum POC _i to the i-th specified quantity, and remove the uncalculated POC _i from the calculated i-th specified quantity at the next time point; and d) by the processor at the 0th time point Provide 1 unit of resources and provide the i-th specified unit of resources to the workload at the i-th point in time. For example, the method in item 1 of the patent scope further includes the following sub-steps: c4) If the smallest and second smallest POC _i are not calculated from the same PPN _i , then use PPN _i individually to calculate POC _(i+1) , which will be used The PPN _i for calculating the minimum POC _(i+1) is set to the i-th specified quantity, and the uncalculated POC _i is removed from the calculated i-th specified quantity at the next time point, where POC _(i+1) is the ( The possible operation cost calculated at i+1) time points. For example, the method of claim 1, wherein the resource is a memory module, CPU, I/O throughput, response time, requests per second, or latency. A method to optimize resource allocation based on reinforcement learning predictions is implemented by the processor determining the number of resource units in the computer cluster to be used, including the following steps: a) Provide more than N units after the 0th time point The prediction of the number of resource units required by the workload at a point in time is given to the processor, which can provide a source of up to M units. U _i is the number of units required at the i-th point in time based on the prediction. N, M, and i are all is a positive integer; b) The processor calculates at least a 0th possible operation cost (POC ₀ ), which is based on the 1st time point in the minimum of U ₁ to U ₁ +A and M At least one possible supply quantity (PPN) of (PPN ₁ ), where POC ₀ is derived from POC ₀ = K + RF x | PPN ₁ – K | + PPN ₁ , where RF is a rebalance between 0 and 1 Factor, A is an integer, and K is a real number; c) The processor repeats the following sub-steps in sequence for the i-th time point, i is from 1 to N: c1) Calculate at least one i-th possible operation cost ( POC _i ), where POC _i is derived from POC _i = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) , where POC _(i-1) is the The possible operation cost calculated at (i-1) time points, PPN _(i+1) is the smallest of U _(i+1) to U _(i+1) +A and M. i+1) time point PPN, PPN _i is the PPN at the i-th time point from U _i to U _i +A and M, whichever is the smallest, used to calculate POC _i and POC _{(i-1 )} have the same value for PPN _i ; c2) find the smallest and second smallest POC _i ; and c3) if the smallest and second smallest POC _i are calculated from the same PPN _i , then the PPN _i that will be used to calculate the smallest POC _i Set to the i-th specified quantity, remove the uncalculated POC _i from the calculated i-th specified quantity at the next time point; and d) provide 1 unit of resources by the processor at the 0th time point and at the i-th Provide the i-th specified unit amount of resources to the workload at the point in time. For example, the method in item 4 of the patent application scope further includes the following sub-steps: c4) If the smallest and second smallest POC _i are not calculated from the same PPN _i , then use PPN _i individually to calculate POC _(i+1) , which will be used The PPN _i for calculating the minimum POC _(i+1) is set to the i-th specified quantity, and the uncalculated POC _i is removed from the calculated i-th specified quantity at the next time point, where POC _(i+1) is the ( The possible operation cost calculated at i+1) time points. For example, in the method of claim 4, the resource is a memory module, CPU, I/O throughput, response time, requests per second, or latency. A method to optimize resource allocation based on reinforcement learning predictions is implemented by the processor determining the number of resource units in the computer cluster to be used, including the following steps: a) Provide more than N units after the 0th time point The prediction of the number of resource units required by the workload at a point in time is given to the processor, which can provide a source of up to M units. U _i is the number of units required at the i-th point in time based on the prediction. N, M, and i are all is a positive integer; b) The processor calculates at least one first possible operation cost (POC ₁ ), which is based on at least one possible provided quantity at the first time point (PPN ₁ ) in U ₁ to M (PPN) and at least one PPN at the 2nd time point (PPN ₂ ) in U ₂ to M, where POC ₀ is derived from POC ₁ = K + RF x |PPN ₁ - K| + PPN ₁ + RF x | PPN ₂ – PPN ₁ | + PPN ₂ , where RF is a rebalancing factor between 0 and 1, and K is a real number; c) Set PPN _{1 used to calculate the minimum POC 1 to} the first specified quantity ; d) The processor repeats the following sub-steps sequentially for time points, i is an even number from 2 to 2 x [N/2]: d1) Calculate at least one (i+1)th possible operation cost POC _{(i +1)} , where POC _(i+1) is obtained from POC _(i+1) = POC _(i-1) + RF x | PPN _(i+1) - PPN _i | + PPN _(i+1) + Wi, _{where Wi is RF} Operation cost, PPN _(i+2) is the PPN at the _{(i+2)th time point in U (i+2)} to M, PPN _(i+1) is the PPN at the _(i+1) to M _PPN at the ( _i + _{1 )} th time point in _i has the same value; where if (i+2) is greater than N, then Wi is omitted from the calculation; d2) find the smallest and second smallest POC _i ; and d3) if the smallest and second smallest POC _i are calculated from the same PPN _i , Then set the PPN _i used to calculate the minimum POC _(i+1) to the i-th specified quantity and the PPN _(i+1) used to calculate the minimum POC _(i+ 1) to the (i+1)-th specified quantity. quantity, excluding the uncalculated POC _(i+1) from the calculated i-th specified quantity at the next time point; and e) 1 unit of resources provided by the processor at the 0th time point and at the jth time point Provides the jth specified unit amount of resources to the workload, where j is in the range 1 to N. For example, in the method of claim 7, the resource is a memory module, CPU, I/O throughput, response time, requests per second, or latency.