CN109101315B

CN109101315B - Cloud data center resource allocation method based on packet cluster framework

Info

Publication number: CN109101315B
Application number: CN201810727411.9A
Authority: CN
Inventors: 陈世平; 陆乐; 陈际阳
Original assignee: University of Shanghai for Science and Technology
Current assignee: University of Shanghai for Science and Technology
Priority date: 2018-07-04
Filing date: 2018-07-04
Publication date: 2021-11-19
Anticipated expiration: 2038-07-04
Also published as: CN109101315A

Abstract

The invention relates to a cloud data center resource allocation method based on a packet cluster frame, which is used for establishing the packet cluster frame and realizing an improved Hungarian method of the packet cluster frame. The invention realizes the sharing of resources, and the virtual machine resources can be shared with each other more easily according to the proposed packet cluster structure. When part of the virtual machine resources are idle, the part of the computing resources can be released to other virtual machines needing the resources; and secondly, multi-objective optimization of the data center. Through the packet cluster framework, metering and monitoring of the server are clearer, and modeling and optimization of targets such as cost, energy consumption and resource utilization rate of cloud computing are simpler. And the third step is the fine management of the virtual machine and the server: the virtual machine placement strategy is converted from the flat mapping relation between the original virtual machine and the original physical machine into the mapping relation between the package and the cluster, the mapping problem between the virtual machine and the physical machine is further degraded in scale, and the distribution efficiency and the distribution speed of the virtual machine are improved.

Description

Cloud data center resource allocation method based on packet cluster framework

Technical Field

The invention relates to a cloud computing resource allocation technology, in particular to a cloud data center resource allocation method based on a packet cluster framework.

Background

In recent years, the rapid development of cloud computing has brought the information technology industry forward, and has attracted the attention of a huge number of enterprises and users. Cloud computing provides services to the internet through large-scale, distributed virtualized data centers. Each data center contains numerous servers, and each physical server can be divided into several independent virtual machines.

When cloud computing technology is just appeared, the attention is directed to how to establish a high-performance computing data center and serve users through on-demand use, wherein one of the most important problems is the management mode of resource allocation, namely how to place new virtual machine requests on physical servers.

The resource management mode of the existing data center has the following three problems.

The distribution hierarchy is too flat: the existing virtual machine allocation strategy is to map virtual machines and physical machines through various algorithms, and under the condition that the number of servers and virtual machines reaches more than ten thousand, the allocation situation is about ten trillion, so that the calculation amount for calculating the optimal solution of virtual machine allocation is difficult to estimate. The communication flow among the virtual machines, the network topology model, the routing, the bandwidth sharing of the data center and the like are further considered, the scale of the problem is further enlarged, and even the dynamic allocation problem of the virtual machines is also considered, so that the redrawing and reallocation of the mapping relationship are very complicated.

The flexibility of allocating resources is too low: in the existing virtual machine allocation process, users need to describe the resource allocation requirement of each (if there are multiple) virtual machine they need in a very detailed manner, and sometimes give traffic, network topology, routing, etc. between virtual machine clusters, so that the data center optimizes the performance including resources, bandwidth, etc. after virtual machine allocation. However, in practical situations, many users may not know the professional knowledge of cloud computing, and all the users may specify the hardware and network resources of cloud computing in a group manner, and the existing virtual machine allocation cannot meet the requirements of users on flexibility, and does not allow resources to be shared among virtual machines. Here, the virtual machine resource sharing means that there is no fixed resource limit for each virtual machine, and the internal resources are allocated in the total amount of one group of virtual machines.

The management structure is too single: the service problems of the structural virtual machines cannot be solved, for example, a service provider is required to provide network building and a topological structure between the virtual machines, or different groups of the applied virtual machine resources are divided according to different conditions (enterprise areas, enterprise departments, employee job levels and the like), or the diversified optimization direction of the cloud computing data center cannot be satisfied: optimizing energy consumption, resource utilization, migration management, cost benefits and the like.

Disclosure of Invention

The invention provides a cloud data center resource allocation method based on a packet cluster framework, which aims at the problems of low efficiency of an allocation mode caused by flattening and fine particles and complexity of resource scheduling management of a cloud computing center, establishes a structured resource management mode and solves the problem of current cloud data resource allocation.

The technical scheme of the invention is as follows: a cloud data center resource allocation method based on a packet cluster frame specifically comprises the following steps:

1) establishing a packet cluster frame: the packet is an abstract expression of a user on the resource requirement of the virtual machine, is a resource-sharing virtual machine group, not only represents a cluster bottom layer packet required by a bottom layer virtual machine, but also can be a recursive definition of a high-level packet consisting of sub-packets; the cluster is an abstract expression of server resources, represents a cluster combination formed by the servers, and is defined by recursion of a high-level cluster formed by sub-clusters;

the mapping relation between the packet and the cluster is as follows: supposing that an arbitrary packet gamma exists in the hierarchical structure of the packet, M virtual machines or sub-packets v may exist under the packet gamma, wherein the number of the virtual machines or the sub-packets is limited to be more than or equal to 1 and less than or equal to M; when a package cluster frame is allocated, a package gamma at the same level is allocated to a cluster rho at the same level, and N servers or sub-clusters p are assumed to be arranged under the cluster rho, wherein p is more than or equal to 1 and less than or equal to N; the solution is that: distributing the M sub-packets to the N sub-clusters by an optimal method on the premise of minimizing energy consumption, and ensuring that sub-packets or virtual machines of the sub-packets are mapped to the sub-clusters or physical machines of the sub-clusters;

the framework relationships for formulating the package cluster map are as follows:

in formula (1), x_v,p[t]Represents whether or not a sub-packet v is allocated to the sub-cluster p in time t; x is the number of_v,p[t]The value of (A) is only 0 or 1; when sub-packet v is allocated to sub-cluster p, then x_v,p[t]1, otherwise x_v,p[t]＝0；

The package cluster relationship requires 1:

the expression (2) represents that at any time t, each sub-packet must be allocated to only one sub-cluster, but one sub-cluster can carry a plurality of sub-packets;

the package cluster relationship requires 2: the total amount of resources available for any sub-cluster must be greater than the total amount of sub-packets allocated on that sub-cluster:

in the formula (3), R_v,i[t]Represents the total amount of demand of each sub-packet v for resource i at time t, where i is 1,2, 3.., J; a. the_p,i[t]Then represents the total amount of resources i available to each sub-cluster P at time t, where i is 1,2, 3.., J;

2) the improved Hungarian algorithm for realizing the packet cluster framework comprises the following steps: calculating cost functions of the Hungarian algorithm and realizing the distribution of diversified packet clusters through a fast reduced order algorithm;

the fast order reduction algorithm comprises the following steps: (1) firstly, one cluster can be distributed with a plurality of packets, and the condition of different packet cluster numbers is solved through reducing the order; (2) when the 0 element in the Hungarian algorithm is an independent 0 element in rows and columns at the same time, the element is called as a 'double-independent 0 element', and the position is an optimal solution; (3) if no independent 0 element exists, the smaller the minimum value of the row or column of the independent 0 element except for the 0 element is, the better the solution of the 0 element is; (4) after selecting the independent 0 element, in order to reduce the number of iterations, all elements of the unassigned row or column are subtracted using the minimum value just except for the 0 element, and a new 0 element is directly constructed.

Calculating a cost function of the Hungarian algorithm in the step 2):

the cost function is as in equation (5):

U_v,p＝U_base+Cost_v,p+y_p (5)

in the formula (5), U_v,pRepresents the cost, U, of sub-packet v being assigned to sub-cluster p_baseRepresenting the existing basic cost of the sub-cluster p, wherein the basic cost includes the sum of the costs of sub-packets distributed on the sub-cluster, as shown in formula (6); cost_v,pRepresents the energy consumption value, y, of the current sub-packet v assigned to the sub-cluster p_pRepresenting whether the sub-cluster p can meet the requirement of the current packet, if not, setting the sub-cluster p to be infinite, and in the Hungarian algorithm, infinite cost values cannot participate in allocation;

based on the usage as a cost of energy consumption value, the following equation:

r in the formula (8)_v,cpu/A_p,cpu、R_v,men/A_p,men、R_v,io/A_p,ioThe three represent the occupation rates of the sub-packet v to the CPU, the memory and the disk of the sub-cluster p respectively, and the sub-packet is distributed to the sub-cluster more evenly so as to ensure that the resource utilization rate is maintained in a stable state.

The specific steps for realizing the distribution of the diversified packet clusters through the rapid order reduction algorithm are as follows:

calculating the cost value of each sub-packet to each sub-cluster according to the formula (5), and taking the number of the sub-clusters needing to be distributed as the number of rows and the number of the sub-packets as the number of columns to form a cost matrix;

converting the number of columns of the cost matrix according to the number of rows, and dividing the cost matrix into one or more matrixes with the number of rows and the number of columns completely equal;

if the last matrix in the partitioned matrix does not meet the condition that the number of rows and the number of columns are equal, complementing the number of rows or the number of columns by multiplying the number of rows or the number of columns to form a matrix with the form that the number of rows and the number of columns are completely equal;

and (3) carrying out the following processing on the matrix after the division: for example, for a matrix without complementary and complementary rows, each element of a row by row subtracts the row minimum; if the matrix is a complementary matrix of columns, subtracting the minimum value of each column from each element of each column, and entering the next step after finding a piece of processing meeting the condition;

step (4) searching the matrix processed in the step (3) for 'double independent 0 elements', checking whether an independent 0 element exists row by row (column), and if the 0 element is also an independent 0 element on a column (row), selecting the 0 element and scratching the column and the row where the 0 element exists; after finding a processing meeting the condition, entering the next step; if no one is satisfied, skipping the step and entering the next step;

step (5) looks for "individual 0 elements", if there are multiple individual 0 elements in the same column (row), then the following a-c substeps are performed:

a. finding respective minor elements in the row (column) where the minor elements are located;

b. selecting the single 0 element of the row (column) with the largest next smallest element, and scratching out the row and the column where the 0 element is positioned;

c. subtracting the next smaller element of its own row (column) from the row (column) to which the next smaller element was just compared;

entering the next step after the treatment; skipping this step if no condition is found to be satisfied;

step (6) if the column (row) has a single 0 element, and the other 0 elements on the row (column) are not the single 0 element on the column (row), selecting the 0 element, scratching out the row and the column where the 0 element is located, and entering the next step; skipping this step if no condition is found to be satisfied;

step (7), if no single 0 element exists on all the rows (columns), any one 0 in the row (column) with the least 0 element is arbitrarily selected, if the rows (columns) have the same condition, the comparison of the number of the 0 elements in the row (column) is carried out, the least 0 element is selected, and the next step is carried out; skipping this step if no condition is found to be satisfied;

step (8) repeating the steps (3) to (7) until all selected 0 elements are found;

the columns (rows) appearing in the step are considered columns firstly and then considered rows; the rows (columns) appear as rows considered first and columns considered second.

The invention has the beneficial effects that: the cloud data center resource allocation method based on the packet cluster framework simultaneously meets the allocation speed and the energy consumption. Under the condition that the number of virtual machines and physical machines is large, the data center can still complete allocation quickly. In terms of energy consumption, the CPU utilization rate is always kept at a more average level so as to achieve the aim of reducing the energy consumption.

Detailed Description

The invention innovatively allocates the resources of the cloud data center from three aspects. Firstly, sharing of resources is achieved, and according to the packet cluster structure provided by the invention, mutual sharing of virtual machine resources can be achieved more easily. When part of the virtual machine resources are idle, the part of the computing resources can be released to other virtual machines needing the resources, so that the wider package resource sharing is realized.

And secondly, multi-objective optimization of the data center. Through the packet cluster framework, metering and monitoring of the server are clearer, and modeling and optimization of targets such as cost, energy consumption and resource utilization rate of cloud computing are simpler.

And the third step is the fine management of the virtual machine and the server: the virtual machine placement strategy is converted from the flat mapping relation between the original virtual machine and the original physical machine into the mapping relation between the package and the cluster, the mapping problem between the virtual machine and the physical machine is further degraded in scale, the distribution efficiency and the distribution speed of the virtual machine are improved, and the virtual machine and the physical machine can be more easily and finely managed.

First, the concept of a package cluster framework is introduced. In the packet cluster mapping framework, a packet is an abstract expression of a user on the resource requirement of a virtual machine, is a resource-shared virtual machine group, not only represents a cluster bottom layer required by a bottom virtual machine, but also can be a recursive definition of a high-level packet composed of sub-packets. Likewise, a cluster is an abstract representation of server resources, representing a combination of clusters of servers, with a recursive definition of higher-level clusters of sub-clusters.

With the above overview, we can further define the package cluster framework. Assuming that there is an arbitrary packet γ in the hierarchical structure of the packet, there may be M virtual machines or sub-packets v under the packet γ, where the number of virtual machines or sub-packets is limited to 1. ltoreq. v. ltoreq.M. When a package cluster frame is allocated, a package gamma at the same level is allocated to a cluster rho at the same level, and N servers or sub-clusters p are assumed to be arranged under the cluster rho, wherein p is more than or equal to 1 and less than or equal to N. The invention needs to solve the following problems: the M sub-packets are distributed to the N sub-clusters through an optimal method on the premise of minimizing energy consumption, and the sub-packets (or virtual machines) of the sub-packets are guaranteed to be mapped to the sub-clusters (or physical machines) of the sub-clusters.

The frame relationship of the package cluster map can be formulated as follows:

in formula (1), x_v,p[t]Representing whether or not sub-packet v is allocated to sub-cluster p in time t. x is the number of_v,p[t]Has a value of only 0 or 1. When sub-packet v is allocated to sub-cluster p, then x_v,p[t]1, otherwise x_v,p[t]＝0。

It is required here that:

equation (2) represents that, at any time t, each sub-packet must be allocated to only one sub-cluster, but one sub-cluster can carry multiple sub-packets, which is to ensure that all required packets can obtain corresponding resources.

Further, assume that there are J resources: 1,2, 3. This resource represents the amount of resource required for each sub-packet, each entry representing a type, as does the corresponding sub-cluster.

With the packet cluster mapping relationship, some constraints need to be imposed on the mapping relationship, that is, the total amount of available resources of any sub-cluster must be greater than the total amount of required sub-packets allocated to the sub-cluster:

in the formula (3), R_v,i[t]Represents the total amount of demand of each sub-packet v for resource i at time t, where i is 1,2, 3. A. the_p,i[t]Then represents the total amount of resources i available to each sub-cluster P at time t, where i is 1,2, 3.

A cloud data center resource allocation method based on a packet cluster frame is obtained by improving a Hungarian algorithm and comprises two parts, wherein the first part is used for calculating a cost function of the Hungarian algorithm, and the second part is used for realizing the allocation of diversified packet clusters through a fast reduced-order algorithm.

The specific method for calculating the first part of cost function is as follows: in the hungarian algorithm, the cost of each resource to allocate objects is different. The cost function is mainly used as a target value optimization direction, the cost for calculating that one packet is placed in one cluster can be diversified in a packet cluster frame, the energy consumption is taken as an example for optimization, so that the cost function is actually the calculation of the energy consumption cost, and the lower the cost of one packet for one cluster is, the lower the energy consumption of the packet for the cluster is; on the other hand, the cost function in the package cluster frame should also contain whether the current cluster can be allocated, and if the cluster can not be allocated, the current cluster does not need to participate in the allocation.

The main idea of the second part of fast order reduction algorithm is as follows: (1) firstly, one cluster can be distributed with a plurality of packets, and the condition of different packet cluster numbers is solved through reducing the order; (2) when the 0 element in the hungarian algorithm is an independent 0 element (hereinafter, referred to as a 'dual-unique 0 element') in rows and columns at the same time, it is shown that the position is the optimal solution; (3) if there is no dual-independent 0 element, the larger the minimum value of the row or column in which the independent 0 element (hereinafter referred to as "independent 0 element") is located, other than the 0 element, the better the solution of the 0 element is; (4) after selecting the independent 0 element, to reduce the number of iterations, all elements of the unassigned row or column may be subtracted using the minimum value just except for the 0 element, thereby directly constructing a new 0 element.

The method is mainly divided into two parts, the first part is the calculation of the cost function of the Hungarian algorithm, and the second part is the distribution of the diversified packet clusters through the fast order-reducing algorithm.

In the first part, a basic cost function is first given, equation (5):

U_v,p＝U_base+Cost_v,p+y_p (5)

in the formula (5), U_v,pRepresents the cost, U, of sub-packet v being assigned to sub-cluster p_baseRepresents the existing base Cost of the sub-cluster p, the base Cost includes the sum of the sub-packets Cost allocated to the sub-cluster, as shown in equation (6), Cost_v,pRepresents the energy consumption value, y, of the current sub-packet v assigned to the sub-cluster p_pWhether the sub-cluster p can meet the requirements of the current packet or not is represented, if the sub-cluster p cannot meet the requirements, the sub-cluster p is set to be infinite, and infinite cost values cannot participate in allocation in the Hungarian algorithm.

There is also a need for Cost_v,pTo illustrate, since the energy consumption value is a real-time generated value, the energy consumption value cannot be calculated before the packet cluster is allocated, but the cost function is the energy consumption as the optimization target, and the energy consumption cost value is based on the usage rate, as shown in the following formula:

r in the formula (8)_v,cpu/A_p,cpu、R_v,men/A_p,men、R_v,io/A_p,ioThe three represent the occupation rates of the sub-packet v to the CPU, the memory and the disk of the sub-cluster p respectively, so that the advantage of more evenly distributing the sub-packet to the sub-cluster is achieved, the resource utilization rate is guaranteed to be maintained in a stable state, and the energy consumption is reduced.

And the second part is that the packet cluster allocation is completed by utilizing the fast reduced-order Hungarian algorithm, compared with the original Hungarian algorithm, the improved method can reduce the iteration times of the Hungarian algorithm and more quickly complete the placement of the virtual machine. Preparation work:

and (3) calculating the cost value of each sub-packet to each sub-cluster according to the formula (5), and forming a cost matrix by using the number of the sub-clusters needing to be distributed as the number of rows and the number of the sub-packets as the number of columns.

The method comprises the following specific steps:

and (2) if the last matrix in the divided matrixes does not meet the condition that the number of rows and the number of columns are equal, complementing the number of rows or the number of columns by using the multiplied value to form a matrix with the form that the number of rows and the number of columns are completely equal.

for example, in the following cost matrix, the 0 element in the 1 st row and the 2 nd column is both the single 0 element in the 1 st row and the single 0 element in the 2 nd column, which is the "dual single 0 element", the 1 st row and the 2 nd column need to be cut out, as shown in the following matrix,

the columns (rows) appearing in the step are considered columns first and then rows.

description of the drawings: the individual 0 elements are selected according to the respective next-smallest element (0 is already the smallest value, so the next-smallest element is selected), the selection mode is the largest value of the next-smallest element, and after selection, the row (column) which is not selected needs to subtract the smallest value of the own row (column).

Description matrix 1 selection: the independent 0 element of row 5, column 1 is selected because row 3, column 1 has the next smallest value of 2, row 5, column 1 has the next smallest value of 3, and column 3 has the largest value, so row 5, column 1 is selected.

for example, in the following cost matrix, there is no single 0 element, that is, the row with the least 0 element can be arbitrarily selected, in the following matrix, the 1 st row (2) and the 5 th row (2), and the same happens when the 0 elements of the columns are further compared, it can be seen that the 0 element of the first column is the least (2), so the 0 element of the 1 st column of the 3 rd row or the 0 element of the 1 st column of the 5 th row is arbitrarily selected;

and (8) repeating the steps (3) to (7) until all the selected 0 elements are found.

Since it is often encountered that there are more packets than clusters in the packet cluster mapping framework (i.e. the number of virtual machines is always more than equal to the number of physical machines), it is assumed that 7 packets are allocated to 5 clusters and it can be ensured that the resources on the 5 clusters are sufficient to satisfy the resources required by the 7 packets (necessary conditions for allocating upper layer clusters), and the cost matrix between them is shown in table 1 (the cost matrix for allocating 7 packets to 5 clusters):

TABLE 1

Cost of	Bag 1	Bag 2	Bag 3	Bag 4	Bag 5	Bag 6	Bag 7
								Cluster A	12	7	9	7	9	3	9
Cluster B	8	9	6	6	6	13	12
								Cluster C	7	17	12	14	9	16	10
Cluster D	15	14	6	6	10	4	5
								Cluster E	4	10	7	10	9	7	13

First, convert to a cost matrix:

and fill in the missing part of the second matrix with x elements:

and then respectively carrying out improved Hungarian algorithm solving on the two matrixes:

matrix 1:

the underline in the matrix after step (5) represents that the row has undergone the c substep of step (5), and the next smallest element of the row is subtracted to generate a new 0 element, and two steps (6) occur because step (8) repeats the operation, does not satisfy the conditions of steps (4) and (5), and directly proceeds to step (6);

matrix 2:

the final allocation scheme is therefore: the first scheme is as follows: packet 1 is assigned to cluster 5, packet 2 is assigned to cluster 1, and packet 3 is assigned to cluster 2; packet 4 is assigned to cluster 4; packet 5 is assigned to cluster 3; packet 6 is assigned to cluster 1; packet 7 is assigned to cluster 4; scheme II: packet 1 is assigned to cluster 5, packet 2 is assigned to cluster 1, and packet 3 is assigned to cluster 4; packet 4 is allocated to cluster 2; packet 5 is assigned to cluster 3; packet 6 is assigned to cluster 1; packet 7 is assigned to cluster 4.

Claims

1. A cloud data center resource allocation method based on a packet cluster frame specifically comprises the following steps:

The package cluster relationship requires 1:

the fast order reduction algorithm comprises the following steps: (1) firstly, one cluster can be distributed with a plurality of packets, and the condition of different packet cluster numbers is solved through reducing the order; (2) when the 0 element in the Hungarian algorithm is an independent 0 element in rows and columns at the same time, the element is called as a 'double-independent 0 element', and the position is an optimal solution; (3) if no independent 0 element exists, the smaller the minimum value of the row or column of the independent 0 element except for the 0 element is, the better the solution of the 0 element is; (4) after the independent 0 element is selected, in order to reduce the iteration number, subtracting all elements of the unassigned row or column by using the minimum value except the 0 element, and further directly constructing a new 0 element;

calculating a cost function of the Hungarian algorithm in the step 2):

the cost function is as in equation (5):

U_v,p＝U_base+Cost_v,p+y_p (5)

r in the formula (8)_v,cpu/A_p,cpu、R_v,men/A_p,men、R_v,io/A_p,ioThe three represent the occupation rates of the sub-packet v to the CPU, the memory and the disk of the sub-cluster p respectively, and the sub-packet is distributed to the sub-cluster more evenly so as to ensure that the resource utilization rate is maintained in a stable state;

the method is characterized in that the specific steps of realizing the distribution of the diversified packet clusters through the rapid order reduction algorithm are as follows: