CN105739929A - Data center selection method for big data to migrate to cloud - Google Patents

Abstract

The present invention proposes a data center selection method when big data is migrated to the cloud. First, considering the unavailability of DC due to factors such as user preferences and legal restrictions, an incomplete graph modeling is carried out; the activation level is adopted To describe the user's data generation; four criteria are defined: fair data placement FDP, preferred data placement PDP, transmission cost minimization data placement TCMDP, and cost minimization data placement CMDP; DC selection is based on the above criteria. The method proposed by the present invention aims at the requirement when BD moves to the cloud, studies the moving mechanism from the user's point of view, can shorten the data access delay, and reduce the data cost. The method of the present invention can reflect the availability of DCs as well as user preferences. The method of the invention can use the network to automatically perform low-cost and low-delay data migration, avoiding the use of hardware, and is beneficial to the implementation of automatic management.

Description

How to choose a data center when migrating big data to the cloud

technical field

The invention relates to the technical field of cloud computing, in particular to a method for selecting a data center when big data is migrated to the cloud.

Background technique

Cloud computing has become the platform of choice for big data (BD) analysis. This is especially true when data is generated from multiple geographically distributed locations, and local users often need to use local data, and sometimes the data needs to be further integrated for further analysis. For example, for a multinational sales company with many subsidiaries all over the world, the subsidiaries in each country need to analyze the data generated by local users in a timely manner for business purposes. All data must be aggregated and analyzed to report to the headquarters, or to support cross-border transactions. Generally speaking, a large cloud is networked in a distributed manner and has multiple geographically distributed data centers (DCs, for example, Amazon has at least 11 DCs spread over 4 continents, and Google has at least 13 DCs spread across 4 continents. DC). Each DC is configured with computing and storage resources on a pay-as-you-go basis. This kind of infrastructure can provide close service, especially suitable for cross-regional distribution.

In order to process BDs in the cloud, the prerequisite is to migrate and store the BDs on a suitable DC. Moving hardware directly is an optional way to move large amounts of data. For example, the Amazon Import/Export service recommends removable storage devices for shipping data. Sometimes, it is even possible to move the entire machine. But this is only suitable for intermittent, or one-time bulk data movement. This method has a large delay and cannot meet the growing demand for real-time data analysis. And it also contradicts the idea of automatic management and requires more labor participation which is becoming more and more expensive. Uploading data over the Internet is very expensive and impractical due to too much latency. According to Amazon data, it takes roughly 13 days to transmit 1TB of data through a 10MBInternet. Real-time data is usually recommended to be sent over a high-speed dedicated connection (such as AWSdirectconnect). This method can speed up the transmission speed. But even relying on high-speed dedicated connections, transferring data across continents is difficult. For example, AWSdirectconnect does not provide services across continents. The international line is too expensive. This limits the ability to move large amounts of data, often spread across the globe, to a single DC. Also, using a DC to store data results in greater latency for more often local data analysis.

Especially in some regions, data security laws require that some data must be stored locally (such as some countries in the European Union). All in all, it is necessary for users to follow some rules to choose a suitable storage location for their data. Just like Amazon suggests: closer to users to reduce data usage delays, meet specific legal requirements, or reduce costs, etc.

Currently, some MapReduce-based frameworks, such as G-Hadoop and G-MR, have been able to implement cross-cluster and DC data analysis. Compared with the mechanism using only one DC, the mechanism using multiple DCs can not only meet the needs of comprehensive analysis, but also ensure faster data usage and lower cost.

The problem of selecting multiple DCs when moving BDs to the cloud is related to the facility location problem (FLP) and the k-midpoint problem. FLP aims to select facilities to serve customers based on different criteria. DCs can be viewed as facilities, while local data users are customers. The k-intermediate point problem tries to find no more than k points, and the remaining unselected points will be assigned to a selected point such that the sum of the edge lengths between these pairs of points is minimized.

Among the variants of the FLP problem, the k-supplier problem requires selecting at most k suppliers (corresponding to DCs) from a given set such that the maximum distance between each customer and its nearest supplier is minimized. Generally, the supplier and customer network is modeled as a complete graph. For a generalized variant of the k-supplier problem, each supplier is assigned a weight, and all selected suppliers are required to have a weight no greater than k. However, due to regulations, some DCs may not be able to serve certain data, so the graph is not always a complete graph. Also, the data is related to the user, not to the DC (supplier).

Uncapacitated facility location (UFL) is another variant of FLP in which facilities have no capacity constraints and each facility has a fixed set-up cost weight. The problem objective is to minimize total fixed costs and total service costs. The resulting algorithms so far have a constant default factor. That is, this means that the number of facilities required by the algorithm is not less than a certain multiple of k. The k-intermediate problem does not consider the weight of facilities, but the movement of data is related to the size of data.

Contents of the invention

In order to solve the problems in the prior art, the present invention proposes a data center selection method when big data migrates to the cloud, and realizes the low-cost, high-speed data access target when BD moves to the cloud in distributed cloud computing .

The present invention is realized through the following technical solutions:

A method for selecting a data center when big data is migrated to the cloud, the method comprising: constructing an incomplete graph at the bottom layer, using activation levels to describe user data generation, defining fair data placement FDP, preferred data placement PDP, Four criteria such as transmission cost minimization data placement TCMDP and cost minimization data placement CMDP, and DC selection based on one of the above criteria; wherein, the incomplete graph G=(U, V, E), U represents the user , V represents DC, side length e _ij ∈ E (i ∈ U, j ∈ V) satisfies the triangle inequality, positive integer k (k ≤ | U |, k ≤ | V |), for any i ∈ U and j ∈ V , if user i's data can be moved to DCj, there is an edge between them; the method aims to find a DC subset D(|D|≤k) from the available DC set V to store according to different criteria Data of all users in U; the FDP criterion is: the distance between the largest user and the assigned DC is minimized, so that each local user can access data with the minimum delay: The PDP criterion is: the weighted distance between the largest user and the assigned DC is minimized, so that local users with more data can access data with the minimum delay: The TCMDP criterion is: the sum of weighted distances between all users and their assigned DCs is minimized: The CMDP) criterion is: the sum of the costs of all users is minimized:

The beneficial effect of the present invention is that: the method proposed by the present invention studies the mobile mechanism from the user's point of view, aiming at the requirement when the BD moves to the cloud, which can shorten the data access delay and reduce the data cost. The present invention studies four criteria: fair data placement FDP, preferred data placement PDP, transmission cost minimization data placement TCMDP and cost minimization data placement CMDP, the method of the present invention can reflect the availability of DC and user's preference, for the first two This criterion, the algorithm can guarantee that the solution found is at least not worse than 3 times the optimal solution. The method of the invention can use the network to automatically perform low-cost and low-delay data migration, avoiding the use of hardware, and is beneficial to the implementation of automatic management.

Description of drawings

Fig. 1 is the incomplete bipartite graph of distributed user data and data center of the present invention;

FIG. 2 is a schematic diagram of a threshold map.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The present invention considers the feasibility of moving cross-regional distributed big data to the cloud, studies multiple target DC selection problems, and uses an incomplete bipartite graph to represent the underlying facilities, thereby overcoming the limitation that existing problems are all complete bipartite graphs. It is more in line with the actual situation where not every DC is available due to user preference or legal restrictions.

Attached Figure 1 simulates an incomplete bipartite graph composed of distributed user data and DCs in distributed cloud computing, where users have preferences or are restricted by security laws, and not every DC can be selected by every user. A side connection indicates that the DC is available or the user does not exclude it.

The present invention seeks faster data access and lower costs. This problem generalizes the traditional k-suppliers problem, UFL and k-midpoint problem.

Consider the underlying incomplete graph G=(U, V, E), where U represents the user, V represents the DC, and the side length e _ij ∈ E (i ∈ U, j ∈ V) satisfies the triangle inequality, and the positive integer k(k≤| U|, k≤|V|), the present invention aims to find a DC subset D (|D|≤k) from the available DC set V to store the data of all users in U according to different criteria. For any i ∈ U and j ∈ V, there exists an edge between them if the data of user i can be moved to DC j (at least not restricted by regulations or excluded by users). Assume that all i are adjacent to at least one j, otherwise the problem has no solution. Assume that |E|=m, where m≤|U|*|V|.

Definition of user weight: each user is assigned a weight w _i , which represents the activation level generated by the current or visible future data, or the importance of the local user. w _i increases with the increase of data volume or importance. Using activation levels instead of data volumes can tolerate dynamic changes in data while providing a reasonable approximation of data volumes. The activation level can be based on the amount of uploaded data per day. For example, for a typical company that uploads 200GB per day, 10GB can be used as the threshold for determining the activation level. If a subsidiary produces less than 10GB of data per day, it can be given a weight of 1. For subsidiaries between 20-30GB, assign a weight of 3, and so on. For user i with activation level w _i , moving data to DCj will require payment of fee w _i e _ij .

DC weight definition: Each DC has different computing and storage resource prices. For more economical storage and processing of data, lower prices are of course preferred. Given DCj, assume that the price of a VM instance processing data per hour is a _j . On average, such an instance can analyze b _j GB of data per hour. Then the price for processing 10GB data is p' _j =10/b _j *a _j . If the storage charge for 10GB data is p" _j , then the total charge on the DC side for a user with activation level 1 is p _j = p' _j + p" _j . For user i with activation level w _i , if it wants to store and process data in DCj, it needs to pay w _i p _j .

The total cost of a user with activation level w _i is w _i (p _j + e _ij . Considering the difference in the quantitative level of e _ij (such as thousands of kilometers) and p _j (a few dollars per hour for Amazon) in the actual environment , we adopt the standardized form: c _ij =w _i (p _j '+e _ij '), where p _j '=p _j /max _h∈V (p _h ), e _ij '=e _ij /max _{l∈ U,h∈V} (e _lh ). Note that the standardized side length e _ij ' still satisfies the triangle inequality.

The objective can then be expressed as:

a) Fair Data Placement (FDP). The distance between the largest user and the assigned DC is minimized, so that each local user can access data with the minimum delay:

b) Preferred Data Placement (PDP). The weighted distance between the largest user and the assigned DC is minimized, so that local users with more data can access data with the smallest delay: We also denote the weighted distance by w(i,j)=w _i e _ij if necessary.

c) Transport Cost Minimized Data Placement (TCMDP). The transmission cost, defined as the minimization of the sum of weighted distances between all users and their assigned DCs:

d) Cost Minimization (CMDP). The total cost, defined as minimizing the sum of the costs of all users:

Since a) and c) are special forms of b) and d) respectively, only the algorithms of b) and d) are given later.

The basic idea of the algorithm of preferred data placement (PDP)

First introduce a few concepts, used to describe the algorithm.

1) Bottleneck graph: Note that the optimal solution of the PDP problem must be reached at a certain user weighted edge, so we should check the weighted edges one by one from small to large until all constraints are satisfied. The construction of the bottleneck map is based on this idea. The m user-weighted edges are sorted in non-decreasing order and denoted as w(1,j)≤w(2,g)≤...≤w(m,h), where j,g,h∈V and may be the same. Bottleneck graphs G ₁ , G ₂ ,...,G _m are edge subgraphs of G, and G _r =(U,V,E _r )(r=1,2,...,m), where E _r ={e _ij |w _i e _ij ≤w(r,g)}. That is, G _r consists of all vertices of G and an edge no larger than the r-th shortest weighted edge w(r,g).

2) Threshold graph: For each G _r , its corresponding threshold graph T _r is constructed on the user set U as follows. For every two vertices u, v ∈ U, an edge (u, v) is in T _r if and only if there exists a DCj ∈ V and both (u, j) and (v, j) are in G _r . For example, in Figure 2, edge (5, 6) does not exist because users 5, 6 have no adjacent common DC in _Gm .

3) Maximum clique: Given an undirected graph H, the clique of H is a complete subgraph. If a clique is not included in other cliques, it is called a maximal clique, denoted as C(H). It is easy to find C(H) in polynomial time. The simple method used in this article is as follows, first add any point of H to C(H), and then add its neighbor points, these neighbor points and all points already in C(H) can form a complete graph. Neighbors of all points in C(H) are checked one by one until no point can be added. In this way, a maximal cluster is found. Now, delete all points of this maximal clique and repeat this process in the remaining points until H becomes empty. In this way, a series of maximal cliques are found, and each maximal clique has no intersection.

Note that if users can be served by the same DC, the corresponding threshold graph is a clique. For example, in FIG. 1 , users 6-9 can be served by a common DC, so their corresponding threshold graph in T _m is the clique containing 6-9 in FIG. 2 . Finding maximal cliques in the threshold graph means grouping users according to the availability of DCs, where each user in a group can be represented by one of its members.

4) Maximum Weight First Algorithm (BWF)

Input G=(U,V,E): user weighted incomplete bipartite graph; k: positive integer

Output D: Selected DC set

Sort user weighted edges in non-decreasing order as follows: w(1,j)≤w(2,g)≤...≤w(m,h)

A corresponding threshold map T _{r is constructed for each G r .} For each T _r find its maximal clique set. Suppose there are H groups. Note that the maximal clique set corresponding to T _r is C _r ＝C(T _r ) _h , where r=1,2,…,m, h=1,…,H

Let t=min{r||C _r |≤k}

For{h＝1,...,H}

{

Let l be a point with maximum weight w _l =max{w _j |j∈C(T _t ) _h , j is not assigned}. Let the neighbor DC of C(T _t ) _h be

e=min _j∈N {e _lj |e _lj ∈G _t }. Let v be a point in N, and e _lv =e. D＝D∪v

}

It can be proved that the BWF algorithm gives a 3-approximate solution to the PDP problem (that is, the value of the obtained solution is not greater than 3 times the value of the optimal solution), where the underlying bipartite graph is incomplete. And this solution is compact (ie no approximation ratio smaller than 3 is possible).

When the underlying graph is a complete bipartite graph, independent sets can be used instead of maximal cliques, the algorithm is faster and the solution quality is the same. Algorithm BWF tries to place the largest amount of data to the nearest DC. It checks the corresponding bottleneck graph one by one according to the order of user weighted edges from small to large, and constructs the corresponding threshold graph. Then find the clique set for each threshold graph. Each clique represents a group of users that can be served by at least one same DC, and the weighted edge between users and the common DC is not larger than the maximum weighted edge of the bottleneck graph. The subscripts of all clique sets whose number of cliques is not greater than k are selected so that the largest user weighted edge is as small as possible. Of course, binary search can be used to further speed up.

For each user in the clique set C _t of the threshold map, find the target DC according to the following method. In each clique, starting from the user with the largest weight, we assign it to the nearest neighbor DC in _Gt . All other users in the same group are also implicitly assigned to this DC. This process repeats until all users in C _t are assigned (for loop). In this way, the algorithm establishes a mapping between users and DCs. Mapping divides the bipartite graph into clusters, with a DC at the center of each cluster.

Basic idea of cost minimization data placement (CMDP) algorithm

For the CMDP problem, the present invention provides the NPA-CMDP algorithm, which attempts to assign each user to the nearest DC (that is, the minimum cost between the user and the DC). If the number of DCs exceeds k, a certain DC will be removed and the corresponding user will be reassigned. This process is repeated until the constraints are satisfied.

Algorithm 2. CMDP Nearest First Algorithm (NPA-CMDP).

Input G=(U,V,E): user weighted incomplete bipartite graph; k: positive integer

Output D: Selected DC set

Assign each user to the nearest DC (the weighted distance (cost) between user and DC is defined as w _i e _ij +w _i p _j ). Record the selected DC set as D

While{|D|>k}

{

Find a DC in D that has the smallest distance-weighted distance from the user assigned to it compared to the rest of the DCs in D. Remove this DC from D.

And reassign the users originally assigned to this DC to the remaining nearest DC in D

}

The present invention considers the feasibility of moving cross-regional distributed big data to the cloud, studies multiple target DC selection problems, and uses an incomplete bipartite graph to represent the underlying facilities, thereby overcoming the limitation that existing problems are all complete bipartite graphs. It is more in line with the actual situation where not every DC is available due to user preference or legal restrictions. Four guidelines were studied, incl. Fair Data Placement (FDP), Preferred Data Placement (PDP), Transmission Cost Minimized Data Placement (TCMDP) and Cost Minimized Data Placement (CMDP), in practical applications, can be selected according to the needs of operators and users.

The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be assumed that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field of the present invention, without departing from the concept of the present invention, some simple deduction or replacement can be made, which should be regarded as belonging to the protection scope of the present invention.

Claims (7)

1. A method for selecting a data center when big data migrates to the cloud, characterized in that the method comprises: Construct the underlying incomplete graph, use the activation level w _i to describe the user's data generation, and define four criteria for fair data placement FDP, optimal data placement PDP, transmission cost-minimized data placement TCMDP, and cost-minimized data placement CMDP , and the selection of a DC based on one of the above criteria; Among them, the incomplete graph G=(U, V, E), U represents the user, V represents the DC, the side length e _ij ∈ E (i ∈ U, j ∈ V) satisfies the triangle inequality, and the positive integer k (k≤ |U|,k≤|V|), for any i ∈ U and j ∈ V, if the data of user i can be moved to DC _j , then there is an edge between them; the method aims at starting from the available DC Find a DC subset D(|D|≤k) in the set V to store the data of all users in U according to different criteria; The FDP criterion is: the distance between the largest user and the assigned DC is minimized, so that each local user can access data with the minimum delay: The PDP criterion is: the weighted distance between the largest user and the assigned DC is minimized, so that local users with more data can access data with the minimum delay: ${\min }_{\mathrm{D.}\⊆V,\left|\mathrm{D.}\right|\≤k}{\max }_{i\∈u,j\∈\mathrm{D.}}\left(w_i{e}_{ij}\right);$ The TCMDP criterion is: the sum of weighted distances between all users and their assigned DCs is minimized: ${\min }_{\mathrm{D.}\⊆V,\left|\mathrm{D.}\right|\≤k}\left({\mathrm{\Σ}}_{i\∈u,j\∈\mathrm{D.}}{w}_i{e}_{ij}\right);$ The CMDP criterion is: the sum of the costs of all users is minimized: c _ij is the total cost of the user with activation level w _i . 2. The method according to claim 1, characterized in that: the incomplete graph is a weighted incomplete graph, and the side length is w _i e _ij , wherein w _i is the activation level of the user, and w _i e _ij is the mobile data A fee will be required to arrive at DC _j . 3. The method according to claim 1, wherein the activation level w _i is determined according to the amount of data uploaded every day. 4. The method according to claim 1, characterized in that: for user i with activation level w _i , if it wants to store and process data in DC _j , it needs to pay w _i p _j , where p _j = p ' _j + p” _j , p' _j is the price of DC processing unit capacity data, and p” _j is the storage cost of DC storing unit capacity data. 5. The method according to claim 1, characterized in that: the selection of DC based on the PDP criterion adopts the maximum weight priority BWF algorithm, and the BWF algorithm comprises the following steps: Sort user weighted edges in non-decreasing order, record as w(1,j)≤w(2,g)≤...≤w(m,h), where j,g,h∈V, bottleneck graph G ₁ ,G ₂ ,...,G _m is an edge subgraph of G, and G _r ＝(U,V,E _r )(r=1,2,...,m), E _r ＝{e _ij |w _i e _ij ≤w (r,g)}; Construct the corresponding threshold graph T _r for each G _r , for every two points u, v∈U, the edge (u, v) is in T _r , if and only if there is a DCj∈V and (u, j) and (v, j) are in G _r ; Find its maximal clique set for each T _r , if a clique is not included in other cliques, it is called a maximal clique; suppose there are H groups, record the maximal clique set corresponding to T _r as C _r = C(T _r ) _h , where r=1,2,…,m, h=1,…,H, let t=min{r||C _r |≤k} for(h=1,...,H) {Let l be a point with maximum weight w _l =max{w _j |j∈C(T _t ) _h , j is not assigned}; Let the neighbor DC of C(T _t ) _h be e=min _j∈N {e _lj |e _lj ∈G _t }; Let v be a point in N, and e _lv =e. D=D∪v}. 6. The method according to claim 5, characterized in that: when the underlying graph is a complete bipartite graph, independent sets are used instead of maximal cliques to accelerate the BWF algorithm. 7. The method according to claim 1, characterized in that: the selection of DC based on the CMDP criterion adopts the CMDP nearest priority NPA-CMDP algorithm, and the NPA-CMDP algorithm comprises the steps of: Assign each user to the nearest DC, the weighted distance between the user and DC is defined as w _i e _ij +w _i p _j , and the selected DC set is recorded as D; While{|D|>k} {Find a DC in D. Compared with other DCs in D, the distance-weighted distance between this DC and the user assigned to it is the smallest; Delete this DC from D; And reassign the users originally assigned to this DC to the remaining nearest DC in D}.