CN105739929A - Data center selection method for big data to migrate to cloud - Google Patents

Abstract

The invention puts forward a data center selection method for big data to migrate to cloud. The data center selection method comprises the following steps: firstly, considering a situation that factors including user preference and legal restriction cause that a DC (Data Center) is not available to carry out non-complete graph modeling; adopting a level activation way to describe the data generation of a user; defining four criterions including FDP (Fair Data Placement), PDP (Preference Data Placement), TCMDP (Transmission cost Minimum Data Placement) and CMDP (Cost Minimum Data Placement); and on the basis of the criterions, carrying out DC selection. The method which is put forward by the invention aims at requirements generated when BD (Big Data) moves to the cloud, researches a movement mechanism from an aspect of the user, and can shorten data access time delay and reduce data cost. The method can reflect the availability of the DC and the user preference. The method can utilize a network to automatically carry out low-cost and low-delay data migration, avoids a hardware way and is favorable for implementing automatic management.

Description

The system of selection of data center when big data migrate to high in the clouds

Technical field

The present invention relates to field of cloud computer technology, particularly relate to the system of selection of data center when a kind of big data migrate to high in the clouds.

Background technology

Cloud computing has had become as the preferred platform that big data (BD) are analyzed.Produce from the place that multiple cross-regions are distributed when data especially, and local user needs often to use local data, and when data need again to integrate to be further analyzed further sometimes, especially true.Such as, one being had to the transnational sales company of the subsidiary much spread all over the world, the subsidiary of each country needs the data in time native country user produced to be analyzed for commercial object.All of data are aggregated analysis again to offer general headquarters, or support Foreign Transactions.In general, a large-scale cloud carries out networking in a distributed manner and has data center's (DC, such as Amazon have at least 11 DC, Google spreading all over 4 continents to have at least 13 DC spreading all over 4 continents) of multiple cross-region distribution.The mode of each DC pay-for-use is configured with calculating and storage resource.This infrastructure can provide and service nearby, is particularly suitable for cross-region distribution.

In order to process BD in cloud, precondition is to be migrated by BD and store on suitable DC.Direct mobile hardware is the optional mode of one of mobile large-scale data.Such as, AmazonImport/Export service recommendation movable memory equipment transports data.Sometimes, it could even be possible to move whole machine.But this is suitable only for intermittent, or disposable high-volume data move.This mode has very big delay, it is impossible to meets ever-increasing data and analyzes demand in real time.And it also contradicts with automatic management philosophy, and need more to become more and more expensive labour force's participation.Data are passed on the net much more expensive at Inter, and impracticable because of too big delay.According to Amazon data, substantially need 13 day time by the data of 10MBInter net transmission 1TB.Real time data is generally proposed to connect by high-speed dedicated and transmits (such as AWSdirectconnect).This mode can accelerate transmission speed.Even if but depend on high-speed dedicated and connect, carry out data transmission still very difficult across continent.Such as, AWSdirectconnect does not provide the service across continent.And IPLC is too expensive.Which limits and the large-scale data usually extending over the whole world is moved on a DC.And, store data with a DC and can cause that more frequent local data analysis delay is bigger.

Especially in some regions, data safety legislation requires that some data must be stored in this locality (some countries such as European Union).Sum it up, user is necessary that following some rules to select suitable storage location for their data.Advising just as Amazon: closer to the user to reduce data use delay, meet the rule requirement of specific law, or reduce cost etc..

Currently, some are based on the framework of MapReduce, such as G-Hadoop and G-MR, have been capable of the data analysis across cluster and DC.Only compare by the mechanism of a DC, use the mechanism of multiple DC can not only meet the demand of comprehensive analysis, and can guarantee that data use and have lower cost faster.

BD is moved to the select permeability of multiple DC during high in the clouds and facility select permeability (facilitylocationproblem, FLP) and k-intermediate point problem is correlated with.FLP is intended to select facility to carry out services client based on different criterions.DC can be seen as facility, and namely local data user is client.K-intermediate point problem attempts to find no more than k point, and all the other do not have selected point will be assigned to a selected point so that these between the length of side and minimum.

In the mutation of FLP problem, k-supplier problem needs to select at most k supplier (corresponding DC) to make the ultimate range between each client and supplier in his nearest minimum from given set.General, supplier and customer network are modeled as the k-supplier problem mutation for a broad sense of the complete graph, and each supplier is endowed weights, it is desirable to all selected suppliers weights are not more than k.But, it being limited to regulation, some DC may not be used for servicing some data, so figure not always complete graph.And, data are with user-dependent, rather than are correlated with DC (supplier).

Without another mutation that the facility select permeability (Uncapacitatedfacilitylocation, UFL) of capacity limit is FLP, wherein facility does not have a capacity limit, and each facility has one fixing to offer cost weight.Problem target is the total fixed cost of minimization and total cost of serving.Algorithm obtained at present has the promise breaking factor of a constant.Namely this means facility number that algorithm needs certain multiple no less than k.And k-intermediate point problem is left out the weight of facility, but the movement of data is relevant to the size of data.

Summary of the invention

In order to solve the problems of the prior art, the present invention proposes the system of selection of data center when a kind of big data migrate to high in the clouds, it is achieved low cost when BD moves to high in the clouds in distributed cloud computing, the data access object of two-forty.

The present invention is achieved through the following technical solutions:

The system of selection of data center when a kind of big data migrate to high in the clouds, described method includes: build bottom noncomplete graph, activation level is adopted to describe the data generation amount of user otherwise, the fair data of definition place FDP, preference data places PDP, transmission cost minimizes data placement TCMDP and cost minimization data place four kinds of criterions such as CMDP, and carry out the selection of DC based on one of above-mentioned criterion；Wherein, described noncomplete graph G=(U, V, E), U represents user, and V represents DC, length of side e_ij∈ E (i ∈ U, j ∈ V) meets triangle inequality, positive integer k (k≤| U |, k≤| V |), for any i ∈ U and j ∈ V, if the data of user i can be moved to DCj, then there is a limit between them；Described method is intended to find a DC subset D (| D |≤k) to come according to the data of all users in different criterions storage U from available DC set V；Described FDP criterion is: the distance minimization between maximum user and the DC being assigned to so that each local user can access data with minimum time delay:Described PDP criterion is: the Weighted distance minimization between maximum user and the DC being assigned to so that the local user with more data can access data with minimum time delay:Described TCMDP criterion is: the Weighted distance between all users and its DC being assigned to and minimization:Described CMDP) criterion is: the cost sum minimization of all users:

The invention has the beneficial effects as follows: the demand when method that the present invention proposes moves to high in the clouds for BD, have studied mobile mechanism from user perspective, it is possible to shorten data access time delay, reduce data cost.The present invention have studied four kinds of criterions: fair data place FDP, preference data places PDP, transmission cost minimizes data and places TCMDP and cost minimization data placement CMDP, the method of the present invention can reflect the availability of DC and the preference of user, for first two criterion, algorithm ensure that the solution found at least optimal solution no worse than 3 times.The method of the present invention can utilize network automatically to carry out low cost, the Data Migration of low latency, it is to avoid adopts hardware mode, is conducive to the enforcement of automated management.

Accompanying drawing explanation

Fig. 1 is the non-fully bipartite graph of the distributed user data of the present invention and data center；

Fig. 2 is the schematic diagram of thresholding figure.

Detailed description of the invention

In order to make the purpose of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein is only in order to explain the present invention, is not intended to limit the present invention.

The present invention considers the big data that cross-region is distributed are moved to the feasibility in high in the clouds, have studied multiple target DC select permeability, represents bottom facilities with non-fully bipartite graph, thus overcoming problematic is all the limitation of Complete Bipartite Graph.More conform in practical situation due to user preference, or legal restrictions and cause be not each DC can situation.

Accompanying drawing 1 simulates the non-fully bipartite graph constituted of the DC in distributed user data and distributed cloud computing, and wherein user has preferred or limited by safety legislation, is not that each DC can be selected by each user.Have limit connect represent this DC can with or user do not repel.

The present invention is purpose is seek data access and lower cost faster.This problem has promoted traditional k-supplier problem, UFL and k-intermediate point problem.

Considering bottom noncomplete graph G=(U, V, E), wherein U represents user, and V represents DC, length of side e_ij∈ E (i ∈ U, j ∈ V) meet triangle inequality, positive integer k (k≤| U |, k≤| V |), finds a DC subset D (| D |≤k) to come according to the data of all users in different criterion storage U it is contemplated that gather from available DC V.For any i ∈ U and j ∈ V, if the data of user i can be moved to DCj (at least do not limited by regulation or do not got rid of by user), then there is a limit between them.Assuming that all of i closes at least one j, otherwise problem is without solution.Assuming that | E |=m, wherein m≤| U | * | V |.

User's weight definition: each user is endowed a weight w_i, represent the activation rank that the data in current or visible future produce or the significance level of local user.w_iIncrease along with the increase of data volume or significance level.Can tolerate that the dynamic change of data provides the appropriateness to data volume to be similar to simultaneously by activation rank rather than data volume.Activate rank can according to every day the amount of uploading data and determine.Such as a company uploading 200GB typical every day, 10GB can be used as the judgement thresholding activating rank.If the data of subsidiary's generation every day are less than 10GB, it is possible to give weights 1.For the subsidiary between 20-30GB, then give weights 3, by that analogy.For user, there is activation rank w_iUser i, move data over into DCj by needs payment expense w_ie_ij。

DC weight definition: each DC has different calculating and storage resource price.For more economical storage and process data, certain preferred less price.Given DCj, it is assumed that the price that VM example processes data per hour is a_j.On average, this example can analyze b per hour_jGB data.The price then processing 10GB data is p'_j=10/b_j*a_j.If the storage charges of 10GB data is p "_j, then, it is p for having the user activating rank 1 total cost in DC side_j=p'_j+p”_j.For having activation rank w_iUser i, if it want DCj storage and process data, it need pay w_ip_j。

There is activation rank w_iThe total cost of user be w_i(p_j+e_ij.Consider e in actual environment_ij(such as several thousand kilometers) and p_jDifference in the number of levels of (several per hour U.S. dollars of Amazon), we adopt the form after standardization: c_ij=w_i(p_j'+e_ij'), wherein p_j'=p_j/max_h∈V(p_h), e_ij'=e_ij/max_l∈U,h∈V(e_lh).Note the length of side e after standardization_ij' still meet triangle inequality.

Then target specifically can be expressed as:

A) fair data place (FDP).Distance minimization between maximum user and the DC being assigned to so that each local user can access data with minimum time delay:

B) preference data places (PDP).Weighted distance minimization between maximum user and the DC being assigned to so that the local user with more data can access data with minimum time delay:If it is required, we are also with w (i, j)=w_ie_ijRepresent Weighted distance.

C) transmission cost minimizes data and places (TCMDP).Transmission cost, the Weighted distance being defined as between all users and its DC being assigned to and minimization:

D) cost minimization (CMDP).Totle drilling cost, is defined as the cost sum minimization of all users:

Because a) and c) being specific form b) and d) respectively, so follow-up provides b) and algorithm d).

Preference data places the algorithm basic thought of (PDP)

First introduce several concept, be used for describing algorithm.

1) bottleneck figure: notice that the optimal solution one of PDP problem fixes on some user's weighting limit and reaches, so we should check weighting limit from small to large one by one, until all of constraint is all satisfied.The structure of bottleneck figure is based on this with thought.Sorted according to non-decreasing order in m bar user's weighting limit, and be denoted as w (1, j)≤w (2, g)≤...≤w (m, h), wherein j, g, h ∈ V and be likely to identical.Bottleneck figure G₁,G₂,…,G_mIt is the edge subgraph of G, and G_r=(U, V, E_r) (r=1,2 ..., m), wherein E_r={ e_ij|w_ie_ij≤w(r,g)}.That is, G_rBy the summit of all G, (r, limit g) forms with being not more than r weighting limit w the shortest.

2) thresholding figure: for each G_r, the thresholding figure T of its correspondence_rGather on U constructed as below user.To each two point u, v ∈ U, (u, v) at T on limit_rIn, and if only if exist a DCj ∈ V and (u, j) and (v, j) at G_rIn.Such as, in fig. 2, limit (5,6) are absent from, because at G_mIn, user 5,6 does not have adjacent public DC.

3) Maximum Clique: the group of given non-directed graph H, H is a complete subgraph.If a group is not included in other group, then it is referred to as Maximum Clique, is denoted as C (H).It is easy in polynomial time, find C (H).The simple method used herein is as follows, first any one point of H is added C (H), is subsequently adding its neighbours' point, and these neighbours point and the institute in C (H) a little can constitute a complete graph.Institute's neighbours' point a little is all checked one by one it is known that do not put and can not add in C (H).Thus have found a Maximum Clique.Now, delete the institute of this Maximum Clique a little, and in left point, repeat this process, until H becomes sky.So, just have found a series of Maximum Clique, and each Maximum Clique is without friendship.

Noticing, if user can be serviced by same DC, then corresponding thresholding figure is a group.Such as, in fig. 1, user 6-9 can be serviced by common DC, and therefore it is at T_mIn corresponding thresholding figure be exactly the group comprising 6-9 in accompanying drawing 2.Finding Maximum Clique in thresholding figure to mean to be grouped user according to the availability of DC, the user in each of which group can represent with one of them member.

4) weight limit priority algorithm (BWF)

Input G=(U, V, E): user's weighting non-fully bipartite graph；K: positive integer

Output D: selected DC set

User's weighting limit is pressed the sequence of non-decreasing order as follows: w (1, j)≤w (2, g)≤...≤w (m, h)

To each G_rBuild corresponding thresholding figure T_r.To each T_rLook for its Maximum Clique set.Suppose there is H group.Note is corresponding to T_rMaximum Clique set be C_r=C (T_r)_h, wherein r=1,2 ..., m, h=1 ..., H

Make t=min{r | | C_r|≤k}

For{h=1 ..., H}

{

L is made to have weight limit w_l=max{w_j|j∈C(T_t)_h, j is not assigned a little }.Make C (T_t)_hNeighbours DC be

E=min_j∈N{e_lj|e_lj∈G_t}.Making v is the point in N, and e_lv=e.D=D ∪ v

}

May certify that, BWF algorithm gives a 3-approximate solution 3 times of value of optimal solution (value of the solution namely obtained be not more than) of PDP problem, and wherein the bipartite graph right and wrong of bottom are completely.And this solution is tight (namely can not have the approximation ratio less than 3).

When bottom figure is Complete Bipartite Graph, it is possible to replace Maximum Clique by independent sets, algorithm faster, solves identical in quality.Algorithm BWF attempts the data of maximum-norm are placed into nearest DC.It checks the bottleneck figure of correspondence one by one according to user's weighting limit order from small to large, and constructs the thresholding figure of correspondence.Then the agglomeration finding each thresholding figure closes.Each group represents one group of user that can be at least serviced by an identical DC, and the weighting limit between user and common DC is not more than the maximum weighted limit of bottleneck figure.The agglomeration conjunction subscript that the number that all agglomerations are rolled into a ball in closing is not more than k is selected, so that maximum user's weighting limit is little as much as possible.It is of course possible to further speed up speed with binary search.

At the agglomeration of thresholding figure, C is closed for each_tIn user, in accordance with the following methods find target DC.In each group, from the user with weight limit, we assign to G it_tIn nearest neighbours DC.All the other users in all same groups are also assigned to this DC by implicit expression.This process repeats, until all C_tIn user be all assigned (for circulation).So, algorithm establishes a mapping between user and DC.Mapping and bipartite graph is divided into multiple cluster, the center of each cluster is a DC.

Cost minimization data place (CMDP) algorithm basic thought

To CMDP problem, The present invention gives NPA-CMDP algorithm, it attempts the DC that each user assigns to recently (i.e. cost minimization between user and DC).If the number of DC has exceeded k, then certain DC will be removed, and corresponding user can be assigned again.This process repeats until constraint satisfaction.

Algorithm 2.CMDP best-first algorithm (NPA-CMDP).

Input G=(U, V, E): user's weighting non-fully bipartite graph；K: positive integer

Output D: selected DC set

Each user assigns to nearest DC, and (Weighted distance (cost) between user and DC is defined as w_ie_ij+w_ip_j).The DC of selection is gathered and is designated as D

While{|D|>k}

{

Find DC at D, compare with all the other DC in D, between this DC and the user assigning to it from weighting from being minimum.This DC is deleted from D.

And the user originally assigning to this DC is assigned to again remaining nearest DC in D

}

The present invention considers the big data that cross-region is distributed are moved to the feasibility in high in the clouds, have studied multiple target DC select permeability, represents bottom facilities with non-fully bipartite graph, thus overcoming problematic is all the limitation of Complete Bipartite Graph.More conform in practical situation due to user preference, or legal restrictions and cause be not each DC can situation.Have studied four kinds of criterions, including.Fair data place (FDP), preference data places (PDP), transmission cost minimizes data and places (TCMDP) and cost minimization data placement (CMDP), in practical application, it is possible to select according to the demand of operator and user.

Above content is in conjunction with concrete preferred implementation further description made for the present invention, it is impossible to assert that specific embodiment of the invention is confined to these explanations.For general technical staff of the technical field of the invention, without departing from the inventive concept of the premise, it is also possible to make some simple deduction or replace, protection scope of the present invention all should be considered as belonging to.

Claims (7)

1. the system of selection of data center when big data migrate to high in the clouds, it is characterised in that described method includes:

Build bottom noncomplete graph, adopt and activate rank w_iMode the data generation amount of user is described, the fair data of definition place FDP, preference data places PDP, transmission cost minimizes data placement TCMDP and cost minimization data place four kinds of criterions such as CMDP, and carry out the selection of DC based on one of above-mentioned criterion；

Wherein, described noncomplete graph G=(U, V, E), U represents user, and V represents DC, length of side e_ij∈ E (i ∈ U, j ∈ V) meets triangle inequality, and positive integer k (k≤| U |, k≤| V |), for any i ∈ U and j ∈ V, if the data of user i can be moved to DC_j, then there is a limit between them；Described method is intended to find a DC subset D (| D |≤k) to come according to the data of all users in different criterions storage U from available DC set V；

Described FDP criterion is: the distance minimization between maximum user and the DC being assigned to so that each local user can access data with minimum time delay:

Described PDP criterion is: the Weighted distance minimization between maximum user and the DC being assigned to so that the local user with more data can access data with minimum time delay: ${\min }_{D\⊆V,\left|D\right|\≤k}{\max }_{i\∈U,j\∈D}\left(w_i{e}_{ij}\right);$

Described TCMDP criterion is: the Weighted distance between all users and its DC being assigned to and minimization: ${\min }_{D\⊆V,\left|D\right|\≤k}\left({\mathrm{\Σ}}_{i\∈U,j\∈D}{w}_i{e}_{ij}\right);$

Described CMDP criterion is: the cost sum minimization of all users:c_ijFor having activation rank w_iThe total cost of user.

2. method according to claim 1, it is characterised in that: described noncomplete graph is weighting noncomplete graph, and the length of side is w_ie_ij, wherein w_iFor user, there is activation rank, w_ie_ijFor moving data over into DC_jBy needs payment expense.

3. method according to claim 1, it is characterised in that: described activation rank w_iDetermine according to the load data volume uploaded every day.

4. method according to claim 1, it is characterised in that: for having activation rank w_iUser i, if it is want at DC_jStorage and process data, it needs to pay w_ip_j, wherein, p_j=p'_j+p”_j, p'_jThe price of unit capacity data, p is processed " for DC_jStorage charges for DC storage cell capacity data is used.

5. method according to claim 1, it is characterised in that: the selection carrying out DC based on PDP criterion adopts the preferential BWF algorithm of weight limit, and described BWF algorithm comprises the steps:

User's weighting limit is pressed non-decreasing order sequence, be denoted as w (1, j)≤w (2, g)≤...≤w (m, h), wherein j, g, h ∈ V, bottleneck figure G₁,G₂,…,G_mIt is the edge subgraph of G, and G_r=(U, V, E_r) (r=1,2 ..., m), E_r={ e_ij|w_ie_ij≤w(r,g)}；

To each G_rBuild corresponding thresholding figure T_r, to each two point u, v ∈ U, (u, v) at T on limit_rIn, and if only if exist a DCj ∈ V and (u, j) and (v, j) at G_rIn；

To each T_rLook for its Maximum Clique set, if a group is not included in other group, be then referred to as Maximum Clique；Suppose there is H group, note is corresponding to T_rMaximum Clique set be C_r=C (T_r)_h, wherein r=1,2 ..., m, h=1 ..., H, make t=min{r | | C_r|≤k}

For (h=1 ..., H)

{ l is made to have weight limit w_l=max{w_j|j∈C(T_t)_h, j is not assigned a little }；

Make C (T_t)_hNeighbours DC beE=min_j∈N{e_lj|e_lj∈G_t}；

Making v is the point in N, and e_lv=e.D=D ∪ v}.

6. method according to claim 5, it is characterised in that: when bottom figure is Complete Bipartite Graph, replace Maximum Clique by independent sets, accelerate BWF algorithm.

7. method according to claim 1, it is characterised in that: the selection carrying out DC based on CMDP criterion adopts CMDP preferential NPA-CMDP algorithm recently, and described NPA-CMDP algorithm comprises the steps:

Each user is assigned to nearest DC, and the Weighted distance between user and DC is defined as w_ie_ij+w_ip_j, the DC of selection is gathered and is designated as D；

While{|D|>k}

Find DC at D, compare with all the other DC in D, and between this DC and the user assigning to it from weighting from being minimum；

This DC is deleted from D；

And the user originally assigning to this DC is assigned to again remaining nearest DC} in D.