CN106161599A - A kind of method reducing cloud storage overall overhead when there is data dependence relation - Google Patents

Abstract

The invention discloses a kind of method reducing cloud storage overall overhead when there is data dependence relation.The method comprises data storage decision model and data storage two parts of strategic decision-making algorithm.In data storage decision model, the decision variable of model is the storage strategy of data, including not storing, the storage of many copies and correcting and eleting codes storage；The constraints of model is that the availability of data needs to meet given approve-useful index；The optimization aim of model is for minimizing system overall overhead, and wherein the overall overhead of data can include storage overhead and computing cost simultaneously；In computing cost, the data genaration time is a stochastic variable, The present invention gives its desired computational methods.In data store strategy decision making algorithm, when there being new data to generate, the storage strategy of direct decision data；At the end of each decision-making period, in units of the connected component in dependence graph, whether exceed threshold value according to the nodes of connected component, the storage strategy of data in use genetic algorithm or traversal pruning algorithms update this connected component respectively.Compared with prior art, the method that the present invention provides has the advantage reducing system overall overhead on the premise of ensure that availability of data.

Description

A kind of method reducing cloud storage overall overhead when there is data dependence relation

Technical field

The present invention relates to cloud storage field, be specifically related to reduce cloud storage overall overhead when one exists data dependence relation Method.

Background technology

Along with developing rapidly of cloud storage, increasing application stores data in high in the clouds, how to reduce data center The storage overhead of mass data becomes an important problem.

In cloud storage, between data, there may be dependence.The video file of such as different definitions can pass through Former video file transcoding generates, and now exists for dependence between the file after original and transcoding.

The scheme reducing cloud storage overall overhead when being currently, there are data dependence relation is: by algorithm decision data be No storage, the now data or do not store of needing, or with the fixing many copies storage of number of copies.For the data not stored, When it receives access request, generate data first with dependence, reoffer access service.Now, the overall overhead of system Comprise the storage overhead that storage data produce and the computing cost generating data generation.

But, it is not enough that existing scheme there is also two aspects: (1), when storing data, the storage strategy of data is fixed.No Consider the situation that data store strategy is variable further.(2) sound when the generation time of data allows is not accounted for beyond user When answering time delay, the disabled problem of data.

Summary of the invention

The problem existed for prior art, present invention is primarily targeted at a kind of data of working as of offer and there is dependence Time, the method reducing system overall overhead on the premise of ensureing availability of data index.For simplifying statement, " number hereinafter According to " refer to cloud storage exists the data of dependence.Data trnascription can be to refer to a copy or the correcting and eleting codes mode of copy mode One coding fragment.The storage strategy of data refers to the storage mode of data, and including not storing, the storage of many copies and correcting and eleting codes deposit Storage.

It is a feature of the present invention that and comprise herein below:

1. data storage decision model

Existing scheme when saving data, uses many copies storage strategy that number of copies is fixing.The decision-making of storage model becomes Amount is for N number of data the need of storage, and individual data only has storage overhead or computing cost, and does not accounts for availability of data Problem.Different from existing scheme, in the present invention, the storage strategy of data is variable, it is characterised in that: 1) decision variable of model For the storage strategy of N number of data, as shown in formula (2)；2) overall overhead of data includes the storage overhead of existing copy simultaneously With the computing cost of generation data when all copies lost efficacy, its computational methods are given by formula (3)；3) object function of model For minimizing system overall overhead, formula (1a) be given；4) constraints of model be data availability need meet to Fixed approve-useful index, is given by formula (1b).

2. data genaration time desired computational methods

When calculating the overall overhead of data, wherein computing cost is relevant with the generation time of data.In the present invention, number According to generation time T_iBeing a stochastic variable, the malfunction of the data directly or indirectly relied on data is relevant.This The bright expectation E (T providing the data genaration time_i) computational methods, formula (8) be given.

3. the computational methods of availability of data

When data do not store or break down, if number can not be generated in the response time that user allows According to, data can be caused unavailable, affect the availability of data.The invention provides availability of dataComputational methods, by Formula (10) is given.

4. data store strategy decision making algorithm

The present invention gives on the premise of ensureing availability of data, reduce the data store strategy of system overall overhead certainly Plan algorithm, the execution step of algorithm is given by flow chart 3.

Accompanying drawing explanation

Fig. 1 is the storage model structure schematic diagram that the present invention proposes.

Fig. 2 is data dependence relation schematic diagram.

Fig. 3 is the overall flow figure of data store strategy decision making algorithm.

Fig. 4 is the flow chart of data store strategy renewal process in Fig. 3.

Fig. 5 is the flow chart of step S2.2.2 in Fig. 4.

Fig. 6 is the flow chart of step S2.2.3 in Fig. 4.

Detailed description of the invention

A kind of method reducing cloud storage overall overhead when there is data dependence relation of the present invention mainly includes two Individual part:

S1, data storage decision model: model mainly includes the structural model of system and the overall overhead model of system；

S2, data store strategy decision making algorithm: algorithm gives on the premise of ensureing availability of data, reduces system whole The data store strategy decision making algorithm of body expense.

Below in conjunction with the accompanying drawings, the detailed description of the invention of the present invention is elaborated.

S1, data storage decision model, mainly comprises with lower part:

S1.1, the structure of storage system

This storage system with the addition of data dependence between the data access interface layer and data storage layer of cloud storage system Relation management layer, as shown in Figure 1.When data dependence relation management level receive data access request, there is situations below:

Situation 1: if data do not exist dependence, then data access request is transmitted to data storage layer；

Situation 2: if data exist dependence and have data available copy, then be transmitted to number by data access request According to accumulation layer；In Fig. 2, data d₅For this storage of two-pack.When two copies at least one available time, system uses available pair This offer accesses service.

Situation 3: if data exist dependence and all copies the most unavailable time, first with dependence generate number According to, then forward data access request；In Fig. 2, when data d₅The memory node of two copies breaks down so that copy the most not Time available, generate data d first with dependence₅.If generating d₅During, data d₅Copy place node recovers normal, Then stop generation process, use former copy to continue to provide the service of access.Otherwise, delete former copy, use newly-generated copy to carry For accessing service.

S1.2, data dependence relation management level structure

In S1.1, the function of data dependence relation management level specifically includes that data dependence relation figure is set up and number of faults According to regenerating.

Data dependence relation figure to set up mode as follows: dependences based on data, the N number of existence in system is relied on The data modeling of relation is the topological structure of directed acyclic graph, i.e. data dependence relation figure.Node table registration evidence in figure, oriented LimitRepresent data d_iGeneration depend on data d_j.Data d_iComprise attribute:s_i, (1≤i≤N) Represent d respectively_iMemory space shared by single copy, d_iThe data set of dependence, d_iThe generation time, d_iGenerating operator and d_i's Storage strategy.Wherein, s_iValued space be 0,1,2,3,4}, and respectively represent data d_iDo not store, single copy stores, two-pack basis Storage, three copy storages and the storage of (10,14) correcting and eleting codes.

It is as follows that fault data generates process: when data d_iTime unavailable, to d_iThe data set relied onPerform to generate and calculate SonElapsed timeAfter regenerate d_i.When the data relied on be concentrated with data unavailable time, need this number of first recursive generation According to, regeneration data d_i。

S1.3, system overall overhead model

The overall overhead of system is equal to the overall overhead sum of N number of data, and the optimization aim of system is to ensure that data are full The approve-useful index A that foot is given^EWhile, reduce system overall overhead.

The optimization object function of system overall overhead:

$\min F(s_1,s_2,...,s_N)=min\underset{i=1}{\overset{N}{\Σ}}{C}_i\left(s_i\right)---\left(1a\right)$

Bound for objective function is:

$A_i^D\≥A^E,(i=1,2,...,N)---\left(1b\right)$

Wherein, F represents the overall overhead of system within decision-making period.C_iWithIt is illustrated respectively in data d in this cycle_i's Overall overhead and the availability of data, use formula (3) (10) to calculate respectively；A^EThe availability of data representing given refers to Mark, such as Amazon needs the availability of data index ensured to be 99.9%.

Thus, the decision variable of object function is the storage strategy that N number of data are corresponding, represents by (2) formula:

V=(s₁, s₂..., s_N) (2)

S1.4, the computational methods of data overall overhead

Only having storage overhead or only computing cost different from data in existing scheme, in the present invention, the entirety of data is opened Pin includes the storage overhead of existing copy and generates the computing cost of data when all copies lost efficacy.In Fig. 2, data d₅'s When storage overhead that overall overhead comprises two copies and two copies all lost efficacy, generate d₅Computing cost.Now, data d_i Overall overhead C_iComputing formula is as shown in (3) formula:

$C_i\left(t\right)=C_i^s\left(t\right)+(1-A_i^H)\·d_i^{\mathrm{\α}}\·\[C_i^s\left(d_i^t\right)+C_i^w\left(d_i^t\right)\]---\left(3\right)$

Wherein,Represent data d respectively_iStorage overhead in time t and computing cost, use formula respectively (4) (5) calculate.Represent and do not consider when data regenerate, data d_iThe availability of self, uses formula (6) to calculate. In express time t, the factor of the overhead that node failure causes, use formula (7) to calculate.

$C_i^s\left(t\right)=r^s\·r\left(s_i\right)\·d_i^{size}\·t---\left(4\right)$

Wherein, r^sRepresentation unit time storage cell data need the storage charges use paid, r (s_i) represent storage strategy s_i Corresponding data trnascription redundancy, works as s_i=0, when 1,2,3,4, r (s_i)=0,1,2,3,1.4.

$C_i^w\left(t\right)=r^w\·E\left(T_i\right)---\left(5\right)$

Wherein, r^wRepresent that the unit of account time needs the computational costs of payment, T_iRepresent data d_iThe generation time, E (T_i) Represent T_iExpected value, with (8) formula calculate.

$A_i^H={\mathrm{\Σ}}_{i=m}^n\left(\begin{array}{c}n\\ i\end{array}\right)\·{p_l}^i\·{(1-p_l)}^{n-i}---\left(6\right)$

Wherein, n represents data d_iCopy amount, m represents the minimum copy amount of needs when data are normal.When data with When many copies mode stores, m takes 1；When data store in the way of correcting and eleting codes, m is that this correcting and eleting codes recovery data need Few number of copies.When such as data store in (10,14) correcting and eleting codes mode, m takes 10.

$d_i^{\mathrm{\α}}=\frac{1}{d_i^t}\·\underset{i=1}{\overset{K}{\Σ}}min(t_i,d_i^t)---\left(7\right)$

Wherein, K, t_iData d in express time t respectively_iThe number of stoppages and the time of i & lt fault.

S1.5, data genaration time desired computational methods

Data d_iGeneration time and data itself and the malfunction of dependence (directly rely on or indirectly rely on) data Relevant.As in figure 2 it is shown, when data d₅When having available copies, data d₅The generation time be 0；When data d₅Two copies the most not Can use and d₄Time available, data d₅The generation time beWhen data d₅, d₄When all copies are the most unavailable, need first to generate d₄, Therefore d₅The generation time be more thanPresent invention stochastic variable T_iRepresent data d_iThe generation time, E (T_i) represent T_iMathematics Expect.Generation time expectation (8) formula calculates.

$E\left(T_i\right)=\left\{\begin{array}{ccc}0& if& s_i\&GreaterEqual;3\\ (1-A_i^H)\left(d_i^t+\underset{d_j\&Element;d_i^{pSet}}{\&Sigma;}E\left(T_j\right)\right)& if& s_i<3\end{array}\right.---\left(8\right)$

Meanwhile, when the degree of depth of iterative computation is more than c, E (T is set_i)=0, terminates iteration.Wherein c meets

p_l ^c≤1-A^E (9)

Wherein, p_lRepresent the probability of malfunction of data memory node.

S1.6, the computational methods of availability of data

When user is to data d_iThe response time allowed isTime, the only generation time of data is less thanTime, Dependence can be utilized to generate data to improve the availability of data.Now, it is considered to data regenerate the sound allowed with user When should postpone two factors, data d_iAvailability be equal toFormula (10) is used to calculate.

S2, data store strategy decision making algorithm

The data store strategy decision making algorithm overall flow that the present invention provides is as shown in Figure 3: (include on new when there being new data The data passed and newly-generated intermediate data) when generating, use new data storage strategic decision-making algorithm (S2.1) decision data Storage strategy；At the end of each decision-making period (present invention is 1 hour), with each connected component in dependence graph it is Unit, updates the storage strategy (S2.2) of wherein data.Detailed process is as shown in Figure 4: first carry out initialization procedure (S2.2.1), Whether the nodes further according to connected component is less than or equal to H, uses traversal pruning algorithms (S2.2.2) and genetic algorithm respectively (S2.2.3) the storage strategy of data in renewal branch.

The key step of algorithm is described in detail as follows:

S2.1, new data storage strategic decision-making algorithm

Step S2.1 in this step corresponding diagram 3, specifically comprises the following steps that

(1-1) when data d_iWhen being the data newly uploaded, s is set_i=3.

(1-2) when data d_iWhen being newly-generated intermediate data, calculate s respectively_iThe data that=j (j=0,1,2,3) is corresponding Overall overheadS is set_i=k, meetsAnd meet

S2.2, data store strategy renewal process

Step S2.2 in this step corresponding diagram 3, particular flow sheet is as shown in Figure 4.Three key steps therein are detailed It is described as follows:

S2.2.1, initialization

Step S2.2.1 in this step corresponding diagram 4, specifically comprises the following steps that

(2-1) minimal-overhead minV=∞, the storage strategy that minimal-overhead is corresponding are initializedH=5.

(2-2) traversal connected component obtains corresponding sequence node.First, replicate connected component's topological structure, obtain figure G, creates an empty queue Q.Secondly, the node that in-degree in figure G is 0 is added enqueue Q, and delete in figure G this node and The limit being associated.Owing to figure G is directed acyclic graph, repeatedly performing this step, when scheming G and being empty, queue Q is corresponding node sequence Row.

S2.3, traversal pruning algorithms

Step S2.2.2 in this step corresponding diagram 4, idiographic flow is as shown in Figure 5.

When queue Q length is less than or equal to H, the storage plan of data in the current connected component of use traversal pruning algorithms renewal Slightly, specifically comprise the following steps that

(3-1) initialize.First, a newly-built tree T only having root node.Secondly, element in Q is gone out team, if this yuan Element in-degree in dependence graph is 0, then be that in T, each leaf node adds two child nodes, and the value of node is respectively 3,4. Otherwise, each leaf node in T being added 4 child nodes, nodal value is respectively 0,1,2,3.Repeatedly perform, until queue Q is Empty.Finally, the most left child nodes that present node is tree T root node is set.

(3-2) availability of present node is calculated.Formula (10) is used to calculate the availability of present node corresponding data WhenTime, jump procedure (3-3).Otherwise, jump procedure (3-4).

(3-3) beta pruning.Terminate the traversal to present node descendant nodes, present node is updated to father's joint of present node Point, jump procedure (3-5).

(3-4) minV, minS are updated.If the leaf node that present node is tree T, by root node to this leaf node road Footpath interior joint value solves V as one.Formula (1a) (3) is used to calculate system overall overhead C corresponding to V, if C < minV, the most more New minV=C, minS=V.If present node is not the leaf node of tree T, then jump procedure (3-5).

(3-5) present node is updated.Present node is updated to next node by the mode of depth-first traversal by tree T After, jump procedure (3-6).

(3-6) traversal terminates to judge.If tree T traversal terminates, jump procedure (3-7), otherwise jump procedure (3-2).

(3-7) updating the storage strategy of data in this connected component is minS.

S2.4, genetic algorithm

Step S2.2.3 in this step corresponding diagram 4, idiographic flow is as shown in Figure 6.

When queue Q length is more than H, the storage strategy of data in the current connected component of use genetic algorithm renewal, specifically Step is as follows:

(4-1) M=50, L=300, P are initialized_c=0.95, P_m=0.1.Wherein, M represents that population scale, L represent heredity The iterations of algorithm, P_cRepresent crossover probability, P_mRepresent mutation probability.

(4-2) coding.The vectorial V of corresponding a length of M is obtained after population is encoded:

V=(v₁, v₂..., v_M)

Wherein, v_iRepresent i-th (1≤i≤M) individuality in population, individual v_iBy n genomic constitution:

v_i=(v_i1, v_i2..., v_im..., v_in)

Wherein, v_ijRepresent individual v_iJth gene, m represents that in queue Q, in-degree is the node number of 0, and n represents queue Q Length, i.e. the node number of connected component.Each gene is the value in a valued space,

As 1≤j≤m, v_ij{ 0,1}, 0,1 represents s to ∈ respectively_iValue is 3,4.

When m < during j≤n, v_i{ 00,01,10,11}, 00,01,10,11 represent s to ∈ respectively_iValue is 0,1,2,3.

(4-3) population V is initialized.Randomly generating M individuality in population, each individual corresponding coding is as follows:

As 1≤j≤m, v_ij{ randomly choose in 0,1} at valued space.

When m < during j≤n, v_ij{ randomly choose in 00,01,10,11} at valued space.

(4-4) select to obtain centre for S.First, the fitness P of each individuality in population is calculated_i, as shown in (11) formula:

$P_i=(1-\frac{F_i\&CenterDot;A_i^D}{{\mathrm{\&Sigma;}}_{i=1}^{i=M}{F}_i\&CenterDot;A_i^D})---\left(11\right)$

Wherein, F_iRepresent the overall overhead that in population, i-th individuality is corresponding, calculate with formula (3).Secondly, according to P_iWill (0,1) M interval it is divided into.Finally, carry out M time to select.Produce the random number in (0,1), according to random number institute every time The individuality that place's interval selection is corresponding, finally obtains middle for S.Jump procedure (4-5).

(4-5) intersection obtains middle for D.Centre selection obtained is for the individual pairing in S, to every couple of individuality (v_i, v_j) Carry out intersection operation and obtain v_i', v_j', generate middle for D.First, the random number k in (0,1) is produced.Secondly, k is worked as > P_c Time, v_i'=v_i, v_j'=v_j.As k≤P_cTime, individual front m and rear (n-m) individual gene are carried out single-point intersection respectively.Redirect Step (4-6).

(4-6) variation obtains filial generation.By centre for each individual v in D_iWith probability P_mMake a variation, generate in filial generation Individual.First, the random number k in (0,1) is produced.Secondly, k is worked as > P_mTime, by individuality v_iIt is placed directly in filial generation.Otherwise, By individuality v_iJth gene change into other values of solution space at random.Jump procedure (4-7).

(4-7) update.Calculate system overall overhead C that in filial generation, optimum individual is corresponding, when C < during minV, updates minV= C, is updated to the storage strategy that optimum individual is corresponding by minS.Jump procedure (4-8).

(4-8) termination condition test.If the iterations of genetic algorithm > L time, terminate genetic algorithm, update this connection In branch, the storage strategy of data is minS.Otherwise, jump procedure (4-4) proceeds iteration.

Claims (6)

1. the method reducing cloud storage overall overhead when there is data dependence relation, it is characterised in that: comprise data storage Decision model and data storage strategic decision-making algorithm two parts.

2. the storage of the data in a claim 1 decision model, it is characterised in that: the decision variable of model is the storage of data Strategy, as shown in formula (2)；The object function of model, for minimizing system overall overhead, is given by formula (1a)；The pact of model Bundle condition is that the availability of data needs to meet given approve-useful index, formula (1b) be given.

3. C in the system overall overhead in a claim 2_iThe circular of (), it is characterised in that: C_i() by Formula (3) is given.

4. in a claim 3,Calculating formula (5) in data genaration time desired computational methods, it is characterised in that: E(T_i) be given by formula (8).

5. in a claim 2, availability of dataCircular, it is characterised in that: consider data again Generate and two factors of response time of user's permission,Be given by formula (10).

6. the data store strategy decision making algorithm in a claim 1, it is characterised in that: the execution step of algorithm is by flow chart 3 are given.