CN106161599A - A kind of method reducing cloud storage overall overhead when there is data dependence relation - Google Patents

Abstract

The invention discloses a kind of method reducing cloud storage overall overhead when there is data dependence relation.The method comprises data storage decision model and data storage two parts of strategic decision-making algorithm.In data storage decision model, the decision variable of model is the storage strategy of data, including not storing, the storage of many copies and correcting and eleting codes storage；The constraints of model is that the availability of data needs to meet given approve-useful index；The optimization aim of model is for minimizing system overall overhead, and wherein the overall overhead of data can include storage overhead and computing cost simultaneously；In computing cost, the data genaration time is a stochastic variable, The present invention gives its desired computational methods.In data store strategy decision making algorithm, when there being new data to generate, the storage strategy of direct decision data；At the end of each decision-making period, in units of the connected component in dependence graph, whether exceed threshold value according to the nodes of connected component, the storage strategy of data in use genetic algorithm or traversal pruning algorithms update this connected component respectively.Compared with prior art, the method that the present invention provides has the advantage reducing system overall overhead on the premise of ensure that availability of data.

Description

Method for reducing overall overhead of cloud storage in presence of data dependency relationship

Technical Field

The invention relates to the field of cloud storage, in particular to a method for reducing the overall overhead of cloud storage when a data dependency relationship exists.

Background

With the rapid development of cloud storage, more and more applications store data in the cloud, and how to reduce the storage overhead of mass data in a data center becomes an important problem.

In cloud storage, there may be dependencies between data. For example, video files with different definitions can be generated by transcoding the original video file, and at this time, a dependency relationship exists between the original file and the transcoded file.

At present, a scheme for reducing the overall overhead of cloud storage in the presence of a data dependency relationship is as follows: whether the data needs to be stored or not is determined through an algorithm, and at the moment, the data is not stored or is stored in a plurality of copies with fixed copy numbers. For data which is not stored, when the data receives an access request, the data is generated by utilizing the dependency relationship, and then the access service is provided. At this time, the overall overhead of the system includes a storage overhead resulting from storing data and a calculation overhead resulting from generating data.

However, the existing solutions are also deficient in two ways: (1) when storing data, the storage policy of the data is fixed. The case where the data storage policy is variable is not further considered. (2) The problem that data is not available when the generation time of the data exceeds the response delay time allowed by the user is not considered.

Disclosure of Invention

Aiming at the problems in the prior art, the invention mainly aims to provide a method for reducing the overall overhead of a system on the premise of ensuring the availability index of data when the data has dependency relationship. For simplicity, the term "data" hereinafter refers to data in which dependencies exist in cloud storage. A data copy may refer to one copy of a copy mode or one coded fragment of an erasure coding mode. The storage strategy of the data refers to a data storage mode, including non-storage, multi-copy storage and erasure code storage.

The present invention is characterized by comprising the following contents:

1. data storage decision model

In the existing scheme, a multi-copy storage strategy with fixed copy number is used when data is stored. The decision variable of the storage model is whether N data need to be stored or not, and a single data only has storage overhead or calculation overhead and does not consider the problem of data availability. Different from the existing scheme, the storage strategy of the data in the invention is variable, and the method is characterized in that: 1) the decision variable of the model is a storage strategy of N data, as shown in formula (2); 2) the overall cost of the data comprises the storage cost of the existing copies and the calculation cost of generating the data when all the copies fail, and the calculation method is given by formula (3); 3) the objective function of the model is to minimize the overall overhead of the system, and is given by formula (1 a); 4) the constraints of the model are that the availability of data needs to meet a given availability index, given by equation (1 b).

2. Method for calculating data generation time expectation

When calculating the overall overhead of the data, the calculation overhead is related to the generation time of the data. In the present invention, the generation time T of data_iIs a random variable that is related to the fault status of data on which the data depends directly or indirectly. The invention provides an expectation of data generation time, E (T)_i) The calculation method of (2) is given by the formula (8).

3. Data availability calculation method

When data is not stored or fails, if the data cannot be generated within the response delay time allowed by the user, the data is unavailable, and the availability of the data is affected. The invention provides data availabilityThe calculation method of (2) is given by the formula (10).

4. Data storage strategy decision algorithm

The invention provides a data storage strategy decision algorithm for reducing the overall overhead of a system on the premise of ensuring the availability of data, and the execution steps of the algorithm are given by a flow chart 3.

Drawings

FIG. 1 is a schematic diagram of a storage model structure according to the present invention.

FIG. 2 is a schematic diagram of data dependencies.

FIG. 3 is an overall flow diagram of a data storage policy decision algorithm.

Fig. 4 is a flowchart of the data storage policy update process of fig. 3.

Fig. 5 is a flowchart of step S2.2.2 in fig. 4.

Fig. 6 is a flowchart of step S2.2.3 in fig. 4.

Detailed Description

The method for reducing the overall overhead of cloud storage in the presence of data dependency mainly comprises two parts:

s1, data storage decision model: the model mainly comprises a structural model of the system and an overall overhead model of the system;

s2, data storage strategy decision algorithm: the algorithm provides a data storage strategy decision algorithm which reduces the overall overhead of the system on the premise of ensuring the availability of data.

The following detailed description of embodiments of the invention refers to the accompanying drawings.

S1, a data storage decision model mainly comprises the following parts:

s1.1, Structure of storage System

The storage system adds a data dependency management layer between a data access interface layer and a data storage layer of the cloud storage system, as shown in fig. 1. When the data dependency management layer receives a data access request, the following conditions exist:

case 1: if the data does not have the dependency relationship, forwarding the data access request to a data storage layer;

case 2: if the data has the dependency relationship and an available data copy exists, forwarding the data access request to a data storage layer; as in fig. 2, data d₅Is a double copy store. When at least one of the two copies is available, the system provides access services using the available copy.

Case 3: if the data has the dependency relationship and all the copies are unavailable, firstly generating the data by using the dependency relationship, and then forwarding the data access request; as in fig. 2, when data d₅When the storage nodes of the two copies fail and the copies are not available, data d is generated by utilizing the dependency relationship₅. If d is being generated₅In the process, data d₅And if the node where the copy is located is recovered to be normal, stopping the generation process, and using the original copy to continuously provide access service. Otherwise, deleting the original copy and providing access service by using the newly generated copy.

S1.2, data dependency management layer structure

In S1.1, the functions of the data dependency management layer mainly include: data dependency graph establishment and failure data regeneration.

The data dependency graph is established as follows: based on the dependency relationship of the data, modeling N pieces of data with dependency relationship in the system into a topological structure of a directed acyclic graph, namely a data dependency relationship graph. Nodes in the graph represent data, with directed edgesRepresenting data d_iIs generated in dependence on the data d_j. Data d_iComprisesThe attributes are as follows:s_i(1. ltoreq. i. ltoreq.N) each represents d_iStorage space occupied by a single copy, d_iDependent data set, d_iGeneration time of (d)_iGeneration operator of d_iThe storage policy of (1). Wherein s is_iThe value space of (a) is {0,1,2,3,4}, which respectively represents the data d_iNo store, single copy store, double copy store, triple copy store, and (10,14) erasure code store.

The fault data generation process is as follows: when data d_iWhen not available, to d_iDependent data setExecuting a generation operatorElapsed timeAfter-regeneration of d_i. When data in the dependent data set is unavailable, the data needs to be generated recursively first and then data d is generated_i。

S1.3, modeling of overall overhead of system

The overall cost of the system is equal to the sum of the overall costs of the N data, and the optimization goal of the system is to ensure that the data meets the given availability index A^EMeanwhile, the overall overhead of the system is reduced.

Optimizing objective function of system overall overhead:

$\min F(s_1,s_2,...,s_N)=min\underset{i=1}{\overset{N}{\Σ}}{C}_i\left(s_i\right)---\left(1a\right)$

the constraints of the objective function are:

$A_i^D\≥A^E,(i=1,2,...,N)---\left(1b\right)$

where F represents the overall overhead of the system during the decision period. C_iAndrespectively representing data d in the period_iThe total overhead and the availability of data are calculated using equations (3) (10), respectively; a. the^EIndicating that a given data availability indicator, such as Amazon, needs to guarantee a data availability indicator of 99.9%.

Therefore, the decision variable of the objective function is a storage strategy corresponding to N data, and is represented by the formula (2):

V＝(s₁，s₂，…，s_N) (2)

s1.4, method for calculating overall data overhead

The overall overhead of the data in the invention comprises the storage overhead of the existing copy and the calculation overhead of generating the data when all the copies fail, which is different from the existing scheme that the data only has the storage overhead or only has the calculation overhead. As in fig. 2, data d₅Including the storage overhead of both copies and when both copies fail, d is generated₅The computational overhead of (2). At this time, data d_iOverall overhead C of_iThe calculation formula is shown as the formula (3):

$C_i\left(t\right)=C_i^s\left(t\right)+(1-A_i^H)\·d_i^{\mathrm{\α}}\·\[C_i^s\left(d_i^t\right)+C_i^w\left(d_i^t\right)\]---\left(3\right)$

wherein,respectively represent data d_iThe storage cost and the calculation cost in the time t are respectively calculated by the formulas (4) and (5).Indicating data d irrespective of data regeneration_iThe availability of itself, calculated using equation (6).The factor representing the overhead caused by the node failure during time t is calculated using equation (7).

$C_i^s\left(t\right)=r^s\·r\left(s_i\right)\·d_i^{size}\·t---\left(4\right)$

Wherein r is^sIndicating the storage fee to be paid per unit time for storing unit data, r(s)_i) Representing a storage policy s_iCorresponding data copy redundancy, when s_iR(s) when equal to 0,1,2,3,4_i)＝0，1，2，3，1.4。

$C_i^w\left(t\right)=r^w\·E\left(T_i\right)---\left(5\right)$

Wherein r is^wRepresents the calculation cost T to be paid per unit time_iRepresenting data d_iGeneration time of (E) (T)_i) Represents T_iThe expected value of (c) is calculated by the equation (8).

$A_i^H={\mathrm{\Σ}}_{i=m}^n\left(\begin{array}{c}n\\ i\end{array}\right)\·{p_l}^i\·{(1-p_l)}^{n-i}---\left(6\right)$

Wherein n represents data d_iM represents the minimum number of copies required when the data is normal. When the data is stored in a multi-copy mode, m is 1; when data is stored in an erasure code, m is the minimum number of copies that the erasure code needs to recover the data. M takes 10, as opposed to when the data is stored as (10,14) erasure codes.

$d_i^{\mathrm{\α}}=\frac{1}{d_i^t}\·\underset{i=1}{\overset{K}{\Σ}}min(t_i,d_i^t)---\left(7\right)$

Wherein, K, t_iRespectively representing data d within time t_iThe number of failures and the time of the ith failure.

S1.5, calculation method of data generation time expectation

Data d_iIs related to the fault status of the data itself and its dependencies (directly or indirectly) on the data. When data d is shown in FIG. 2₅Data d when there is a copy available₅The generation time of (3) is 0; when data d₅Is not available and d₄When available, data d₅Is generated at a time ofWhen data d₅，d₄When all copies are not available, d needs to be generated first₄Thus d is d₅Is generated for a time greater thanRandom variable T for the invention_iRepresenting data d_iGeneration time of (E) (T)_i) Represents T_iThe mathematical expectation of (2). The generation time is desirably calculated by equation (8).

$E\left(T_i\right)=\left\{\begin{array}{ccc}0& if& s_i\&GreaterEqual;3\\ (1-A_i^H)\left(d_i^t+\underset{d_j\&Element;d_i^{pSet}}{\&Sigma;}E\left(T_j\right)\right)& if& s_i<3\end{array}\right.---\left(8\right)$

Meanwhile, when the depth of the iterative computation exceeds c, E (T) is set_i) The iteration is terminated, 0. Wherein c is satisfied

p_l ^c≤1-A^E(9)

Wherein p is_lRepresenting the probability of failure of the data storage node.

S1.6, data availability calculation method

When the user is to the data d_iThe allowable response delay time isOnly data is generated for less thanOnly then can the dependency be utilized to generate data to improve the availability of the data. At this time, data d is generated in consideration of two factors of data regeneration and response delay allowed by the user_iIs equal toThe calculation is performed using equation (10).

S2 data storage strategy decision algorithm

The overall flow of the data storage strategy decision algorithm provided by the invention is shown in fig. 3: when new data (including newly uploaded data and newly generated intermediate data) is generated, a new data storage strategy decision algorithm (S2.1) is used for deciding a storage strategy of the data; at the end of each decision cycle (1 hour in the invention), updating the storage strategy of the data in the dependency graph by taking each connected branch as a unit (S2.2). The specific process is shown in fig. 4: and (3) executing an initialization process (S2.2.1), and updating storage strategies of data in the branches by respectively using a traversal pruning algorithm (S2.2.2) and a genetic algorithm (S2.2.3) according to whether the number of nodes of the connected branches is less than or equal to H.

The main steps of the algorithm are described in detail as follows:

s2.1, new data storage strategy decision algorithm

This step corresponds to step S2.1 in fig. 3, and the specific steps are as follows:

(1-1) when data d_iWhen it is newly uploaded data, s is set_i＝3。

(1-2) when data d_iWhen it is newly generated intermediate data, s is calculated respectively_iJ (j is 0,1,2, 3) corresponds toData overhead ofSetting s_iK, satisfiesAnd satisfy

S2.2, data storage strategy updating process

This step corresponds to step S2.2 in fig. 3, and the detailed flowchart is shown in fig. 4. The three main steps are described in detail as follows:

s2.2.1, initialization

This step corresponds to step S2.2.1 in fig. 4, and the specific steps are as follows:

(2-1) initializing a storage strategy corresponding to minimum cost minV ═ infinity and minimum costH＝5。

And (2-2) traversing the connected branches to obtain the corresponding node sequences. Firstly, a connected branch topological structure is copied to obtain a graph G, and an empty queue Q is created. Second, add the node with an in-degree of 0 in graph G to queue Q, and delete the node and associated edge in graph G. Since the graph G is a directed acyclic graph, this step is repeatedly performed, and when the graph G is empty, the queue Q is the corresponding node sequence.

S2.3 traversal pruning algorithm

This step corresponds to step S2.2.2 in fig. 4, and the specific flow chart is shown in fig. 5.

When the length of the queue Q is less than or equal to H, updating the storage strategy of the data in the current connected branch by using a traversal pruning algorithm, and specifically comprising the following steps:

and (3-1) initializing. First, a tree T with only root nodes is created. And secondly, dequeuing the element in the Q, and if the in-degree of the element in the dependency graph is 0, adding two child nodes to each leaf node in the T, wherein the values of the nodes are 3 and 4 respectively. Otherwise, 4 child nodes are added to each leaf node in the T, and the node values are 0,1,2 and 3 respectively. This is repeated until queue Q is empty. And finally, setting the current node as the leftmost child node of the root node of the T tree.

And (3-2) calculating the availability of the current node. Calculating the availability of the current node corresponding data using equation (10)When in useAnd (4) jumping to the step (3-3). Otherwise, jumping to step (3-4).

(3-3) pruning. And (5) stopping traversing the descendant nodes of the current node, updating the current node to be the father node of the current node, and jumping to the step (3-5).

And (3-4) updating minV and minS. And if the current node is a leaf node of the tree T, taking the node value in the path from the root node to the leaf node as a solution V. The overall system overhead C for V is calculated using equations (1a) (3), and if C < minV, minV ═ C and minS ═ V are updated. And (3-5) if the current node is not the leaf node of the tree T.

And (3-5) updating the current node. And (3) after the current node is updated to the next node according to the depth-first traversal mode for the tree T, skipping to the step (3-6).

And (3-6) traversing ending judgment. And (5) if the traversal of the fruit tree T is finished, jumping to the step (3-7), otherwise, jumping to the step (3-2).

And (3-7) updating the storage strategy of the data in the connected branch to minS.

S2.4, genetic Algorithm

This step corresponds to step S2.2.3 in fig. 4, and the specific flow chart is shown in fig. 6.

When the length of the queue Q is larger than H, updating the storage strategy of the data in the current connected branch by using a genetic algorithm, and specifically comprising the following steps:

(4-1) initializing M-50, L-300, P_c＝0.95，P_m0.1. Wherein M represents the population size, L represents the iteration number of the genetic algorithm, and P_cIndicates the cross probability, P_mRepresenting the probability of variation.

And (4-2) encoding. After the population is coded, obtaining a vector V with a corresponding length of M:

V＝(v₁，v₂，...，v_M)

wherein v is_iRepresents the ith (1. ltoreq. i. ltoreq.M) individual, the individual v in the population_iConsists of n genes:

v_i＝(v_i1，v_i2，...，v_im，...，v_in)

wherein v is_ijRepresenting an individual v_iM represents the number of nodes with an in-degree of 0 in the queue Q, and n represents the length of the queue Q, i.e., the number of nodes of the connected branches. Each gene is a value in a value space,

when j is more than or equal to 1 and less than or equal to m, v_ij∈ {0,1}, 0,1 respectively representing s_iThe value is 3, 4.

When m is<When j is less than or equal to n, v_i∈ {00,01,10,11}, 00,01,10,11 respectively represent s_iThe value is 0,1,2, 3.

And (4-3) initializing the population V. Randomly generating M individuals in the population, wherein each individual corresponds to the following codes:

when j is more than or equal to 1 and less than or equal to m, v_ijRandomly selected in the value space {0,1 }.

When m is<When j is less than or equal to n, v_ijRandomly selected within the value space 00,01,10, 11.

(4-4) selecting to obtain an intermediate generation S. First, the fitness P of each individual in the population is calculated_iAs shown in formula (11):

$P_i=(1-\frac{F_i\&CenterDot;A_i^D}{{\mathrm{\&Sigma;}}_{i=1}^{i=M}{F}_i\&CenterDot;A_i^D})---\left(11\right)$

wherein, F_iThe total cost corresponding to the ith individual in the population is represented and calculated by formula (3). Secondly, according to P_iDivide (0,1) into M intervals. Finally, M selections are made. Generating a random number in (0,1) every time, selecting a corresponding individual according to the interval of the random number, and finally obtaining an intermediate generation S. And (4) skipping to step (5).

(4-5) crossing to obtain an intermediate generation D. Pairing the individuals in the selected intermediate generation S, and pairing each pair of individuals (v)_i，v_j) Performing a crossover operation to obtain v_i′，v_j', intermediate generation D is generated. First, a random number k within (0,1) is generated. Second, when k is>P_cWhen, v_i′＝v_i，v_j′＝v_j. When k is less than or equal to P_cIn this case, the first m and the last (n-m) genes of an individual are crossed at a single point. And (4) skipping to step (6).

(4-6) mutating to obtain progeny. Each individual v in the intermediate generation D_iWith probability P_mAnd performing mutation to generate individuals in the filial generation. First, a random number k within (0,1) is generated. Second, when k is>P_mWhen, the individual v_iDirectly put into offspring. Otherwise, the individual v_iRandomly changed to other values of the solution space. And (4) skipping to step (7).

And (4-7) updating. And calculating the whole system overhead C corresponding to the optimal individual in the filial generation, updating minV to C when C is less than minV, and updating minS to the storage strategy corresponding to the optimal individual. And (4) skipping to step (8).

And (4-8) finishing the condition test. And if the iteration times of the genetic algorithm are greater than L, ending the genetic algorithm, and updating the storage strategy of the data in the connected branch to minS. Otherwise, the jump step (4-4) continues to iterate.

Claims (6)

1. A method for reducing the overall overhead of cloud storage when a data dependency relationship exists is characterized in that: the method comprises a data storage decision model and a data storage strategy decision algorithm.

2. A data storage decision model as claimed in claim 1 wherein: the decision variable of the model is the storage strategy of the data, as shown in formula (2); the objective function of the model is to minimize the overall overhead of the system, and is given by formula (1 a); the constraint of the model is that the availability of data needs to meet a given availability index, given by equation (1 b).

3. A system overhead of claim 2, wherein C_iThe specific calculation method of (DEG) is characterized in that: c_i(. cndot.) is given by equation (3).

4. In a method as set forth in claim 3,the method for calculating the data generation time expectation in the calculation formula (5), wherein: e (T)_i) Given by equation (8).

5. A method as in claim 2 wherein data availabilityThe specific calculation method of (2) is characterized in that: considering both the data regeneration and the user-allowed response delay time,given by equation (10).

6. A data storage policy decision algorithm as claimed in claim 1 wherein: the steps of the algorithm execution are given by the flow chart of fig. 3.