CN103336785A - Distributed storage method and distributed storage device based on network coding - Google Patents

Abstract

The invention discloses a distributed storage method and a distributed storage device based on network coding, and belongs to the technical field of computer storage. The method and the device solve the problem that disk IO (Input/Output) of storage nodes is too large in the existing distributed storage method based on the network coding. The distributed storage method is suitable for a distributed storage system, and comprises the steps of data coding, data decoding and data repair. The distributed storage device comprises a data coding module, a data decoding module and a data repair module. According to the method and the device, when a data node is damaged, data not more than the size of a primary file D is downloaded from d data nodes, and the damaged data is repaired, so that the repair band width is reduced effectively; and randomly selected gamma coded data blocks are downloaded directly from the d data nodes, and the data blocks are not subjected to linear operation in the data nodes, so that the disk IO of the storage nodes can be reduced on the premise that the high usability of the data is ensured, and the disk IO efficiency of the data nodes is improved effectively.

Description

A kind of distributed storage method of coding Network Based and device thereof

Technical field

The invention belongs to computer memory technical field, be specifically related to a kind of distributed storage method and device thereof of coding Network Based.

Background technology

Along with the degree of social informatization improves constantly, the generation of information presents explosive growth, and people are also constantly uprising the storage demand of information.At commercial field, the business datum amount of company increases doublely, comprises multi-medium data, electronic commerce data, Webpage search data etc.More and more huger data storage is also more and more higher to the requirement of memory technology.

Distributed memory system has following advantage with respect to traditional file system: extensibility, reliability, high-throughput and system cost are low, the cluster scale of the distributed memory system of current large-scale Internet firm inside is in rapid expansion, and carrying cost is also in continuous rising.Along with the increase of node number in the cluster, the probability of node failure is also constantly increasing.In order to guarantee the high availability of data, adopt the mode of complete data redundancy to carry out fault-tolerant processing in the existing distributed storage system usually.All adopt three replication policies as GFS (Google File System) and HDFS (Hadoop Distributed File System), namely each part file is all preserved 3 parts of copies at memory node.Three replication policies are realized simple, can guarantee that also data have higher availability, but replication policy can bring sizable space consuming to system, and storage efficiency is low.After data quantity stored in the file system reached the PB level, for guaranteeing the availability of data, a copy of every increase all means can increase by 1,000,000 grades of other carrying costs.

Correcting and eleting codes is the popularization of copy, in face of mass data, is guaranteeing that correcting and eleting codes can significantly reduce carrying cost under the fault-tolerant ability identical with replication policy.Be that the file D of M is divided into k data block: d with size ₁, d ₂... d _k, each data block size is M ₁, produce n coded data block: C after the coding ₁, C ₂... C _n, each data block size is M ₁, store this n data block into n different back end: D ₁, D ₂... D _n, as a data node D _iDuring damage, be from k available back end (D _I1, D _I2... D _Ik) k data block (C of lining download _I1, C _I2... C _Ik), the data block C that recovers to lose _i

Just used RS class correcting and eleting codes to realize reliable memory cheaply in the second generation GFS of Google (the being ColossUs) system, see " Google ' s globally-distributed database " Proceedings of the10th USENIX conference on Operating Systems Design and Implementation.USENIX Association, 2012.251-264.Microsoft has also used the correcting and eleting codes technology in the Azure of its exploitation platform.See " Erasure coding in windows azure storage " Proceedings of the2012USENIX conference on Annual Technical Conference.USENIX Association, 2012.2.But there is a problem in correcting and eleting codes when repairing the back end that damages: repair M ₁The data block of size need be downloaded k * M altogether by network connection from k different node ₁The data block of size, it is expensive repairing bandwidth like this.

At the problem of the reparation bandwidth of correcting and eleting codes, the reparation bandwidth that network code can reduce back end effectively when damaging, but can cause the disk I quantitative change of back end inside big.See " Network Coding for Distributed Storage Systems " Proc.of IEEE INFOCOM2007.

Summary of the invention

The invention provides a kind of distributed storage method of coding Network Based, its device is provided simultaneously, solve the excessive problem of disk I of the existing memory node of distributed storage method of existing coding Network Based, when back end damages, by network connection d (the individual data node of d 〉=k), download the data of no more than source document D size, repair the data of damaging, reduce to repair bandwidth effectively.And γ the coded data block (C that download is selected at random from d data node _E1, C _E2... C _{E γ}), data block is not carried out linear operation in back end, directly read from back end, can improve the disk I efficient of back end effectively.

The distributed storage method of a kind of coding Network Based provided by the present invention is applicable to distributed memory system, comprises data coding step, data decode step and data reparation step, and distributed memory system is by a namenode NS and P memory node { DS ₁, DS ₂, DS ₃... DS _pConstitute, P 〉=3, the memory node that wherein is used for the storage file piecemeal is called back end, is n, 3≤n≤p; It is characterized in that:

(1) data coding step comprises following substep:

(1.1) deblocking:

Source document D is divided into big or small original data block D such as c piece _g, g=0,1...c-1 is for the residue raw data D of a raw data block size of less than _B, write down D earlier _BBig or small L _B, again it is used zero padding to supply and is the raw data block size, as original data block D _c

C=k * (d+1+i-k)-(i+1) * i/2, wherein, k is for recovering the required minimal data interstitial content of original, 2≤k＜n; The number of data available node when d damages node for one of reparation, k≤d＜n; I is the coding redundancy parameter, 0≤i≤k-1;

(1.2) redundancy encoding:

With c original data block D _aWith encoder matrix M _eCarry out Galois field 2 ^qInterior computing is encoded to r coded data block C _b, q=4,8,16,32 or 64; B=0,2 ... r-1; R=(d+1+i-k) * n;

$C_b={\mathrm{\Σ}}_{g=0}^{c-1}{a}_{b,g}{D}_g;$

Wherein, encoder matrix M _eIn matrix element a _{B, g}For belonging to Galois field 2 ^qInteger, 0≤a _{B, g}≤ 2 ^q-1, encoder matrix M _eIt is the Fan Demeng matrix of the capable c row of r; Each C _bAll be c original data block (D _g) _G=0,1...c-1Linear combination, linear combination coefficient corresponds to encoder matrix M _eThe vectorial V of the row that b is capable _b, i.e. each C _bCorresponding encoder matrix M _eThe vectorial V of the row that b is capable _b

(1.3) generator data file D _Meta:

With encoder matrix M _eAnd parameter n, k, d, i, q and L _BBe kept at meta data file D _MetaIn;

(1.4) data storage:

With r coded data block C _bLeave n data node d in _fOn, f=0,1 ... n-1, α=d+1+i-k coded data block of each back end storage, and store a D _MetaCopy; Back end d _fData blocks stored is C _t, t=f * α, f * α+1 ... (f+1) α-1;

(2) data decode step comprises following substep:

(2.1) obtain file metadata information:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

(2.2) download available block:

Judge that whether data available node number is then data read failure less than k in n the data node, withdraw from; Otherwise select k data available node arbitrarily, comprise r in k the data node _kThe individual coded data block of=k * α=k * (d+1+i-k), corresponding encoder matrix M altogether _eMiddle r _kIndividual row vector:

From encoder matrix M _eThis r _kSelect c row vector in the individual capable vector, the square formation M that requires this c row vector to form _E1Reversible, download this c the vectorial corresponding c coded data block of row: C then _B1, C _B2... C _Bc

(2.3) redundant decode:

To described square formation M _E1Matrix inversion obtains its inverse matrix M _E1 ^-1, inverse matrix M _E1 ^-1Middle element is designated as b _Gj, line number g=0 wherein, 1 ... c-1, columns j=0,1 ... c-1; With inverse matrix M _E1 ^-1Do Galois field 2 with c the coded data block of downloading ^qInterior computing obtains c original data block D _g,

$D_g={\mathrm{\Σ}}_{j=0}^{c-1}{b}_{\mathrm{gj}}{C}_{\mathrm{bj}},$ G=0 wherein, 1...c-1;

D _gBe c coded data block C _B0, C _B1... C _{B (c-1)}Linear combination, the coefficient of linear combination is inverse matrix M _E1 ^-1The corresponding vectorial V of row _Di

(2.4) recover data:

With c the original data block D that obtains behind the redundant decode _gBy target order D under it ₀, D ₁... D _C-1Be written among the recovery file D0 last piece original data block D successively _C-1Only write its preceding L _BIndividual byte forms recovery file D0 in recovery file D0;

(3) data are repaired step, as a data node d _vDuring damage, v is 0,1 ... or n-1, the reparation of the coded data block of its storage comprises following substep:

(3.1) obtain file metadata information:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

The initial value of downloading data block number variable γ is set:

γ＝(2×c×d)/((2×k-i-1)×i+2×k×(d-k+1))；

(3.2) computational data piece restoration information comprises following process:

(3.2.1) putting cycle index variable N1=0, whether judge in n the data node data available node number less than d, is data repairing failure then, withdraws from; Otherwise the process of carrying out (3.2.2);

(3.2.2) from d data available node, select γ coded data block at random, with they corresponding codes matrix M _eγ the vectorial V of row _hBe combined as the capable c column matrix of γ Vs, h=1,2... γ; Put N1=N1+1;

(3.2.3) generate the reparation matrix M that one (d+1+i-k) row γ is listed as _r=[m _{P, h}], each element m wherein _{P, h}From Galois field 2 ^qInterior random value, p=1,2 ... (d+1+i-k), h=1,2 ... γ;

(3.2.4) set up the new coding matrix M that the capable c of r is listed as _e', M _e' by original capable vector sum newline vector V ' _pConstitute, original row vector is the included coded data block corresponding codes matrix M of data available node _eIn row vector, by it at M _eIn original position be present in M _e' in, do Galois field 2 ^qInterior matrix M _rWith matrix V s multiplying, obtain newline vector V ' _z:

$V_p^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{V}_h,$

With newline vector V ' _pReplace encoder matrix M _eThe middle back end d that damages _vThe vectorial V of row of the α that stores a coded data block correspondence _z, z=v * α wherein, v * α+1 ... (v+1) * α-1;

(3.2.5) check described new coding matrix M _e' whether satisfy MDS character, be then to carry out substep (3.3), otherwise the process of carrying out (3.2.6);

(3.2.6) judging whether N1≤L, is then to turn over journey (3.2.2); Otherwise put N1=0, put γ=γ+1, turn over journey (3.2.2) then, maximum cycle L=1000～3000;

(3.3) update metadata file:

With metadata ` spare D _MetaIn encoder matrix M _eReplace with new coding matrix M _e', form the meta data file D after upgrading _Meta', copy it to each back end;

(3.4) repair data piece:

Download in (3.2.2) γ the coded data block (C of selection at random of institute _E1, C _E2... C _{E γ}), do Galois field 2 ^qInterior matrix M _rWith γ coded data block (C _E1, C _E2... C _{E γ}) computing, the data block C that obtains repairing _p':

${C_p}^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{C}_{\mathrm{eh}};$

C _p' be γ coded data block (C _E1, C _E2... C _{E γ}) linear combination, the coefficient of linear combination is for repairing matrix M _rThe corresponding vectorial V of row _r

(3.5) storage data block:

With the data block C that repairs _p' store on the new data available node.

Among the present invention, the data coding step is the basis, and it is mutually independently that data decode step and data are repaired step; When the user need carry out code storage to source document, file is carried out encoding operation, carry out the data coding step; When the user need download source document, file is carried out decode operation, carry out the data decode step; When a data node damages, the data block on this back end is repaired operation, carry out data and repair step.

In the described substep (1.1), according to the different values of coding redundancy parameter i, the effect difference of coding.When i=0, each back end data quantity stored α _RMinimum, α _R=M/k; When repairing the corrupt data node, in d the data available node, be γ from each back end data downloaded amount _R, γ _R=(M * d)/(k * (d-k+1)), wherein M is the size of source document.

When i=(k-1), each back end data quantity stored α _RFor:

α _R=(2 * M * d)/(2 * k * d-k ²+ k); When repairing the corrupt data node, in d the data available node, from each back end data downloaded amount γ _RMinimum, γ _R=(2 * M * d)/(2 * k * d-k ²+ k).

When 0＜i＜(k-1), each node data quantity stored α _RFor:

M/k＜α _R＜(2×M×d)/(2×k×d-k ²+k)。When repairing the corrupt data node, in d the data available node, from each back end data downloaded amount γ _RFor:

(2×M×d)/(2×k×d-k ²+k)＜γ _R＜(M×d)/(k×(d-k+1))。

Encoder matrix M in the described substep (1.2) _eBe the Fan Demeng matrix of the capable c row of r, satisfy ultimate range and can divide (MDS) character that namely the data of any k node can recover the data of original in n node;

From d data node, download γ the V that selects at random in the described substep (3.4) _iCorresponding codes data block (C _E1, C _E2... C _{E γ}), data block is not carried out linear operation at d data intranodal, directly reads from back end, can improve the disk I efficient of back end effectively.

The distributed storage devices of a kind of coding Network Based provided by the present invention is applicable to distributed memory system, comprises data coding module, data decode module and data reparation module.Distributed memory system is by a namenode NS and P memory node { DS ₁, DS ₂, DS ₃... DS _pConstitute, P 〉=3, the memory node that wherein is used for the storage file piecemeal is called back end, is n, 3≤n≤p; It is characterized in that:

(1) data coding module comprises following submodule:

(1.1) deblocking submodule:

Source document D is divided into big or small original data block D such as c piece _g, g=0,1...c-1 is for the residue raw data D of a raw data block size of less than _B, write down D earlier _BBig or small L _B, again it is used zero padding to supply and is the raw data block size, as original data block D _c

C=k * (d+1+i-k)-(i+1) * i/2, wherein, k is for recovering the required minimal data interstitial content of original, 2≤k＜n; The number of data available node when d damages node for one of reparation, k≤d＜n; I is the coding redundancy parameter, 0≤i≤k-1;

(1.2) redundancy encoding submodule:

With c original data block D _aWith encoder matrix M _eCarry out Galois field 2 ^qInterior computing is encoded to r coded data block C _b, q=4,8,16,32 or 64; B=0,2 ... r-1; R=(d+1+i-k) * n;

$C_b={\mathrm{\Σ}}_{g=0}^{c-1}{a}_{b,g}{D}_g;$

Wherein, encoder matrix M _eIn matrix element a _{B, g}For belonging to Galois field 2 ^qInteger, 0≤a _{B, g}≤ 2 ^q-1, encoder matrix M _eIt is the Fan Demeng matrix of the capable c row of r; Each C _bAll be c original data block (D _g) _G=0,1...c-1Linear combination, linear combination coefficient corresponds to encoder matrix M _eThe vectorial V of the row that b is capable _b, i.e. each C _bCorresponding encoder matrix M _eThe vectorial V of the row that b is capable _b

(1.3) generator data file D _MetaSubmodule:

With encoder matrix M _eAnd parameter n, k, d, i, q and L _BBe kept at meta data file D _MetaIn;

(1.4) data sub module stored:

With r coded data block C _bLeave n data node d in _fOn, f=0,1 ... n-1, α=d+1+i-k coded data block of each back end storage, and store a D _MetaCopy; Back end d _fData blocks stored is C _t, t=f * α, f * α+1 ... (f+1) α-1;

(2) data decode module comprises following submodule:

(2.1) obtain file metadata information submodule:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

(2.2) download the available block submodule:

Judge that whether data available node number is then data read failure less than k in n the data node, withdraw from; Otherwise select k data available node arbitrarily, comprise r in k the data node _kThe individual coded data block of=k * α=k * (d+1+i-k), corresponding encoder matrix M altogether _eMiddle r _kIndividual row vector:

From encoder matrix M _eThis r _kSelect c row vector in the individual capable vector, the square formation M that requires this c row vector to form _E1Reversible, download this c the vectorial corresponding c coded data block of row: C then _B1, C _B2... C _Bc

(2.3) redundant decode submodule:

To described square formation M _E1Matrix inversion obtains its inverse matrix M _E1 ^-1, inverse matrix M _E1 ^-1Middle element is designated as b _Gj, line number g=0 wherein, 1 ... c-1, columns j=0,1 ... c-1; With inverse matrix M _E1 ^-1Do Galois field 2 with c the coded data block of downloading ^qInterior multiplying obtains c original data block D _g,

$D_g={\mathrm{\Σ}}_{j=0}^{c-1}{b}_{\mathrm{gj}}{C}_{\mathrm{bj}},$ G=0 wherein, 1...c-1;

D _gBe c coded data block C _B0, C _B1... C _{B (c-1)}Linear combination, the coefficient of linear combination is inverse matrix M _E1 ^-1The corresponding vectorial V of row _Di

(2.4) recover the data submodule:

With c the original data block D that obtains behind the redundant decode _gBy target order D under it ₀, D ₁... D _C-1Be written among the recovery file D0 last piece original data block D successively _C-1Only write its preceding L _BIndividual byte forms recovery file D0 in recovery file D0;

(3) data are repaired module, as a data node d _vDuring damage, v is 0,1 ... or n-1, the reparation of the coded data block of its storage comprises following submodule:

(3.1) obtain file metadata information submodule:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

The initial value of downloading data block number variable γ is set:

γ＝(2×c×d)/((2×k-i-1)×i+2×k×(d-k+1))；

(3.2) computational data piece restoration information submodule comprises following unit:

Cycle index variable N1=0 is put in unit (3.2.1), whether judges in n the data node data available node number less than d, is data repairing failure then, withdraws from; Otherwise change unit (3.2.2);

γ coded data block selected, at random with they corresponding codes matrix M in unit (3.2.2) from d data available node _eγ the vectorial V of row _hBe combined as the capable c column matrix of γ Vs, h=1,2... γ; Put N1=N1+1:

Unit (3.2.3) generates the reparation matrix M that one (d+1+i-k) row γ is listed as _r=[m _{P, h}], each element m wherein _{P, h}From Galois field 2 ^qInterior random value, p=1,2 ... (d+1+i-k), h=1,2 ... γ;

The new coding matrix M that the capable c of r is listed as is set up in unit (3.2.4) _e', M _e' by original capable vector sum newline vector V ' _pConstitute, original row vector is the included coded data block corresponding codes matrix M of data available node _eIn row vector, by it at M _eIn original position be present in M _e' in, do Galois field 2 ^qInterior matrix M _rWith matrix V s multiplying, obtain newline vector V ' _p:

$V_p^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{V}_h,$

With newline vector V ' _pReplace encoder matrix M _eThe middle back end d that damages _vThe vectorial V of row of the α that stores a coded data block correspondence _z, z=v * α wherein, v * α+1 ... (v+1) * α-1;

Unit (3.2.5) checks described new coding matrix M _e' whether satisfy MDS character, be rotor module (3.3) then, otherwise change unit (3.2.6);

Unit (3.2.6) judges whether N1≤L, and being then changes unit (3.2.2); Otherwise put N1=0, put γ=γ+1, change unit (3.2.2) then, maximum cycle L=1000～3000;

(3.3) update metadata file submodule:

With meta data file D _MetaIn encoder matrix M _eReplace with new coding matrix M _e', form the meta data file D after upgrading _Meta', copy it to each back end;

(3.4) repair data piece submodule:

Download in (3.2.2) γ the coded data block (C of selection at random of institute _E1, C _E2... C _{E γ}), do Galois field 2 ^qInterior matrix M _rWith γ coded data block (C _E1, C _E2... C _{E γ}) computing, the data block C that obtains repairing _p':

${C_p}^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{C}_{\mathrm{eh}};$

C _p' be γ coded data block (C _E1, C _E2... C _{E γ}) linear combination, the coefficient of linear combination is for repairing matrix M _rThe corresponding vectorial V of row _r

(3.5) storage data block submodule:

With the data block C that repairs _VpStore on the new data available node.

Compared with prior art, the present invention has the following advantages:

(1) guarantees to solve under the prerequisite of data high availability and use huge space consuming that replication policy brings and the increase of carrying cost in the existing distributed storage system.

(2) when data node damages, (the individual data node of d 〉=k) is downloaded the data of no more than source document M size, repairs the data of damaging, and reduces to repair bandwidth effectively by network connection d.

When (3) repairing the back end of a damage, from d data node, download γ the coded data block (C that selects at random _E1, C _E2... C _{E γ}), data block is not carried out linear operation in back end, directly read from back end, can improve the disk I efficient of back end effectively.

Description of drawings

Fig. 1 is the structural representation of the embodiment of the invention;

Fig. 2 is embodiments of the invention data coding step schematic flow sheet;

Fig. 3 is embodiments of the invention data decode steps flow chart synoptic diagram;

Fig. 4 repairs the steps flow chart synoptic diagram for the embodiments of the invention data.

Embodiment

The present invention is further described below in conjunction with drawings and Examples.

As shown in Figure 1, the embodiment of the invention comprises data coding step, data decode step and data reparation step, is applicable to by a namenode NS and 5 memory node { DS ₁, DS ₂, DS ₃... DS ₅The distributed memory system that constitutes, n=4 the memory node that wherein is used for the storage file piecemeal is called back end, is d ₀, d ₁, d ₂, d ₃

(1) data coding step as shown in Figure 2, comprises following substep:

(1.1) deblocking:

Be the original data block D that the raw data D of 64M is divided into c=k * (d+1+i-k)-(i+1) * sizes such as i/2=4 piece with size _g, g=0,1...3, k=2, d=3, coding redundancy parameter i=0;

D wherein ₀, D ₁, D ₂, D ₃Size be 16M; The size of raw data D is the integral multiple of c, so L _B=0;

(1.2) data coding:

With 4 original data block D _gWith encoder matrix M _eCarry out Galois field 2 ⁸Interior computing is encoded to r=(d+i+i-k) * n=8 coded data block C _b, b=0,2 ... 7;

$C_b={\mathrm{\Σ}}_{g=0}^{c-1}{a}_{b,g}{D}_g;$

Wherein, encoder matrix M _eIn matrix element a _{B, g}For belonging to Galois field 2 ⁸Integer, 0≤a _{B, g}≤ 2 ⁸-1, encoder matrix M _eIt is the Fan Demeng matrix of 8 row, 4 row; Each C _bAll be 4 original data block (D _g) _G=0,1...3Linear combination, linear combination coefficient corresponds to encoder matrix M _eThe vectorial V of the row that b is capable _b, i.e. each C _bCorresponding encoder matrix M _eThe vectorial V of the row that b is capable _b

$M_e=\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\\ 1& 1& 1& 1\\ 1& 2& 4& 8\\ 1& 3& 5& 15\\ 1& 4& 16& 64\end{array},$

(1.3) generator data file:

With encoder matrix M _eAnd parameter n=4, k=2, d=3, i=0 and L _B=0 is kept at meta data file D _MetaIn;

(1.4) data storage:

With 8 coded data block C ₀, C ₁, C ₂... C ₇Leave 4 data node d in ₀, d ₁... d ₃On, each back end storage α=2 coded data block, back end d _uData blocks stored is C _{U * α}, C _{U * α+1}All store a D on each back end _MetaCopy;

(2) data decode step as shown in Figure 3, comprises following substep:

(2.1) obtain file metadata information:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _aAnd parameter n=4, k=2, d=3, i=0 and L _B=0;

(2.2) download available block:

Judging whether data available node number is less than 2 in 4 data nodes, is then data read failure, withdraws from; Otherwise select 2 data available node: d arbitrarily ₂, d ₃, d ₂, d ₃The coded data block that comprises in these 2 data nodes is: C ₄, C ₅, C ₆, C ₇, be total to corresponding encoder matrix M _eIn 4 row vectors: V ₄, V ₅, V ₆, V ₇These 4 row vectors are pressed V ₄, V ₅, V ₆, V ₇Order form square formation M _E1Coding M _eSatisfy MDS character, so square formation M _E1Reversible, download these 4 corresponding 4 coded data block: the C of row vector then ₄, C ₅, C ₆, C ₇

$M_{e1}=\begin{array}{cccc}1& 1& 1& 1\\ 1& 2& 4& 8\\ 1& 3& 5& 15\\ 1& 4& 16& 64\end{array},$

(2.3) redundant decode:

To described square formation M _E1Matrix inversion obtains its inverse matrix M _E1 ^-1, inverse matrix M _E1 ^-1Middle element is designated as b _Gj, line number g=0 wherein, 1 ... 3, columns j=0,1 ... 3; With inverse matrix M _E1 ^-1With 4 coded data block C that download ₄, C ₅, C ₆, C ₇Do Galois field 2 ⁸Interior computing obtains 4 original data block D _g,

$D_g={\mathrm{\Σ}}_{j=0}^{c-1}{b}_{\mathrm{gj}}{C}_{\mathrm{bj}},$ G=0 wherein, 1...3;

D _gBe 4 coded data block C _B0, C _B1... C _{B (c-1)}Linear combination, the coefficient of linear combination is inverse matrix M _E1 ^-1The corresponding vectorial V of row _Di

$M_{d1}=\begin{array}{cccc}166& 245& 210& 128\\ 150& 220& 1& 075\\ 122& 244& 142& 0\\ 75& 221& 93& 203\end{array},$

(2.4) read data:

Earlier with the original data block D that obtains behind the redundant decode ₀, D ₁, D ₂, D ₃By target order D under it ₀, D ₁, D ₂, D ₃Be written to successively among the recovery file D0, because L _B=0, last piece original data block D ₃All write among the recovery file D0; Form recovery file D0 then;

(3) data are repaired step, as a data node d ₀During damage, the reparation of the coded data block of its storage comprises following substep, as shown in Figure 4:

(3.1) obtain file metadata information:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n=4, k=2, d=3, i=0 and L _B=0;

(3.2) computational data piece restoration information comprises following process:

(3.2.1) putting cycle index variable N1=0, whether judge in 4 data nodes data available node number less than 3, is data repairing failure then, withdraws from; Otherwise the process of carrying out (3.2.2), back end d ₁, d ₂, and d ₃All available, so the process of carrying out (3.2.2);

(3.2.2) from 3 data available node d ₁, d ₂, and d ₃In select 3 coded data blocks at random, with they corresponding codes matrix M _e3 vectorial V of row _hBe combined as 3 row, 4 column matrix Vs, h=1,2...3; Put N1=N1+1;

$V_s=\begin{array}{cccc}0& 0& 0& 1\\ 1& 2& 4& 8\\ 1& 3& 5& 15\end{array},$

(3.2.3) generate the reparation matrix M that (d+1+i-k)=2 row 3 is listed as _r=[m _{P, h}], each element m wherein _{P, h}From Galois field 2 ⁸Interior random value, p=1,2, h=1,2,3;

$M_r=\begin{array}{ccc}15& 15& 8\\ 14& 13& 10\end{array},$

(3.2.4) set up the new coding matrix M that 8 row 4 are listed as _e', M _e' by original capable vector sum newline vector V ' _pConstitute, original row vector is the included coded data block corresponding codes matrix M of data available node _eIn row vector, by it at M _eIn original position be present in M _e' in, do Galois field 2 ^qInterior matrix M _rWith matrix V s multiplying, obtain newline vector V ' _pFor:

P=v * α wherein, v * α+1 ... (v+1) * α-1;

With newline vector V ' _pReplace encoder matrix M _eThe middle back end d that damages _vThe vectorial V of row of the α that stores a coded data block correspondence _z, z=v * α wherein, v * α+1 ... (v+1) * α-1:

${V^{\′}}_p=\begin{array}{cccc}7& 6& 20& 15\\ 7& 4& 22& 0\end{array},$

${M^{\′}}_e=\begin{array}{cccc}7& 6& 20& 15\\ 7& 4& 22& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\\ 1& 1& 1& 1\\ 1& 2& 4& 8\\ 1& 3& 5& 15\\ 1& 4& 16& 64\end{array},$

(3.2.5) check described new coding matrix M _e' whether satisfy MDS character, be then to carry out substep (3.3), otherwise the process of carrying out (3.2.6).Check result is M _e' satisfy MDS character, so, carry out substep (3.3);

(3.2.6) judging whether N1≤L, is then to turn over journey (3.2.2); Otherwise put N1=0, put γ=γ+1, turn over journey (3.2.2) then, maximum cycle L=2000;

(3.3) update metadata file:

With meta data file D _MetaIn encoder matrix M _eReplace with new coding matrix M _e', form the meta data file D after upgrading _Meta', copy it to each back end;

(3.4) repair data piece:

Download in (3.2.2) 3 the coded data block (C of selection at random of institute _E1, C _E2, C _E3), do Galois field 2 ⁸Interior matrix M _rWith 3 coded data block (C _E1, C _E2, C _E3) computing, with the back end d that damages ₀The data blocks stored reparation of middle institute is C _p':

P=v * α wherein, v * α+1 ... (v+1) * α-1;

C _p' be 3 coded data block (C _E1, C _E2, C _E3) linear combination, the coefficient of linear combination is for repairing matrix M _rThe corresponding vectorial V of row _r

(3.5) storage data block:

With the data block C that repairs _VpStore a new data available node d into ₅On.

Claims (2)

1. the distributed storage method of a coding Network Based is applicable to distributed memory system, comprises data coding step, data decode step and data reparation step, and distributed memory system is by a namenode NS and P memory node { DS ₁, DS ₂, DS ₃... DS _pConstitute, P 〉=3, the memory node that wherein is used for the storage file piecemeal is called back end, is n, 3≤n≤p; It is characterized in that:

(1) data coding step comprises following substep:

(1.1) deblocking:

Source document D is divided into big or small original data block D such as c piece _g, g=0,1...c-1 is for the residue raw data D of a raw data block size of less than _B, write down D earlier _BBig or small L _B, again it is used zero padding to supply and is the raw data block size, as original data block D _c

C=k * (d+1+i-k)-(i+1) * i/2, wherein, k is for recovering the required minimal data interstitial content of original, 2≤k＜n; The number of data available node when d damages node for one of reparation, k≤d＜n; I is the coding redundancy parameter, 0≤i≤k-1;

(1.2) redundancy encoding:

With c original data block D _aWith encoder matrix M _eCarry out Galois field 2 ^qInterior computing is encoded to r coded data block C _b, q=4,8,16,32 or 64; B=0,2 ... r-1; R=(d+1+i-k) * n;

$C_b={\mathrm{\Σ}}_{g=0}^{c-1}{a}_{b,g}{D}_g;$

Wherein, encoder matrix M _eIn matrix element a _{B, g}For belonging to Galois field 2 ^qInteger, 0≤a _{B, g}≤ 2 ^q-1, encoder matrix M _eIt is the Fan Demeng matrix of the capable c row of r; Each C _bAll be c original data block (D _g) _G=0,1...c-1Linear combination, linear combination coefficient corresponds to encoder matrix M _eThe vectorial V of the row that b is capable _b, i.e. each C _bCorresponding encoder matrix M _eThe vectorial V of the row that b is capable _b

(1.3) generator data file D _Meta:

With encoder matrix M _eAnd parameter n, k, d, i, q and L _BBe kept at meta data file D _MetaIn;

(1.4) data storage:

With r coded data block C _bLeave n data node d in _fOn, f=0,1 ... n-1, α=d+1+i-k coded data block of each back end storage, and store a D _MetaCopy; Back end d _fData blocks stored is C _t, t=f * α, f * α+1 ... (f+1) α-1;

(2) data decode step comprises following substep:

(2.1) obtain file metadata information:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

(2.2) download available block:

Judge that whether data available node number is then data read failure less than k in n the data node, withdraw from; Otherwise select k data available node arbitrarily, comprise r in k the data node _kThe individual coded data block of=k * α=k * (d+1+i-k), corresponding encoder matrix M altogether _eMiddle r _kIndividual row vector: From encoder matrix M _eThis r _kSelect c row vector in the individual capable vector, the square formation M that requires this c row vector to form _E1Reversible, download this c the vectorial corresponding c coded data block of row: C then _B1, C _B2... C _Bc

(2.3) redundant decode:

To described square formation M _E1Matrix inversion obtains its inverse matrix M _E1 ^-1, inverse matrix M _E1 ^-1Middle element is designated as b _Gj, line number g=0 wherein, 1 ... c-1, columns j=0,1 ... c-1; With inverse matrix M _E1 ^-1Do Galois field 2 with c the coded data block of downloading ^qInterior computing obtains c original data block D _g,

$D_g={\mathrm{\Σ}}_{j=0}^{c-1}{b}_{\mathrm{gj}}{C}_{\mathrm{bj}},$ G=0 wherein, 1...c-1;

D _gBe c coded data block C _B0, C _B1... C _{B (c-1)}Linear combination, the coefficient of linear combination is inverse matrix M _E1 ^-1The corresponding vectorial V of row _Di

(2.4) recover data:

With c the original data block D that obtains behind the redundant decode _gBy target order D under it ₀, D ₁... D _C-1Be written among the recovery file D0 last piece original data block D successively _C-1Only write its preceding L _BIndividual byte forms recovery file D0 in recovery file D0;

(3) data are repaired step, as a data node d _vDuring damage, v is 0,1 ... or n-1, the reparation of the coded data block of its storage comprises following substep:

(3.1) obtain file metadata information:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

The initial value of downloading data block number variable γ is set:

γ＝(2×c×d)/((2×k-i-1)×i+2×k×(d-k+1))；

(3.2) computational data piece restoration information comprises following process:

(3.2.1) putting cycle index variable N1=0, whether judge in n the data node data available node number less than d, is data repairing failure then, withdraws from; Otherwise the process of carrying out (3.2.2);

(3.2.2) from d data available node, select γ coded data block at random, with they corresponding codes matrix M _eγ the vectorial V of row _hBe combined as the capable c column matrix of γ Vs, h=1,2... γ; Put N1=N1+1;

(3.2.3) generate the reparation matrix M that one (d+1+i-k) row γ is listed as _r=[m _{P, h}], each element m wherein _{P, h}From Galois field 2 ^qInterior random value, p=1,2 ... (d+1+i-k), h=1,2 ... γ;

(3.2.4) set up the new coding matrix M that the capable c of r is listed as _e', M _e' by original capable vector sum newline vector V ' _pConstitute, original row vector is the included coded data block corresponding codes matrix M of data available node _eIn row vector, by it at M _eIn original position be present in M _e' in, do Galois field 2 ^qInterior matrix M _rWith matrix V s multiplying, obtain newline vector V ' _z:

$V_p^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{V}_h,$

With newline vector V ' _pReplace encoder matrix M _eThe middle back end d that damages _vThe vectorial V of row of the α that stores a coded data block correspondence _z, z=v * α wherein, v * α+1 ... (v+1) * α-1;

(3.2.5) check described new coding matrix M _e' whether satisfy MDS character, be then to carry out substep (3.3), otherwise the process of carrying out (3.2.6);

(3.2.6) judging whether N1≤L, is then to turn over journey (3.2.2); Otherwise put N1=0, put γ=γ+1, turn over journey (3.2.2) then, maximum cycle L=1000～3000;

(3.3) update metadata file:

With meta data file D _MetaIn encoder matrix M _eReplace with new coding matrix M _e', form the meta data file D after upgrading _Meta', copy it to each back end;

(3.4) repair data piece:

Download in (3.2.2) γ the coded data block (C of selection at random of institute _E1, C _E2... C _{E γ}), do Galois field 2 ^qInterior matrix M _rWith γ coded data block (C _E1, C _E2... C _{E γ}) computing, the data block C that obtains repairing _p':

${C_p}^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{C}_{\mathrm{eh}};$

C _p' be γ coded data block (C _E1, C _E2... C _{E γ}) linear combination, the coefficient of linear combination is for repairing matrix M _rThe corresponding vectorial V of row _r

(3.5) storage data block:

With the data block C that repairs _p' store on the new data available node.

2. the distributed storage devices of a coding Network Based is applicable to distributed memory system, comprises data coding module, data decode module and data reparation module.Distributed memory system is by a namenode NS and P memory node { DS ₁, DS ₂, DS ₃... DS _pConstitute, P 〉=3, the memory node that wherein is used for the storage file piecemeal is called back end, is n, 3≤n≤p; It is characterized in that:

(1) data coding module comprises following submodule:

(1.1) deblocking submodule:

Source document D is divided into big or small original data block D such as c piece _g, g=0,1...c-1 is for the residue raw data D of a raw data block size of less than _B, write down D earlier _BBig or small L _B, again it is used zero padding to supply and is the raw data block size, as original data block D _c

C=k * (d+1+i-k)-(i+1) * i/2, wherein, k is for recovering the required minimal data interstitial content of original, 2≤k＜n; The number of data available node when d damages node for one of reparation, k≤d＜n; I is the coding redundancy parameter, 0≤i≤k-1;

(1.2) redundancy encoding submodule:

With c original data block D _aWith encoder matrix M _eCarry out Galois field 2 ^qInterior computing is encoded to r coded data block C _b, q=4,8,16,32 or 64; B=0,2 ... r-1; R=(d+1+i-k) * n;

$C_b={\mathrm{\Σ}}_{g=0}^{c-1}{a}_{b,g}{D}_g;$

Wherein, encoder matrix M _eIn matrix element a _{B, g}For belonging to Galois field 2 ^qInteger, 0≤a _{B, g}≤ 2 ^q-1, encoder matrix M _eIt is the Fan Demeng matrix of the capable c row of r; Each C _bAll be c original data block (D _g) _G=0,1...c-1Linear combination, linear combination coefficient corresponds to encoder matrix M _eThe vectorial V of the row that b is capable _b, i.e. each C _bCorresponding encoder matrix M _eThe vectorial V of the row that b is capable _b

(1.3) generator data file D _MetaSubmodule:

With encoder matrix M _eAnd parameter n, k, d, i, q and L _BBe kept at meta data file D _MetaIn;

(1.4) data sub module stored:

With r coded data block C _bLeave n data node d in _fOn, f=0,1 ... n-1, α=d+1+i-k coded data block of each back end storage, and store a D _MetaCopy; Back end d _fData blocks stored is C _t, t=f * α, f * α+1 ... (f+1) α-1;

(2) data decode module comprises following submodule:

(2.1) obtain file metadata information submodule:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

(2.2) download the available block submodule:

Judge that whether data available node number is then data read failure less than k in n the data node, withdraw from; Otherwise select k data available node arbitrarily, comprise r in k the data node _kThe individual coded data block of=k * α=k * (d+1+i-k), corresponding encoder matrix M altogether _eMiddle r _kIndividual row vector: From encoder matrix M _eThis r _kSelect c row vector in the individual capable vector, the square formation M that requires this c row vector to form _E1Reversible, download this c the vectorial corresponding c coded data block of row: C then _B1, C _B2... C _Bc

(2.3) redundant decode submodule:

To described square formation M _E1Matrix inversion obtains its inverse matrix M _E1 ^-1, inverse matrix M _E1 ^-1Middle element is designated as b _Gj, line number g=0 wherein, 1 ... c-1, columns j=0,1 ... c-1; With inverse matrix M _E1 ^-1Do Galois field 2 with c the coded data block of downloading ^qInterior multiplying obtains c original data block D _g,

$D_g={\mathrm{\Σ}}_{j=0}^{c-1}{b}_{\mathrm{gj}}{C}_{\mathrm{bj}},$ G=0 wherein, 1...c-1;

D _gBe c coded data block C _B0, C _B1... C _{B (c-1)}Linear combination, the coefficient of linear combination is inverse matrix M _E1 ^-1The corresponding vectorial V of row _Di

(2.4) recover the data submodule:

With c the original data block D that obtains behind the redundant decode _gBy target order d under it ₀, D ₁... D _C-1Be written among the recovery file D0 last piece original data block D successively _C-1Only write its preceding L _BIndividual byte forms recovery file D0 in recovery file D0;

(3) data are repaired module, as a data node d _vDuring damage, v is 0,1 ... or n-1, the reparation of the coded data block of its storage comprises following submodule:

(3.1) obtain file metadata information submodule:

Download the meta data file D of source document D _Meta, obtain encoder matrix M _eAnd parameter n, k, d, i, q and L _B

The initial value of downloading data block number variable γ is set:

γ＝(2×c×d)/((2×k-i-1)×i+2×k×(d-k+1))；

(3.2) computational data piece restoration information submodule comprises following unit:

Cycle index variable N1=0 is put in unit (3.2.1), whether judges in n the data node data available node number less than d, is data repairing failure then, withdraws from; Otherwise change unit (3.2.2);

γ coded data block selected, at random with they corresponding codes matrix M in unit (3.2.2) from d data available node _eγ the vectorial V of row _hBe combined as the capable c column matrix of γ Vs, h=1,2... γ; Put N1=N1+1;

Unit (3.2.3) generates the reparation matrix M that one (d+1+i-k) row γ is listed as _r=[m _{P, h}], each element m wherein _{P, h}From Galois field 2 ^qInterior random value, p=1,2 ... (d+1+i-k), h=1,2 ... γ;

The new coding matrix M that the capable c of r is listed as is set up in unit (3.2.4) _e', M _e' by original capable vector sum newline vector V ' _pConstitute, original row vector is the included coded data block corresponding codes matrix M of data available node _eIn row vector, by it at M _eIn original position be present in M _e' in, do Galois field 2 ^qInterior matrix M _rWith matrix V s multiplying, obtain newline vector V ' _z:

$V_p^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{V}_h,$

With newline vector V ' _pReplace encoder matrix M _eThe middle back end d that damages _vThe vectorial V of row of the α that stores a coded data block correspondence _z, z=v * α wherein, v * α+1 ... (v+1) *+α-1;

Unit (3.2.5) checks described new coding matrix M _e' whether satisfy MDS character, be rotor module (3.3) then, otherwise change unit (3.2.6);

Unit (3.2.6) judges whether N1≤L, and being then changes unit (3.2.2); Otherwise put N1=0, put γ=γ+1, change unit (3.2.2) then, maximum cycle L=1000～3000;

(3.3) update metadata file submodule:

With meta data file D _MetaIn encoder matrix M _eReplace with new coding matrix M _e', form the meta data file D after upgrading _Meta', copy it to each back end;

(3.4) repair data piece submodule:

Download in (3.2.2) γ the coded data block (C of selection at random of institute _E1, C _E2... C _{E γ}), do Galois field 2 ^qInterior matrix M _rWith γ coded data block (C _E1, C _E2... C _{E γ}) computing, the data block C that obtains repairing _p':

${C_p}^{\′}={\mathrm{\Σ}}_{h=1}^{\mathrm{\γ}}{m}_{p,h}{C}_{\mathrm{eh}};$

C _p' be γ coded data block (C _E1, C _E2... C _{E γ}) linear combination, the coefficient of linear combination is for repairing matrix M _rThe corresponding vectorial V of row _r

(3.5) storage data block submodule:

With the data block C that repairs _VpStore on the new data available node.

Images