WO2018171111A1

WO2018171111A1 - Multi-fault tolerance mds array code encoding and repair method

Info

Publication number: WO2018171111A1
Application number: PCT/CN2017/097669
Authority: WO
Inventors: 侯韩旭; 李柏晴; 韩永祥
Original assignee: 东莞理工学院
Priority date: 2017-07-12
Filing date: 2017-08-16
Publication date: 2018-09-27
Also published as: CN107395207A; CN107395207B

Abstract

The present invention relates to the field of data processing, and provides a multi-fault tolerance MDS array code encoding, a component thereof being a C(k,3,p) code, expressing data blocks as k(p-1)τ information bits and encoding to generate 3(p-1)τ redundant bits, (p-1)τ being a positive integer, τ=2^k-2, p being a prime number, k≥3, τ extra bits being added to each (p-1)τ information bit to form a message vector. The advantageous effects of the present invention are: the fault tolerance of a system is improved; the calculation complexity of encoding and decoding is relatively low, and a repair bandwidth is thereby significantly reduced.

Description

Multi-faulty MDS array code encoding and repair method

[Technical Field]

The present invention relates to the field of data processing, and in particular, to a multi-fault-tolerant MDS array code encoding and repair method.

【Background technique】

Modern distributed storage systems use erasure codes to maintain data availability to avoid failure of storage nodes. The binary maximum distance separable (MDS) array code is a special erasure code that achieves fault tolerance with minimal storage redundancy and low computational complexity. Specifically, the binary array code is composed of an array of k+r columns, each column having L bits, wherein for the k+r columns, the k column information column stores information bits, and the r column parity column stores redundancy The remaining position. The L bits in each column are stored in the same storage node. If any k in the k+r column is sufficient to reconstruct all of the k-column information columns, then such a code can be referred to as an MDS (ie, it can tolerate any r-column failure). Examples of binary MDS array codes include: X code [1] and RDP code [2], both of which are double fault tolerant (ie, r = 2); and STAR code [3], generalized RDP code [4] And TIP code, all three are three fault-tolerant (ie r = 3).

When a node in a distributed storage system fails, the failed column should be repaired by downloading the bit from the unfailed node d, where k ≤ d ≤ k + r-1. The repair bandwidth is defined as the number of download bits in the repair operation. Minimizing repair bandwidth is critical to speeding up repair operations and minimizing fragile windows, especially in distributed storage systems where network transport often becomes a bottleneck. The repair problem was first elaborated and studied by Dimakis et al. [5] based on the information flow diagram. As stated in [5], the minimum repair bandwidth is subject to minimum storage redundancy, also known as the Minimum Memory Regeneration (MSR) point, which is calculated as follows:

At this time d=k+1. Although it is achievable to minimize the bandwidth in a sufficiently large finite field [5], [6], how to construct a binary MDS array code that achieves the minimum repair bandwidth is still a very challenging problem.

A number of studies have focused on reducing the repair bandwidth of a single failed column in a binary MDS array code. Some methods minimize the disk reading of RDP code [7] and X code [8], but the repair bandwidth is only suboptimal, still 50% larger than the minimum value in (1). The MDR codes [9], [10], and ButterFly codes [11] are binary MDS array codes that implement optimal repair, but they only provide double fault tolerance (ie, r = 2). How to construct a binary MDS array code with optimal repair and higher fault tolerance (ie r>2) is still a problem to be solved. Such a configuration would be beneficial to maintain data availability in a distributed storage system with frequent failures.

[Summary of the Invention]

In order to solve the problems in the prior art, the present invention provides a multi-fault-tolerant MDS array code encoding and repairing method, which solves the problem that the prior art cannot combine optimal repair and higher fault tolerance.

The invention is realized by the following technical solutions: a multi-fault-tolerant MDS array code coding is designed and manufactured, and the component thereof is a C(k, 3, p) code, and the data block is divided into k(p-1)τ. The information bits are coded to produce 3(p-1)τ redundant bits, (p-1)τ is a positive integer, τ=2 ^k-2 , p is a prime number, k≥3, and the information bits are expressed as

Redundant bits are expressed as

Where j is k+1, k+2, and k+3, i=1, 2,...,k; each (p-1)τ information bit is appended with τ extra bits and forms a message vector.

As a further improvement of the present invention, the information column is represented by a polynomial, and each corresponding information column is a data polynomial, and the corresponding three parity columns form an encoded polynomial, and the data polynomial and the encoded polynomial form a column vector [s ₁ (x), s ₂ (x), ..., s _k+3 (x)].

As a further improvement of the present invention, the column vector takes [s ₁ (x), s ₂ (x), ..., s by the algorithm embodied in R _pτ :=F ₂ [x]/(1+x ^pτ ) _k+3 (x)]=[s ₁ (x), s ₂ (x),...,s _k (x)]. The product of G is calculated, where G is the matrix of the kxk unit matrix I and a kx3 matrix The kx(k+3) composed of P generates a matrix.

As a further improvement of the invention: the C(k,3,p) code is _such that a systematic linear code is penetrated within R _pτ .

The invention also provides a method for repairing a multi-fault-tolerant MDS array code, comprising the steps of: obtaining an information column that has failed; if the information column has failed

Then fix the information bits s _l,f by the first parity, where l mod 2 ^f ∈{0,1,2,...2 ^f-1 -1}, otherwise repaired by the second parity Information bits s _l,f , where l mod 2 ^f ∈{2 ^f-1 , 2 ^f-1 +1,2 ^f-1 +2,...,2 ^f -1}; if the information column has failed

The information bits s _l,f are fixed by the first parity, where l mod 2 ^f ∈{0,1,2,...,2 ^f-1 -1}, otherwise, pass the third parity To fix the information bits s _l,f , where l mod 2 ^f ∈{2 ^f-1 , 2 ^f-1 +1,2 ^f-1 +2,...,2 ^f -1}.

As a further improvement of the present invention: the information column that has failed

The repaired broadband is (p-1)((k+2)2 ^k-3 -2 ^kf-2 ).

As a further improvement of the present invention, the parity sets of the second parity column and the third parity column are not corresponding to those columns of information in a straight line in the array, but correspond to those columns of information in a broken line; The number of lines is divisible by 2 ^k-2 .

The invention has the beneficial effects that the fault tolerance of the system is improved; the computational complexity of the codec process is lower, and the repair of the broadband is greatly reduced.

[Description of the Drawings]

1 is a schematic diagram of an embodiment of a memory code used in a three parity check column of the present invention.

【detailed description】

The invention will now be further described with reference to the drawings and specific embodiments.

Abbreviations and definitions of key terms

MDS Maximum Distance Separable Maximum Distance Separable

RDP Row-Diagonal Parity line diagonal check

A multi-fault-tolerant MDS array code coding, the component of which is a C(k,3,p) code, which divides the data block into k(p-1)τ information bits and encodes to generate 3(p-1)τ redundancy. The remaining bits, (p-1)τ is a positive integer, τ=2 ^k-2 , p is a prime number, k≥3, and the information bits are expressed as

Redundant bits are expressed as

The information column is represented by a polynomial, and each corresponding information column is a data polynomial, and the corresponding three parity columns form a coded polynomial, and the data polynomial and the coded polynomial form a column vector [s ₁ (x), s ₂ (x),...,s _k+3 (x)].

The column vector takes [s ₁ (x), s ₂ (x), ..., s _k+3 (x)] by the algorithm embodied in R _pτ :=F ₂ [x]/(1+x ^pτ ) =[s ₁ (x), s ₂ (x),...,s _k (x)]. The product of G is calculated, where G is kx (k) consisting of a kxk unit matrix I and a kx3 coding matrix P +3) Generate a matrix.

The C(k,3,p) code is a systematic linear code penetrating into R _pτ .

Information column that has expired

The repaired broadband is (p-1) ((k+2) 2 ^k-3 -2 ^kf-2 ).

The parity sets of the second parity column and the third parity column do not correspond to those columns of information in a line in the array, but correspond to those columns of information in a line; the number of rows of the array is 2 ^k-2 Divisible.

In an embodiment, the multi-fault-tolerant MDS array code is constructed as follows:

Let

^{k ≥} 3 and L = (p-1) τ be positive integers, where τ = 2 ^k-2 , and p is a prime number and 2 is a primitive in the Z _p domain. Consider a document size of k(p-1) τ bits, by information bits

Represents (i = 1, 2, ..., k), which can be used to generate 3 (p-1) τ redundant bits

For l = 1, 2, ..., k + 3 and μ = 0, 1, ... τ-1, the following shorthand notation is defined:

Weigh

For extra bits,

Related. For example, when p=3, k=4, and τ=4,

Extra bit is

For l = 1, 2, ..., k + 3, the bits in column l are passed through the polynomial s _l (x) on the ring F ₂ [x]

Extra bit with τ

Co-expressed as a polynomial s _l (x):

The polynomial s _i (x) corresponding to the i-th information column (i = 1, 2, ..., k) is called a data polynomial, corresponding to the jk parity column (j = k + 1, .k + 2,. The polynomial of k + 3) is called a coding polynomial.

Write k data polynomials and 3 coding polynomials as column vectors

[s ₁ (x), s ₂ (x),...,s _k+3 (x)], (3)

It can be utilized

The algorithm embodied in [S ₁ (x), s ₂ (x),..., s _k+3 (x)]=[s ₁ (x), s ₂ (x),...,s _{The product of k} (x)]·G is calculated. The kx(k+3) generation matrix G is composed of a kxk unit matrix I and a kx3 coding matrix P,

In the ring R _pτ , the variable x represents the cyclic right shift operator on the polynomial. This is critical to reducing the repair bandwidth for a column failure. The proposed code is represented as C(k,3,p). Please note that extra bits are not stored on disk; they are for convenience only. Consider an example where k=4 and p=3, 32 information bits are represented by s _0,i , s _1,i ,...,s _{7, i} , where i=1, 2, 3, 4. The encoding matrix for this example is

This example is illustrated in Figure 1, where the bolded bit elements are extra bits.

The encoding process can be described in the form of a polynomial as follows. Given k(p-1)τ information bits, τ extra bits are appended for each (p-1)τ information bit and form a message vector [s ₁ (x), s ₂ (x),...,s _k (x)].

After obtaining the vector in (3), the number of times of storage is 0 to the term coefficient in the (p-1) τ-1 polynomial. The proposed array code can be viewed as a systematic linear code that operates within Rpτ.

A progressive optimal repair of information failure, in one embodiment, will show how to repair the bits stored in information column f by accessing bits from k-1 other columns of information and 2 parity columns. _{0, f} , s _{1, f} , ..., s _{(p-1) τ -1, f} and have a progressive optimal repair bandwidth, where 1 ≤ _f ≤ k. Remember that extra bits can be calculated by (2). For convenience of representation, the bit elements of column i are represented as pτ bits s _0,i , s _1,i ,...,s _pτ-1,i . Before the repair algorithm is given, the parity set is formally defined as follows.

Definition 1. For 0 ≤ l ≤ pτ-1, define the first parity set of the first column, the second column, and the third column as

with

Please note that all indices in definition 1 and in the full text are modulo pτ. By definition 1, parity set

Includes multiple information bits that can be used to generate redundant bits

When it is said that one information bit is repaired by a parity column, this means that in addition to the erased bits, the redundant bits of the parity column and all the information bits in the parity column are accessed. Consider the example given in Figure 1. Assume that the first column is erased. One can access the bit bits s _0,2 , s _0,3 , s _0,4 and the redundant bits s _0,1 +s _0,2 +s _0,3 +s _0,4 to pass s _0,2 +s _0,3 +s _0,4 +(s _0,1 +s _0,2 +s _0,3 +s _0,4 ) to reconstruct s _0,1 .

The repair algorithm is stated in Algorithm 1. Consider again the example given in Figure 1 to elaborate on the repair process. In this example, k=5, d=5 and τ=4. Assume that the first column of information (ie, node 1) fails, that is, f=1. According to steps 2 and 3 in Algorithm 1, the bit can be repaired by the first parity column.

among them

And 0 ≤ l ≤ 7. More specifically, the bits s _0,1 , s _2,1 , s _4,1 , s _{6,1 are} from s _0,1 =s _0,2 +s _0,3 +s _0,4 +(s _{0, 1} +s _0,2 +s _0,3 +s _0,4 )s _2,1 =s _2,2 +s _2,3 +s _2,4 +(s _2,1 +s _2,2 +s _{2 ,3} +s _2,4 )s _4,1 =s _4,2 +s _4,3 +s _4,4 +(s _4,1 +s _4,2 +s _4,3 +s _4,4 )s _6,1 =s _6,2 +s _6,3 +s _6,4 +(s _6,1 +s _6,2 +s _6,3 +s _6,4 ) reconstruction.

Since f=1∈{1,2}, other information bits

Repaired by the second parity column, where

And 0 ≤ l ≤ 7.

Therefore, the bits s _1,1 , s _3,1 , s _5,1 , s _7,1 are s _1,1 =s _0,2 +s _10,3 +s _2,4 +(s _1,1 +s _0,2 +s _10,3 +s _2,4 )s _3,1 =s _2,2 +s _0,3 +s _4,4 +(s _3,1 +s _2,2 +s _0,3 + s _4,4 )s _5,1 =s _4,2 +s _2,3 +s _6,4 +(s _5,1 +s _10,2 +s _8,3 +s _0,4 )s _7,1 =s _6,2 +s _4,3 +s _8,4 +(s _11,1 +s _10,2 +s _8,3 +s _0,4 )+(s _3,1 +s _2,2 +s _0,3 +s _4,4 ) Reconstruction.

Since s _10,3 can be calculated by s _6,3 +s _2,3 and s _{8,4 is} calculated by s _4,4 +s _0,4 , there is no need to download bits s _10,3 and s _8,4 . Therefore, it is considered necessary to download four bits from each of the three columns of information and two columns of parity. To repair the bits of the first information column, a total of 20 bits need to be downloaded from five columns. That is, only half of the bits of the data column used for repair are accessed. In Figure 1, the bits in the solid line box are downloaded to fix the information bits s _0,1 , s _2,1 , s _4,1 , s _6,1 and the bits in the dashed box are used. Fix the information bits s _1,1 , s _3,1 , s _5,1 , s _7,1 . Assume that the second column of information (ie, node 2) fails, that is, f=2. According to steps 2 and 3 in Algorithm 1, you can pass

Repair bits s _0,2 , s _1,2 , s _4,2 , s _5,2 .

Similarly, it can pass s _2,2 =s _3,1 +s _0,3 +s _4,4 +(s _3,1 +s _2,2 +s _0,3 +s _4,4 )s _3,2 =s _4,1 +s _1,3 +s _5,4 +(s _4,1 +s _3,2 +s _1,3 +s _5,4 )s _6,2 =s _7,1 +s _{4, 3} +s _0,4 +s _4,4 +(s _11,1 +s _10,2 +s _8,3 +s _0,4 )+(s _3,1 +s _2,3 +s _0,3 + s _4,4 )s _7,2 =s _0,1 +s _4,1 +s _5,3 +s _1,4 +s _5,4 +(s _0,1 +s _11,2 +s _9,3 +s _1,4 )+(s _4,1 +s _3,2 +s _1,3 +s _5,4 ). Repair bits s _2,2 ,s _3,2 ,s _6,2 ,s _7,2 .

Therefore, the eight bits stored in the second information column can be downloaded by downloading six bits from the first information column and from the third information column, the fourth information column, the first parity column, and the second parity column. The four bits in each column are restored. A total of 22 bits were downloaded during the repair process. For the code given in Figure 1, the third column of information and the last column of information can be reconstructed by accessing 22 bits and 20 bits from 5 columns, respectively.

Theorem 3: When

At the time, the information column f obtained by Algorithm 1 repairs the bandwidth as

(p-1) ((k+2) 2 ^k-3 -2 ^kf-2 ).

Proof: according to algorithm 1, bit

Through the parity set of the first parity column

Repair, where l mod 2 ^f ∈{0,1,2,...,2 ^f-1 -1} and

Therefore, it is necessary to access (p-1)τ/2 bits from each column of the remaining k-1 information columns.

Where i∈{1,2,...,f-1,f+1,...,k} and l mod 2 ^f ∈{0,1,2,...,2 ^f-1 -1} And download (p-1)τ/2 redundant bits from the first parity column

Where l mod 2 ⁱ ∈ {0,1,2,..., ^2i-1 -1}. It follows that there are k(p-1)τ/2 bits that need to be downloaded.

For l mod 2 ^f ∈{2 ^f-1 , 2 ^f-1 +1,2 ^f-1 +2,...,2 ^f -1}, bit

by

Make a repair. please remember

Therefore, you need to access (p-1)τ/2 redundant bits.

For column i of i ∈ {1, 2, ..., f-1}, (p-1) τ / 2 bits are required

Where all values of l mod 2f are located in the set {0,1,...,2 ^f-1 -2 ^i-1 -1,2 ^f -2 ^i-1 ,2 ^f -2 ^i-1 +1,.. ., 2 ^f -1}. For column i of i∈{f+1,f+2,...,k}, (p-1)τ/2 bits are required.

Where l mod 2 ^f ∈ {0,1,2,...,2 ^f-1 -1}.

Please note that during the repair process, the bit

(where l mod 2 ^f ∈{0,1,2,...,2 ^f-1 -1} and

) has been downloaded through the first parity column. Thus, it is only necessary to download (p-1)τ/2 redundant bits from the second parity column and download (p-1) 2 ^k+if-3 bits from column i, where i=1, 2,...,f-1.

It can be calculated that the total number of bits that need to be downloaded from the k+2 column for the repair information column f is

when

According to Algorithm 1, the repair bandwidth of the information column k+1-f is the same as the repair bandwidth of the column f. Therefore, only consider

The situation. According to Theorem 3, as f increases, the repair bandwidth increases. When f=1, the repair bandwidth is

Thus, the optimal value is obtained in (1). Even for

In the worst case scenario, the repair bandwidth is (p-1)((k+2)2 ^k-3 -2 ^k-[k/2]-2 )<(p-1)(k+2)2 ^k-3 , which is strictly below the median of (1)

Times.

Therefore, when k is large enough, the repair bandwidth of any one information failure can be gradually optimized in (1).

It should be noted that the parity set of the first parity column in the proposed code is the same as the parity set of the first parity column in RDP and EVENODD. The key difference between the proposed code and the existing binary MDS array code is the construction of the second and third parity columns. First, the parity sets of the second and third parity columns in the proposed code do not correspond to those bits that are straight in the array, but rather to those bits that are in a broken line. Second, the number of rows in the array in the proposed code can be divisible by 2 ^k-2 . These two properties are critical to reducing the repair bandwidth.

references

[1] L. Xu and J. Bruck, "X Code: MDS Array Code with Optimal Coding" IEEE Information Theory Conference, Vol. 45, No. 1, 1999, pp. 272-276.

[2] P. Corbett, B. English, A. Goel, T. Grcanac, S. Kleiman, J. Leong, and S. Sankar, "Line-Diagonal Parity for Dual Disk Failure Correction", Section 3rd USENIX Conference Document and Storage Technology Proceedings, 2004, pp. 1–14.

[3] C. Huang and L. Xu, “STAR: An Effective Coding Scheme for Correcting Failure of Triple Storage Nodes”, IEEE Computer Report, Vol. 57, No. 7, 2008, pp. 889–901.

[4] M. Blaum, “A series of MDS array codes with minimum coding operands”, IEEE International Symposium on Information Theory, 2006, 2784–2788.

[5] A. Dimakis, P. Godfrey, Y. Wu, M. Wainwright, and K. Ramchanandran, "Network Coding for Distributed Storage Systems," IEEE Information Theory Conference, September 2010, Volume 56, Section 9. Issues 4539–4551.

[6] I. Tamo, Z. Wang and J. Bruck, “Zigzag: MDS Array Code with Optimal Reconstruction” IEEE Information Theory Conference, Vol. 59, No. 3, 2013, pp. 1597–1616.

[7] L. Xiang, Y.Xu, J. Lui, and Q. Chang, “Optimal Repair of Single Disk Failure in RDP Coded Storage Systems” ACM SIGMETRICS Performance Evaluation Edition, 2010 ACM Vol. 38, No. 1, No. 119 –130 pages.

[8]S.Xu, R.Li, PLelei, Y.Zhu, L.Xiang, Y.Xu, and J.Lui, "Single Disk Failure Recovery for X Code Base Parallel Storage Systems" IEEE Computer Report, Volume 6, Issue 4, 2014 995–1007 pages.

[9] Y. Wang, X.Yin, and X. Wang, "MDR Coding: A RAID-6 Coding Class with Optimal Reconstruction and Coding Modes" IEEE Communications Selected Region Journal, Vol. 32, No. 5, 2013 Issues 1008–1018.

[10] --, "Two new categories of double parity MDS array codes with optimal repair", IEEE Communications Letters, Vol. 20, No. 7, 2016, pp. 1293–1296.

[11] L. Pamies-Juarez, F. Blagojevic, R. Mateescu, C. Gyuot, EEGad, and Z. Bandic, “Breaking the Bound: The Real Repair Performance of MSR Coding” 14th USENIX Document and Storage Technology Conference (FAST 16), 2016, pp. 81–94.

[12] KWShum, H.Hou, M. Chen, H.Xu, and H.Li, “Regenerating Code in Binary Cyclic Codes”, IEEE Information Theory Symposium Proceedings, Honolulu, July 2014, 1046 – 1050 pages.

[13] H. Hou, K.W. Shum, M. Chen, and H. Li, "New MDS Array Codes for Multiple Disk Failures", Global Communications Conference, 2015, pp. 2369–2374.

[14]-, "BASIC Code: Low Complexity Reproduction Code for Distributed Storage Systems" IEEE Information Theory Conference, Vol. 62, No. 6, 2016, pp. 3053–369.

The above is a further detailed description of the present invention in connection with the specific preferred embodiments, and the specific embodiments of the present invention are not limited to the description. It will be apparent to those skilled in the art that the present invention may be made without departing from the spirit and scope of the invention.

Claims

A multi-fault-tolerant MDS array code coding, characterized in that the component is a C(k, 3, p) code, and the data block is represented as a k(p-1)τ information bit and encoded to generate 3 (p- 1) τ redundant bits, (p-1) τ is a positive integer, τ = 2 k-2 , p is a prime number, k ≥ 3, information bits are represented as s 0,i , s 1,i ,... , s (p-1)τ-1 ,
Redundant bits are denoted as s 0,j , s 1,j ,...,s (p-1)τ-1 ,
Where j is k+1, k+2, and k+3, i=1, 2,...,k; each (p-1)τ information bit is appended with τ extra bits and forms a message vector.
The multi-fault-tolerant MDS array code encoding according to claim 1, wherein the information column is represented by a polynomial, and each corresponding information column is a data polynomial, and the corresponding three parity columns form an encoded code. The polynomial, the data polynomial and the coded polynomial form the column vector [s 1 (x), s 2 (x), ..., s k+3 (x)].
The multi-fault-tolerant MDS array code encoding according to claim 1, wherein said column vector is obtained by an algorithm embodied in R pτ :=F 2 [x]/(1+x pτ ) [s 1 (x), s 2 (x),...,s k+3 (x)]=[s 1 (x), s 2 (x),...,s k (x)]. Where G is a kx (k+3) generation matrix composed of a kxk unit matrix I and a kx 3 coding matrix P.
The multi-fault-tolerant MDS array code encoding according to claim 1, wherein the C(k,3,p) code is a systemized linear code penetrating within R pτ .
A method for repairing a multi-faulty MDS array code, comprising: the steps of: obtaining an information column that has failed; if the information column has failed
Then fix the information bits s l,f by the first parity, where l mod 2 f ∈{0,1,2,...2 f-1 -1}, otherwise repaired by the second parity Information bits s l,f , where l mod 2 f ∈{2 f-1 , 2 f-1 +1,2 f-1 +2,...,2 f -1}; if the information column has failed
The information bits s l,f are fixed by the first parity, where l mod 2 f ∈{0,1,2,...,2 f-1 -1}, otherwise, pass the third parity To fix the information bits s l,f , where l mod2 f ∈{2 f-1 , 2 f-1 +1,2 f-1 +2,...,2 f -1}.
The repairing method according to claim 5, characterized in that the information column has failed
The repaired broadband is (p-1) ((k+2) 2 k-3 -2 kf-2 ).
The repairing method according to claim 5, wherein the parity sets of the second parity column and the third parity column do not correspond to those columns of the line in the array, but correspond to the dotted line Those columns of information; the number of rows in the array is divisible by 2 k-2 .