WO2017082750A1 - Method and apparatus for encoding data for storage - Google Patents

Method and apparatus for encoding data for storage Download PDF

Info

Publication number
WO2017082750A1
WO2017082750A1 PCT/RU2015/000758 RU2015000758W WO2017082750A1 WO 2017082750 A1 WO2017082750 A1 WO 2017082750A1 RU 2015000758 W RU2015000758 W RU 2015000758W WO 2017082750 A1 WO2017082750 A1 WO 2017082750A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
codes
matrix
code
encoding
Prior art date
Application number
PCT/RU2015/000758
Other languages
French (fr)
Inventor
Peter Vladimirovich Trifonov
Yuangang WANG
Sixiao YANG
Original Assignee
Huawei Technologies Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co., Ltd. filed Critical Huawei Technologies Co., Ltd.
Priority to CN201580084349.0A priority Critical patent/CN108352845B/en
Priority to PCT/RU2015/000758 priority patent/WO2017082750A1/en
Publication of WO2017082750A1 publication Critical patent/WO2017082750A1/en

Links

Classifications

    • HELECTRICITY
    • H03ELECTRONIC CIRCUITRY
    • H03MCODING; DECODING; CODE CONVERSION IN GENERAL
    • H03M13/00Coding, decoding or code conversion, for error detection or error correction; Coding theory basic assumptions; Coding bounds; Error probability evaluation methods; Channel models; Simulation or testing of codes
    • H03M13/37Decoding methods or techniques, not specific to the particular type of coding provided for in groups H03M13/03 - H03M13/35
    • H03M13/373Decoding methods or techniques, not specific to the particular type of coding provided for in groups H03M13/03 - H03M13/35 with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes

Definitions

  • the present invention relates to a method and an apparatus for encoding data for storage in n storage devices such that the data is recoverable after a failure of up to r storage devices and up to s block failures.
  • the invention also relates to an apparatus for recovering part-erased encoded data.
  • the present invention also relates to a computer-readable storage medium storing program code, the program code comprising instructions for carrying out a method for encoding data or for recovering part-erased encoded data.
  • n storage devices e.g. disks, NVRAM chips, etc. Any of these devices may fail either completely or partially, i.e. some data blocks (disk sectors, memory pages, etc.) of the storage device may become permanently unavailable. Erasure coding techniques are commonly used to protect the data against such failures.
  • the architecture known as Redundant Arrays of Independent Disks (RAID) solves this problem by allocating r storage devices for storing parity data, so that any failures within at most r devices can be recovered.
  • RAID Redundant Arrays of Independent Disks
  • block failures are much more frequent than device failures, and it is ex- tremely unlikely that many blocks within the same stripe fail simultaneously, provided that appropriate mapping of logical onto physical blocks is used. Hence, the redundancy of such schemes appears to be too high.
  • a sector-disk (SD) code is a (v n, v(n— r)— s) code, which is able to correct up to r column erasures, and additionally any configuration of up to s block erasures.
  • the objective of the present invention is to provide an apparatus and a method for encoding data for storage in storage devices, wherein the apparatus and the method overcome one or more of the above-mentioned problems of the prior art.
  • an objective of this invention can include providing computationally efficient erasure coding techniques for storage systems which suffer from device and block failures.
  • a first aspect of the invention provides a method for encoding data for storage in n storage devices such that the data is recoverable after a failure of up to r storage devices and up to s block failures, wherein the method comprises the steps:
  • the proposed approach to data protection in the presence of device and block failures comprises employing generalized concatenated codes (GCC) over GF(2 m ), i.e. a concatenation of one or more inner and one or more outer codes, wherein the parameters of inner and outer codes are selected so that the desired protection level is achieved.
  • GCC generalized concatenated codes
  • the proposed encoding method can include first computing global check symbols (i.e. parity symbols of the outer code), and then proceeding with calculation of local check symbols (i.e. parity symbols of the inner code).
  • the one or more outer codes and/or the one or more inner codes are Reed-Solomon codes.
  • the method of the first aspect can be implemented using any MDS codes (e.g. gen- eralized Reed-Solomon, Cauchy Reed-Solomon) as inner and outer codes.
  • Reed-Solomon codes are particularly simple to construct. In particular, if Reed-Solomon codes are used as inner codes, a low complexity encoding is possible.
  • the method of the first aspect involves multiplying the data by a binary matrix, and then by a block-diagonal matrix with blocks consisting of Galois field basis elements.
  • a binary matrix For example, cyclic load balancing similar to that used in RAID-5 may be implemented.
  • the outer-coded data is written into one or more rows of a matrix and encoding the outer-coded data is performed by applying the one or more inner codes on one or more columns, in particular all columns, of the matrix. This has the advantage that the encoding can be performed particularly efficient.
  • erator matrix of the i-th outer code C i5 and G i . is an i-th row of the generator matrix of the 0- th inner code C 0 with columns n— r, ... , n— 1 excluded.
  • This implementation has the advantage that the number of global check symbols can be reduced. Thus, the encoding can performed more efficiently.
  • the method further comprises the steps:
  • y t c ⁇ A'L
  • c (i) (c i(n _ r)> c i(n _ r)+1 , ... , c i+lKn ⁇ r ⁇ )
  • y t is an inner parity vector comprising computed inner parity symbols
  • A' A'L
  • L is a block-diagonal matrix consisting of blocks (/? 0 , ⁇ 1 , ...
  • G ( ⁇ ⁇ ') is an r-th inner generator matrix of the C r nested code in canonical form.
  • the one or more outer parity symbols are computed before the one or more inner parity symbols are computed.
  • a second aspect of the invention refers to a method for recovering data from part-erased en- coded data, wherein the encoded data has been encoded using a method according of the first aspect or one of the implementations of the first aspect.
  • a third aspect of the invention refers to an apparatus for encoding data in a storage system such that the data is recoverable after a failure of up to r out of n devices and up to 5 block failures, the apparatus comprising:
  • a first encoder for outer-coding the data with one or more outer codes to obtain outer- coded data
  • a second encoder for encoding the outer-coded data with one or more inner codes to obtain encoded data, wherein the inner codes are C;(n, n— i, ⁇ + 1) nested codes over GF(2 m ) for 0 ⁇ f ⁇ n,
  • the apparatus is configured to carry out the method of the first aspect or one of the implementations of the first aspect.
  • the first encoder and/or the second encoder is implemented in hardware, in particular as ASIC and/or FPGA.
  • the apparatus according to the third aspect is particularly suited to be implemented in hardware.
  • the method of the first and/or the second aspect can be implemented within a controller of a storage system.
  • the apparatus of the third aspect can be a controller of a storage system.
  • the controller can be implemented either in software, or in hardware (e.g. ASIC, FPGA).
  • the controller can be directly connected to the storage devices or it can be connected to the storage devices through a network connection, wherein e.g. the storage devices are connected to the network through a further controller.
  • a fourth aspect of the invention relates to an apparatus for recovering part-erased encoded data, wherein the encoded data has been encoded using the method of the first aspect or one of its implementations.
  • the apparatus can be configured to carry out the method of the first and/or second aspect.
  • an apparatus can comprise both the functionality of the first aspect and the second aspect, i.e., the same apparatus can be configured to encode data and to recover encoded data after a device and/or block failure.
  • a fifth aspect of the invention refers to a computer-readable storage medium storing program code, the program code comprising instructions for carrying out the method of the first or second aspect and/or one of their implementations.
  • FIG. 1 is a flow chart of a method according to an embodiment of the invention
  • FIG. 2 is a schematic illustration of a storage system comprising an apparatus in accordance with an embodiment of the invention
  • FIG. 3 is a schematic illustration of a method according to an embodiment of the present invention
  • FIG. 4 is a schematic illustration of a further method in accordance with a further embodiment of the present invention
  • FIG. 5 shows a performance comparison between a method according to the present invention and alternative methods.
  • FIG. 1 is a flow chart of a method according to an embodiment of the invention.
  • the method comprises a first step 1 10 of outer-coding data with or more outer codes to obtain outer-coded data.
  • the method further comprises a second step 120 of encoding the outer-coded data with one or more inner codes to obtain encoded data.
  • FIG. 2 is a schematic illustration of a storage system 200 comprising an apparatus 210 in accordance with an embodiment of the invention, wherein the apparatus 210 is connected to an array of storage devices 222, 224, 226.
  • the apparatus 210 comprises a first encoder 212 and a second encoder 214.
  • the first encoder is configured to carry out the first step 1 10 of the method shown in FIG. 1
  • the second encoder 214 is configured to cany out the sec- ond step 120 of the method shown in FIG. 1.
  • the concatenation of the inner and outer codes is in the following referred to as a generalized concatenated code (GCC).
  • a codeword of the GCC can be obtained by arranging the data into a n x v rectangular table, so that ki symbols of data are stored in the z ' -th row, encoding each row with the corresponding outer code, and encoding each column with inner code C 0 .
  • FIG. 3 is a schematic illustration of a method according to an embodiment of the present invention.
  • Payload data indicated with reference number 310, comprises symbols K0, K l , K2.
  • these symbols are outer- encoded with outer codes.
  • This outer-encoding step is performed in a systematic manner, i.e., the payload symbols are contained in the resulting codewords 330 of the outer codes.
  • the matrix 330 of outer-coded codewords comprises the payload data 332 and parity data 334.
  • the codewords of the outer codes are encoded with inner codes to obtain codewords 350 of a generalized concatenated code. These codewords are then stored on a plurality of storage devices 360. Indeed, if r devices fail, i.e. r row erasures occur, then the minimum distance of the inner codes drops to ;— r + l,j ⁇ r. Hence, the codewords with r erased columns still differ in min ⁇ r (v — K j ; + 1) (J— r + 1) > 5 positions, i.e. the code is able to recover at least s block failures.
  • the encoding scheme illustrated in FIG. 3 is non-systematic, i.e.
  • the payload data does not appear as a subvector of the codewords of the encoded data. This property may be undesired for a practical system, since it requires one to perform calculations to extract the payload data even in the absence of device failures. There is a need to construct an efficient systematic encoding algorithm, which would implement such mapping from the set of payload data vectors to the set of codewords, so that payload data appears as a subvector of a codeword.
  • Any linear code has many different generator matrices. The choice of generator matrix does not affect erasure correction capability of the code.
  • G ⁇ is a generator matrix of the i-th outer code
  • Gi _ is the i- th row of the genera- tor matrix of C 0 without inner check symbols.
  • G ' (I ⁇ A) - QG.
  • matrix A corresponds to "global" check symbols. Their number is less by rv than the total number of check symbols, and is close to s, which is typically a small value. Therefore, multiplication by A is a simple task.
  • Multiplication by matrices A and A r can be performed as follows. Let ⁇ 0 , ⁇ 1 ⁇ ... , fi m -i be a basis of GF 2 m ). A similar approach can be used for multiplication by A r , i.e. one can represent it as A r — A r L r , where A r is a binary matrix.
  • FIG. 4 is an illustration of a further method in accordance with a further embodiment of the present invention.
  • Reference number 410 indicates payload data.
  • a global check symbol calculation unit 410 which is a first encoder for outer-coding the payload data, comprises a first sub-unit 422 and a second sub-unit 424.
  • the resulting outer codewords are temporarily stored in a first row 432, a second row 434 and a third row 436.
  • the first row 432 only comprises information symbols
  • the second row comprises one information symbol and two global check symbols
  • the third row comprises two information symbols, and one local global check symbol.
  • the array of devices 460 comprises a device 0, a device 1, a device 2 and a device 3.
  • the resulting encoded data z is stored in a first row 452, a second row 454 and a third row
  • the first row comprises information symbols, that are stored on devices 0 to 2, and a local check symbol that is stored on a device 3.
  • the second row 454 comprises an information symbol that is stored on device 0 and information symbols that are stored on devices 1 to 3.
  • the third row 456 comprises information symbols that are stored on devices 0 and 1 and local check symbols that are stored on devices 2 and 3.
  • q cB L where c is the vector of non-erased symbols, B is the matrix consisting of B i;s entries. Observe that this approach requires at most tm multiplicaions.
  • computer optimization can be used to find an efficient algorithm for computing cB.
  • the construction is possible for any r, s.
  • Encoding complexity comparison for different (n, r, v, s) with STAIR codes is shown in FIG. 5.
  • STAIR codes employ Multiply and XOR operations (total number of operations is 2 times the one shown in the figure), while for methods according to the present invention (referred to as GCC in FIG. 5) multiplications and XORs are counted separately. It can be seen that the proposed approach requires smaller number of arithmetic operations compared to STAIR codes.

Landscapes

  • Physics & Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Error Detection And Correction (AREA)
  • Detection And Correction Of Errors (AREA)

Abstract

The invention provides a method for encoding data for storage in n storage devices such that the data is recoverable after a failure of up to r storage devices and up to s block failures, wherein the method involves outer- and inner-coding the data to obtain encoded data. The invention also provides a method for recovering data from part-erased encoded data.

Description

METHOD AND APPARATUS FOR ENCODING DATA FOR STORAGE
TECHNICAL FIELD
The present invention relates to a method and an apparatus for encoding data for storage in n storage devices such that the data is recoverable after a failure of up to r storage devices and up to s block failures. The invention also relates to an apparatus for recovering part-erased encoded data.
The present invention also relates to a computer-readable storage medium storing program code, the program code comprising instructions for carrying out a method for encoding data or for recovering part-erased encoded data.
BACKGROUND
Consider a storage system comprising n storage devices (e.g. disks, NVRAM chips, etc). Any of these devices may fail either completely or partially, i.e. some data blocks (disk sectors, memory pages, etc.) of the storage device may become permanently unavailable. Erasure coding techniques are commonly used to protect the data against such failures. The architecture known as Redundant Arrays of Independent Disks (RAID) solves this problem by allocating r storage devices for storing parity data, so that any failures within at most r devices can be recovered. However, block failures are much more frequent than device failures, and it is ex- tremely unlikely that many blocks within the same stripe fail simultaneously, provided that appropriate mapping of logical onto physical blocks is used. Hence, the redundancy of such schemes appears to be too high.
The problem of data protection in the presence of device and block failures can be addressed by employing partial-MDS and sector-disk (SD) array codes. Their codewords are represented as v x n arrays. Each column within an array corresponds to a storage device, and rows within a column correspond to different blocks. A partial-MDS code is defined as a (v n, v(n - r)— s) code over GF(q), which is able to correct up to Sj + rerasures in each row ij of an array corresponding to a codeword, provided that 0≤ i1 < i2 < --- < it≤v - 1, and ∑]·=ι Sj = s. A sector-disk (SD) code is a (v n, v(n— r)— s) code, which is able to correct up to r column erasures, and additionally any configuration of up to s block erasures. Computer search was used to obtain SD-codes for r < 3 and s < 3, although for r=s=3, the SD property was verified only partially due to complexity limitations. Constructions of partial-MDS codes for r = 1, s > 1 and r > 1, s = 1 were given. Explicit construction of SD-codes for the case of r > 1, s = 2 were suggested. Both of these constructions require v < n < q. A simplified construction of a SD-like code was given, which requires max(v, n) < q and can recover 1 disk failure and some combinations of up to 2 block failures. An alternative to this approach was suggested, where a two-dimensional encoding scheme is based on MDS codes. However, none of these suggested a fast algorithm for encoding the data with the corresponding code.
Some of the prior art publications do not provide efficient encoding and erasure recovery algorithms, and impose severe limitations on achievable code parameters.
SUMMARY OF THE INVENTION
The objective of the present invention is to provide an apparatus and a method for encoding data for storage in storage devices, wherein the apparatus and the method overcome one or more of the above-mentioned problems of the prior art. In particular, an objective of this invention can include providing computationally efficient erasure coding techniques for storage systems which suffer from device and block failures. A first aspect of the invention provides a method for encoding data for storage in n storage devices such that the data is recoverable after a failure of up to r storage devices and up to s block failures, wherein the method comprises the steps:
outer-coding the data with one or more outer codes to obtain outer-coded data, wherein the outer codes are Q(v, K v - Kt + 1) codes over GF(2m), v < 2m for 0 < i≤ n, wherein K = ... = Kr = 0, Kj = [v - + lj , r + 1 < < n and (v - Kj + - r>s and encoding the outer-coded data with one or more inner codes to obtain encoded data, wherein the inner codes are Cj (n, n— i, i + 1) nested codes over GF(2m) for 0 < t < n.
The proposed approach to data protection in the presence of device and block failures comprises employing generalized concatenated codes (GCC) over GF(2m), i.e. a concatenation of one or more inner and one or more outer codes, wherein the parameters of inner and outer codes are selected so that the desired protection level is achieved. The proposed encoding method can include first computing global check symbols (i.e. parity symbols of the outer code), and then proceeding with calculation of local check symbols (i.e. parity symbols of the inner code).
Both global and local check symbols can be given by expressions like z = xA, where x is some input vector, and A is a fixed matrix (typically different for local and global check symbols) consisting of GF(2m) elements. An efficient evaluation of such expressions can be based on a representation A = A L, where A is a binary matrix, and L is a block-diagonal matrix. This makes it possible to significantly reduce the number of expensive GF(2m) multiplications. Similarly, efficient approaches can be used to obtain efficient erasure recovery.
In a first implementation of the method according to the first aspect, the one or more outer codes and/or the one or more inner codes are Reed-Solomon codes.
In general, the method of the first aspect can be implemented using any MDS codes (e.g. gen- eralized Reed-Solomon, Cauchy Reed-Solomon) as inner and outer codes. Reed-Solomon codes are particularly simple to construct. In particular, if Reed-Solomon codes are used as inner codes, a low complexity encoding is possible.
One can further reduce the encoding and erasure repair complexity by employing fast algo- rithms for multiplication by matrices A, which may exploit some specific properties of inner and outer codes being used.
Preferably, the method of the first aspect involves multiplying the data by a binary matrix, and then by a block-diagonal matrix with blocks consisting of Galois field basis elements. There may be different ways to map codeword symbols onto devices. For example, cyclic load balancing similar to that used in RAID-5 may be implemented. In a second implementation of the method according to the first aspect, the outer-coded data is written into one or more rows of a matrix and encoding the outer-coded data is performed by applying the one or more inner codes on one or more columns, in particular all columns, of the matrix. This has the advantage that the encoding can be performed particularly efficient.
In a third implementation of the method according to the first aspect, the method is a method for systematic encoding of the data, further comprising a step of obtaining a generalized con- catenated code generator matrix as G = where Gw is an i-th outer gen-
Figure imgf000006_0001
erator matrix of the i-th outer code Ci5 and Gi . is an i-th row of the generator matrix of the 0- th inner code C0 with columns n— r, ... , n— 1 excluded.
This implementation has the advantage that the number of global check symbols can be reduced. Thus, the encoding can performed more efficiently.
In a fourth implementation of the method according to the first aspect, the method further comprises a step of applying Gaussian elimination to the generalized concatenated code matrix to obtain a further generalized concatenated code matrix G' = QG = (I \A)P, wherein P is a permutation matrix, / is an identity matrix and A is a matrix comprising elements Atj = ∑?=~0 ΐ Αϋϊβε » wherein Aijs ε GF(2) and β0 , β , - , βη-\ is a basis of GF(2m).
This implementation provides an efficient method for implementing systematic encoding, because it mostly involves compute a product of vector and matrix - for which very efficient implementations are available on most platforms. The total number of multiplications that need to be performed at the encoding can be reduced. In a fifth implementation of the method according to the first aspect, the method further comprises a step of computing one or more outer parity symbols as z = xAL, wherein x is a data vector of the data, z is an outer parity vector comprising computed outer parity symbols, A is a matrix comprising the Aijs values and L is a block-diagonal matrix comprising blocks βο, .,. , β^Υ, so that Λ = AL.
This represents a simple and efficient way of implementing the multiplications required in the method according to the fourth implementation. In a sixth implementation of the method according to the first aspect, the method further comprises the steps:
constructing an intermediate vector c = xG' = (x\z)P,
computing one or more inner parity symbols of the one or more inner codes as yt = c^A'L, wherein c(i) = (ci(n_r)> ci(n_r)+1, ... , c i+lKn^r}→), yt is an inner parity vector comprising computed inner parity symbols, A' = A'L, L is a block-diagonal matrix consisting of blocks (/?0 , β1, ... ,
Figure imgf000007_0001
and G = (Ι \Α') is an r-th inner generator matrix of the Cr nested code in canonical form.
This presents an efficient implementation of an encoder for the inner codes.
In a seventh implementation of the method according to the first aspect, the one or more outer parity symbols are computed before the one or more inner parity symbols are computed.
A second aspect of the invention refers to a method for recovering data from part-erased en- coded data, wherein the encoded data has been encoded using a method according of the first aspect or one of the implementations of the first aspect.
In a first implementation of the method according to the second aspect, the method comprises a step of computing c = BL, wherein is a recovered code-word, c is a vector comprising part-erased encoded data, B is a matrix that is predetermined based on one or more erasure positions, wherein B comprises elements Βψ, wherein Z?i - =∑^=o ^5β5 , Bi -S e GF ), with H'— \B) for a check matrix H' in canonical form and wherein L is a block-diagonal matrix comprising blocks (β0, ... , with β0 , βχ, ... , /?m--1 a basis of GF(2m). The first implementation presents an implementation of the data recovery method of the first aspect that is computationally particularly efficient. A third aspect of the invention refers to an apparatus for encoding data in a storage system such that the data is recoverable after a failure of up to r out of n devices and up to 5 block failures, the apparatus comprising:
a first encoder for outer-coding the data with one or more outer codes to obtain outer- coded data, wherein the outer codes are Cj(v, Kit v— Kt + 1) codes over GF(2m), v < 2m for 0 < i < n, wherein Kt = ... = Kr = 0, Kj = [v - + lj , r + 1 < < n and (v — Kj + l)(y - r) > 5, and
a second encoder for encoding the outer-coded data with one or more inner codes to obtain encoded data, wherein the inner codes are C;(n, n— i, ί + 1) nested codes over GF(2m) for 0 < f < n,
wherein in particular the apparatus is configured to carry out the method of the first aspect or one of the implementations of the first aspect.
In a first implementation of the apparatus of the third aspect, the first encoder and/or the second encoder is implemented in hardware, in particular as ASIC and/or FPGA.
Compared to other computational operations, multiplications are particularly difficult to implement in hardware. Since the first and the second encoder according to the third aspect implement methods that require a reduced number of multiplication operations, the apparatus according to the third aspect is particularly suited to be implemented in hardware.
The method of the first and/or the second aspect can be implemented within a controller of a storage system. In particular, the apparatus of the third aspect can be a controller of a storage system. To this end, the controller can be implemented either in software, or in hardware (e.g. ASIC, FPGA).
The controller can be directly connected to the storage devices or it can be connected to the storage devices through a network connection, wherein e.g. the storage devices are connected to the network through a further controller. A fourth aspect of the invention relates to an apparatus for recovering part-erased encoded data, wherein the encoded data has been encoded using the method of the first aspect or one of its implementations.
In particular, the apparatus can be configured to carry out the method of the first and/or second aspect.
It is understood that an apparatus can comprise both the functionality of the first aspect and the second aspect, i.e., the same apparatus can be configured to encode data and to recover encoded data after a device and/or block failure.
A fifth aspect of the invention refers to a computer-readable storage medium storing program code, the program code comprising instructions for carrying out the method of the first or second aspect and/or one of their implementations.
BRIEF DESCRIPTION OF THE DRAWINGS
To illustrate the technical features of embodiments of the present invention more clearly, the accompanying drawings provided for describing the embodiments are introduced briefly in the following. The accompanying drawings in the following description are merely some embodiments of the present invention, but modifications on these embodiments are possible without departing from the scope of the present invention as defined in the claims.
FIG. 1 is a flow chart of a method according to an embodiment of the invention,
FIG. 2 is a schematic illustration of a storage system comprising an apparatus in accordance with an embodiment of the invention,
FIG. 3 is a schematic illustration of a method according to an embodiment of the present invention, FIG. 4 is a schematic illustration of a further method in accordance with a further embodiment of the present invention, and
FIG. 5 shows a performance comparison between a method according to the present invention and alternative methods.
Detailed Description of the Embodiments FIG. 1 is a flow chart of a method according to an embodiment of the invention. The method comprises a first step 1 10 of outer-coding data with or more outer codes to obtain outer-coded data. The method further comprises a second step 120 of encoding the outer-coded data with one or more inner codes to obtain encoded data. FIG. 2 is a schematic illustration of a storage system 200 comprising an apparatus 210 in accordance with an embodiment of the invention, wherein the apparatus 210 is connected to an array of storage devices 222, 224, 226. The apparatus 210 comprises a first encoder 212 and a second encoder 214. Preferably, the first encoder is configured to carry out the first step 1 10 of the method shown in FIG. 1 and the second encoder 214 is configured to cany out the sec- ond step 120 of the method shown in FIG. 1.
Optionally, the one or more inner codes that are used in second step 120 are a family of nested inner codes Cj (n, n— i, dj , i.e. Ci+1 => Ct and the outer codes are a family of outer codes Cj(v, Ki, Di) over GF(q), 0≤ i < n. The concatenation of the inner and outer codes is in the following referred to as a generalized concatenated code (GCC).
A codeword of the GCC can be obtained by arranging the data into a n x v rectangular table, so that ki symbols of data are stored in the z'-th row, encoding each row with the corresponding outer code, and encoding each column with inner code C0. The dimension of the obtained code is given by K = Kt , length is N = v n, and minimum distance is δ≥
Figure imgf000010_0001
Assuming that both inner and outer codes are Reed-Solomon ones with dj = i + 1 and
D[ = v— Ki + 1, one obtains that the minimum distance of the corresponding GCC is given by d— min0 i<n (i + l)(v + 1 - K ).
Ki>0 Such code can be also considered as an instance of 2-dimensional Reed-Solomon code, i.e. its codewords can be represented as c = (c0,
Figure imgf000011_0001
ct— ( [, t), ( £, y() ε A x 2?, where A and B are some subsets of GF(q) of size v and n, respectively, and
/(*. ) = ΣΓ- ο1∑¾i_1 _1
Figure imgf000011_0002
> fij e GFfa), is the message polynomial. Generalized concatenated codes and, in particular, 2-dimensional Reed-Solomon codes, can be naturally used to implement protection against device and block failures. Assume that the system should survive r device failures and s block failures. This can be achieved by employing generalized concatenated code of length vn with inner Reed-Solomon codes Ci (n, n— i, i + 1), and outer codes Q ( , Kit v — Kt + 1), 0 < i < n, where Kj = 0, 0 < j < r, and (v - Kj + 1)0' - r + 1) > s, r≤ j < n. The i-th row of a table corresponding to a GCC codeword should be stored in the i-t device, as illustrated in FIG. 3.
FIG. 3 is a schematic illustration of a method according to an embodiment of the present invention. Payload data, indicated with reference number 310, comprises symbols K0, K l , K2. In a first processing step, indicated with arrows 320 in FIG. 3, these symbols are outer- encoded with outer codes. This outer-encoding step is performed in a systematic manner, i.e., the payload symbols are contained in the resulting codewords 330 of the outer codes. In other words, the matrix 330 of outer-coded codewords comprises the payload data 332 and parity data 334.
In a step indicated with reference number 340, the codewords of the outer codes are encoded with inner codes to obtain codewords 350 of a generalized concatenated code. These codewords are then stored on a plurality of storage devices 360. Indeed, if r devices fail, i.e. r row erasures occur, then the minimum distance of the inner codes drops to ;— r + l,j≥ r. Hence, the codewords with r erased columns still differ in min≥ r(v — Kj ; + 1) (J— r + 1) > 5 positions, i.e. the code is able to recover at least s block failures. The encoding scheme illustrated in FIG. 3 is non-systematic, i.e. the payload data does not appear as a subvector of the codewords of the encoded data. This property may be undesired for a practical system, since it requires one to perform calculations to extract the payload data even in the absence of device failures. There is a need to construct an efficient systematic encoding algorithm, which would implement such mapping from the set of payload data vectors to the set of codewords, so that payload data appears as a subvector of a codeword.
Any linear code has many different generator matrices. The choice of generator matrix does not affect erasure correction capability of the code. For any (n, k) linear code with generator matrix G one can construct another generator matrix G' = QG = (I \A)P, where Q is an in- vertible matrix, / is an identity matrix, P is a permutation matrix, and A is some k x (n—
For the sake of simplicity, /'will be omitted in what follows. Obviously, systematic encoding operation can be implemented as c = xG'— {x \xA~), i.e. one needs just to compute xA . In general, this operation costs k(n— k) operations. In the following, we present a more efficient method for systematic encoding of the considered GCCs.
Let us temporarily exclude from consideration check symbols of inner code Cr, i.e. we puncture inner code codewords at appropriate positions. The remaining code is still a GCC one with the same outer codes and punctured inner codes. Its codewords can be represented as
(n— r) x v tables. One can construct its generator matrix as G =
Figure imgf000012_0001
where G ^ is a generator matrix of the i-th outer code, and Gi _ is the i- th row of the genera- tor matrix of C0 without inner check symbols. Then one can construct another generator matrix in systematic form G ' = (I \A) - QG. Observe that matrix A corresponds to "global" check symbols. Their number is less by rv than the total number of check symbols, and is close to s, which is typically a small value. Therefore, multiplication by A is a simple task.
Having computed global check symbols, one can put them into appropriate places within the (n— r) x v table. To compute additional r rows corresponding to check symbols of Cr, one can again represent the generator matrix of Cr as Gr' = (I \Ar) = Q Gr for some non-singular matrix Q, and multiply each (transposed) column of (n— r) x v table by Ar. Hence, one obtains a n x v table being a codeword of GCC.
Multiplication by matrices A and Ar can be performed as follows. Let β0 , β ... , fim-i be a basis of GF 2m). A similar approach can be used for multiplication by Ar, i.e. one can represent it as Ar— ArLr, where Ar is a binary matrix.
FIG. 4 is an illustration of a further method in accordance with a further embodiment of the present invention.
Reference number 410 indicates payload data. A global check symbol calculation unit 410, which is a first encoder for outer-coding the payload data, comprises a first sub-unit 422 and a second sub-unit 424. The global check symbol calculation unit 410 is configured to efficiently compute the outer- coded data by having the first sub-unit 422 compute y = xA.
The first sub-unit 422 is configured to compute y = xA, wherein vector x comprises payload data, and the second sub-unit 424 is configured to compute z = yL, wherein vector z com- prises the outer-coded data. The resulting outer codewords are temporarily stored in a first row 432, a second row 434 and a third row 436. Therein, the first row 432 only comprises information symbols, the second row comprises one information symbol and two global check symbols, and the third row comprises two information symbols, and one local global check symbol.
A local check symbol calculation unit 440, which is a second encoder for encoding the outer- coded data, comprises a first sub-unit 442 configured to compute yt = XiAr and a second sub- unit 444 configured to compute zt = yiLr. The array of devices 460 comprises a device 0, a device 1, a device 2 and a device 3.
The resulting encoded data z, is stored in a first row 452, a second row 454 and a third row
456. The first row comprises information symbols, that are stored on devices 0 to 2, and a local check symbol that is stored on a device 3. The second row 454 comprises an information symbol that is stored on device 0 and information symbols that are stored on devices 1 to 3. The third row 456 comprises information symbols that are stored on devices 0 and 1 and local check symbols that are stored on devices 2 and 3. The present invention also provides a method for erasure recovery. Consider an (n, k) linear block code with check matrix H, and assume that symbols in positions j0) ... ,Λ-ι are erased. It can be assumed without loss of generality that ji = i, 0≤ i≤ t < n— k. Gaussian elimination can be used to transform check matrix into canonical form H' = QH— (I \B). This enables one to recover erased codeword symbols as ct =∑ =o BijCn-k+j · Similarly to the above considered case of encoding, one can construct expansion B^ =∑s=o 5i s/?s , 5iJS G GF 2), and compute q
Figure imgf000014_0001
cB L, where c is the vector of non-erased symbols, B is the matrix consisting of Bi;s entries. Observe that this approach requires at most tm multiplicaions. Similarly to the case of encoding, computer optimization can be used to find an efficient algorithm for computing cB. Observe that construction of matrix B and finding an efficient algorithm for multiplication by it needs to be performed only once, after the corresponding failure configuration is initially detected. In the case of generalized concatenated codes, erasure patterns consisting of at most r erasures in each column can be corrected by decoding only inner code (Cr. This not only reduces the computational complexity of erasure decoding, but also reduces the amount of data which needs to be accessed during the recovery stage. This may further improve the performance of a storage system.
The proposed approach allows simpler arithmetic implementation due to smaller required field size q = 2m≥ max(v, n)— 1 compared to prior art methods. The construction is possible for any r, s. Encoding complexity comparison for different (n, r, v, s) with STAIR codes is shown in FIG. 5. Observe that STAIR codes employ Multiply and XOR operations (total number of operations is 2 times the one shown in the figure), while for methods according to the present invention (referred to as GCC in FIG. 5) multiplications and XORs are counted separately. It can be seen that the proposed approach requires smaller number of arithmetic operations compared to STAIR codes. The foregoing descriptions are only implementation manners of the present invention, the protection of the scope of the present invention is not limited to this. Any variations or replacements can be easily made through person skilled in the art. Therefore, the protection scope of the present invention should be subject to the protection scope of the attached claims.

Claims

Method for encoding data (310) for storage in n storage devices (222-226, 360) such that the data is recoverable after a failure of up to r storage devices and up to s block failures, wherein the method comprises the steps:
outer-coding (1 10, 320) the data with one or more outer codes to obtain outer- coded data (330, 332, 334, 432, 434, 436), wherein the outer codes are
Cj(v, Kt, v - Ki + 1) codes over GF(2m), v≤ 2m for 0 < i≤ n, wherein
^ = ... = Kr = 0, Kj = [v - + lj , r + 1 < ; < n and (v - Kj + - r>s and
encoding (120, 340) the outer-coded data with one or more inner codes to obtain encoded data (350, 450), wherein the inner codes are C[ (n, n— i, i + 1) nested codes over GF(2m) for 0 < i < n.
The method of claim 1, wherein the one or more outer codes and/or the one or more inner codes are Reed-Solomon codes.
The method of one of the previous claims, wherein the outer-coded data is written into one or more rows (432, 434) of a matrix (334, 436) and encoding the outer-coded data is performed by applying the one or more inner codes on one or more columns, in particular all columns, of the matrix.
The method of one of the previous claims, wherein the method is a method for systematic encoding of the data, further comprising a step of obtaining a generalized con- catenated code generator matrix as G where is an i-th
Figure imgf000016_0001
outer generator matrix of the i-th outer code Ci t and G;__ is an i-th row of a generator matrix of a 0-th inner code (C0 with columns n— r, ... , n— 1 excluded.
5. The method of claim 4, further comprising a step of applying Gaussian elimination to the generalized concatenated code matrix to obtain a further generalized concatenated code matrix G'— QG = (I \A)P, wherein P is a permutation matrix, / is an identity matrix and A is a matrix comprising elements =
Figure imgf000017_0001
» wherein Ai]S e GF(2) and β0 , βι, - , βη-ι is a basis oi GF(2m .
The method of claim 5, further comprising a step of computing one or more outer pari ty symbols as z = xAL, wherein x is a data vector of the data, z is an outer parity vector comprising computed outer parity symbols, A is a matrix comprising the Ai]S values and L is a block-diagonal matrix comprising blocks (β0, ... , /?m_i)T, so that A = AL.
The method of claim 5 or 6, further comprising the steps:
constructing an intermediate vector c = xG' = (x\z)P,
computing one or more inner parity symbols of the one or more inner codes as yt = c A'L, wherein (i) = ( i (n_r), ci(n_r)+1< ... , c^^^), yt is an inner parity vector comprising computed inner parity symbols, A' = A'L, L is a block-diagonal matrix consisting of blocks (/?0 , β1, ... ,
Figure imgf000017_0002
and G = (Ι\Α') is an r-th inner generator matrix of an r-th inner code Cr in canonical form.
The method of claim 7, wherein the one or more outer parity symbols are computed before the one or more inner parity symbols are computed.
Method for recovering data from part-erased encoded data, wherein the encoded data has been encoded using a method according to one of the previous claims.
The method of claim 9, comprising a step of computing c = cBL, wherein c is a recovered code-word, c is a vector comprising part -erased encoded data, B is a matrix that is predetermined based on one or more erasure positions, wherein B comprises elements Bijs, wherein Bu = Βί]3β5 , Bijs 6 GF(2), with H' = (I\B) for a check matrix H' in canonical form and wherein L is a block-diagonal matrix comprising blocks (β0 βτη-χΥ, with β0 > β1 /?m_i a basis of GF(2m).
1 1. An apparatus (210) for encoding data in a storage system (200) such that the data is recoverable after a failure of up to r out of n storage devices (222-226) and up to s block failures, the apparatus comprising:
a first encoder (212) for outer-coding (1 10, 320) the data with one or more outer codes to obtain outer-coded data (330, 332, 334, 432, 434, 436), wherein the outer codes are C^v, K0 v - Kt + 1) codes over GF(2m), v < 2m for 0 < ί < n, wherein K = ... = Kr = 0, Kj = |v - + lj , r + 1 < j < n and (v -
Figure imgf000018_0001
a second encoder (214) for encoding the outer-coded data with one or more inner codes to obtain encoded data (350, 450), wherein the inner codes are C; (n, n— i, i + 1) nested codes over GF(2m) for 0 < ί≤ n, wherein in particular the apparatus is configured to carry out the method of one of claims 1 to 10.
12. The apparatus of claim 1 1, wherein the first encoder and/or the second encoder is implemented in hardware, in particular as ASIC and/or FPGA.
13. An apparatus for recovering part-erased encoded data, wherein the encoded data has been encoded using the method of one of claims 1 to 8, and wherein in particular the apparatus is configured to carry out the method of one of claims 9 or 10.
14. A computer-readable storage medium storing program code, the program code comprising instructions for carrying out the method of one of claims 1 to 10.
PCT/RU2015/000758 2015-11-10 2015-11-10 Method and apparatus for encoding data for storage WO2017082750A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201580084349.0A CN108352845B (en) 2015-11-10 2015-11-10 Method and device for encoding storage data
PCT/RU2015/000758 WO2017082750A1 (en) 2015-11-10 2015-11-10 Method and apparatus for encoding data for storage

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/RU2015/000758 WO2017082750A1 (en) 2015-11-10 2015-11-10 Method and apparatus for encoding data for storage

Publications (1)

Publication Number Publication Date
WO2017082750A1 true WO2017082750A1 (en) 2017-05-18

Family

ID=56097258

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/RU2015/000758 WO2017082750A1 (en) 2015-11-10 2015-11-10 Method and apparatus for encoding data for storage

Country Status (2)

Country Link
CN (1) CN108352845B (en)
WO (1) WO2017082750A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109257049B (en) * 2018-08-09 2020-11-06 东莞理工学院 Construction method for repairing binary array code check matrix and repairing method
WO2021163973A1 (en) * 2020-02-20 2021-08-26 Intel Corporation On-ssd erasure coding with uni-directional commands

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100138717A1 (en) * 2008-12-02 2010-06-03 Microsoft Corporation Fork codes for erasure coding of data blocks
US20140380114A1 (en) * 2013-06-20 2014-12-25 Emc Corporation Data encoding for data storage system based on generalized concatenated codes

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7484165B2 (en) * 2003-04-30 2009-01-27 Ericsson Ab Forward error correction coding
CN101779379B (en) * 2007-08-08 2014-03-12 马维尔国际贸易有限公司 Encoding and decoding using generalized concatenated codes (GCC)
WO2011062111A1 (en) * 2009-11-17 2011-05-26 三菱電機株式会社 Error correction method and device, and communication system using the same
RU2571587C2 (en) * 2014-04-10 2015-12-20 Самсунг Электроникс Ко., Лтд. Method and device for encoding and decoding data in convoluted polar code
WO2018030910A1 (en) * 2016-08-12 2018-02-15 Huawei Technologies Co., Ltd. Coding and decoding of polar codes extended to lengths which are not powers of two

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100138717A1 (en) * 2008-12-02 2010-06-03 Microsoft Corporation Fork codes for erasure coding of data blocks
US20140380114A1 (en) * 2013-06-20 2014-12-25 Emc Corporation Data encoding for data storage system based on generalized concatenated codes

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
MARIO BLAUM, STEVEN HETZLER: "Generalized Concatenated Types of Codes for Erasure Correction", INTERNET ARTICLE, 11 July 2014 (2014-07-11), XP002759583, Retrieved from the Internet <URL:http://arxiv.org/pdf/1406.6270.pdf> [retrieved on 20160707] *
MARTIN BOSSERT: "Channel Coding for Telecommunications", 1999, JOHN WILEY & SONS, pages: 287 - 298, XP002759585 *
SALIM EL ROUAYHEB, KANNAN RAMCHANDRAN: "Fractional Repetition Codes for Repairin Distributed Storage Systems", INTERNET ARTICLE, 2010, XP002759584, Retrieved from the Internet <URL:http://www.ece.iit.edu/~salim/docs/FractionalRepetition_Allerton10.pdf> [retrieved on 20160707] *

Also Published As

Publication number Publication date
CN108352845A (en) 2018-07-31
CN108352845B (en) 2021-02-23

Similar Documents

Publication Publication Date Title
US10146618B2 (en) Distributed data storage with reduced storage overhead using reduced-dependency erasure codes
US9600365B2 (en) Local erasure codes for data storage
US8739005B2 (en) Error correction encoding apparatus, error correction decoding apparatus, nonvolatile semiconductor memory system, and parity check matrix generation method
US9465692B2 (en) High reliability erasure code distribution
EP2218003B1 (en) Correction of errors in a memory array
CN111078460B (en) Fast erasure code calculation method
CN111240597B (en) Method, apparatus, device and computer readable storage medium for storing data
EP3119004B1 (en) Method, device, and computer storage medium supporting low bit rate encoding
JP4982580B2 (en) Memory system and memory system data writing / reading method
WO2012008921A9 (en) Data encoding methods, data decoding methods, data reconstruction methods, data encoding devices, data decoding devices, and data reconstruction devices
Shahabinejad et al. A class of binary locally repairable codes
US10606697B2 (en) Method and apparatus for improved data recovery in data storage systems
US9960788B2 (en) Memory controller, semiconductor memory device, and control method for semiconductor memory device
US9876511B2 (en) Memory system and memory control method
Balaji et al. On partial maximally-recoverable and maximally-recoverable codes
WO2017082750A1 (en) Method and apparatus for encoding data for storage
CN110941505A (en) Method for generating error correction circuit
US20190158119A1 (en) One-sub-symbol linear repair schemes
CN110113058B (en) Coding and decoding method, device, equipment and computer readable storage medium
WO2018029212A1 (en) Regenerating locally repairable codes for distributed storage systems
KR102532623B1 (en) Bose-chaudhuri-hocquenghem (bch) encoding and decoding method tailored for refundant array of inexpensive disks(raid), and apparatus there-of
US20170288697A1 (en) Ldpc shuffle decoder with initialization circuit comprising ordered set memory
US9236890B1 (en) Decoding a super-code using joint decoding of underlying component codes
WO2017061891A1 (en) Coding for distributed storage system
US20160043741A1 (en) Coding method and device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15866373

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15866373

Country of ref document: EP

Kind code of ref document: A1