CN113391946A - Coding and decoding method for erasure code in distributed storage - Google Patents

Abstract

The invention discloses an encoding and decoding method of erasure codes in distributed storage. The existing method based on the Van der Mongolian coding matrix has nonlinear increase of the computational complexity and influences the computational efficiency. The method firstly carries out truncation operation on the erasure code to obtain the truncation code with the same check bit as the mother code. When coding, firstly, a coding matrix composed of a whole sparse binary matrix is set, any two lines are set to zero to obtain a truncated coding matrix, and the truncated coding matrix and the information matrix are utilized to obtain a coded coding block matrix. During decoding, arranging all data of the two check blocks in the check block matrix and the data block matrix into a reconstructed coding matrix, deleting two rows with zero set, inserting row vectors of the two check blocks, multiplying the row vectors with the reconstructed coding matrix after inversion, and completing data reconstruction. The method has the advantages of less exclusive or times and high efficiency in the encoding and decoding process, and the encoding and decoding efficiency is superior to the scheme based on the Van der Menu encoding matrix.

Description

Coding and decoding method for erasure code in distributed storage

Technical Field

The invention belongs to the technical field of cloud storage, particularly relates to the technical field of distributed storage data coding, relates to an encoding and decoding method of erasure codes in distributed storage, and particularly relates to a method for performing truncation encoding and decoding on data through a binary sparse matrix.

Background

With the explosion of applications such as cloud computing, big data, 5G, edge computing technology, e-commerce, social networking, video sharing and the like, users of the internet contribute more and more contents, and the remarkable characteristics of data in scale are large volume and explosive growth, which provides a significant challenge for constructing a good storage system. The system should both guarantee the reliability of the data and provide high availability. The traditional Redundant Array of Independent Disks (RAID) cannot meet the requirement of large data storage on scalability and economic cost. Distributed storage systems have thus found extremely wide application in data storage and data management.

For distributed storage, to ensure the reliability of data, there are two main types of common fault-tolerant techniques: one is a multi-copy technique, which is fault tolerant by simple duplication; the other is erasure code fault-tolerant technology, which provides fault-tolerant capability through coding. Although the common 3-copy replication scheme is simple and easy to implement, the utilization rate of the storage space is too low, so that the storage space is wasted, and the problem that the consistency of the copies is difficult to maintain exists. In contrast, the erasure coding scheme can significantly reduce the storage space overhead while ensuring the same data fault-tolerant capability, and reduce the storage cost to a great extent, but generally adopts a scheme based on a vandermonde coding matrix, without considering the efficiency problem during encoding and decoding, and when the data set is large, a large number of matrix multiplication operations in a finite field gf (q) are used, so that the computational complexity is increased nonlinearly, and the computational efficiency is greatly influenced.

Therefore, only by providing an erasure code encoding method with high calculation efficiency and low storage overhead, the practical application problem of erasure codes in a large-scale distributed storage system can be really solved, the response speed of the system and the user experience are improved, and the storage cost is reduced.

Disclosure of Invention

The invention aims to provide an erasure code coding and decoding method in distributed storage, which adopts a scheme of binary sparse matrix and truncation and is matched with a matrix multiplication frame to carry out efficient coding and decoding on erasure codes, so that the performance can be improved, and the storage cost can be reduced.

The inventive method comprises an encoding method and a decoding method.

Firstly, truncating an erasure code, and deleting t information bits from an erasure code mother code (n, k) which has the length of n and comprises k information bits and n-k check bits to obtain a truncated code (n-t, k-t); the truncated code has the same check bits as the mother code.

The encoding method specifically comprises the following steps:

(1) setting a coding matrix consisting of binary matrices that are sparse as a whole

Wherein I is k × k of k × k

Identity matrix phi_(n-k)×kA check matrix of (n-k) × k corresponding to Ψ_n×kThe coded vector of line i is Ψ_iI ═ 1,2, …, n; truncated coding matrix

Wherein Q is_k×kFrom I_k×kSetting all the two lines to zero to obtain the product;

(2) the codewords stored at n nodes in the distributed system are represented as an n x 1 matrix C, using truncated coding matrix Ψ'_n×kAnd obtaining a coded coding block matrix C ═ psi 'from the information matrix M'_n×kX M; the information matrix M is a k x 1 column vector,

wherein, the data block d_jIs a one-dimensional array, j is 1,2, …, k;

the coded coding block matrix C comprises a data block matrix C₁And check block matrix C₂Matrix of data blocks C₁Is a column vector of k × 1, C₁＝Q_k×kX M, check block matrix C₂A column vector of (n-k) × 1, C₂＝φ_(n-k)×k×M；

Data C of ith row of coded block matrix C_iData c stored for the ith node_i＝Ψ′_i×M，Ψ′_iIs corresponding to Ψ'_n×kThe code vector of line i.

The decoding method, i.e. data reconstructionWhen the user needs to read data, the data collector extracts the data block matrix C₁All data in (2), and a secondary check block matrix C₂Two check blocks are extracted for recovering the truncated data block; the method comprises the following steps:

(3) the data collector arranges the extracted data blocks and check blocks into a reconstruction coding matrix B in a matrix form_k×1Wherein the check block is placed below the data block, reconstructing the coding matrix B_k×1A column vector of kx 1;

(4) will Q_k×kDeleting two rows with all zeroes, and adding two rows of row vectors corresponding to the two extracted check blocks into Q_k×kIn the last two rows, a k × k reconstruction matrix R is formed_k×k；

(5) For the reconstruction matrix R_k×kInverting to obtain an inverse matrix R'_k×k；

(6) Will reverse matrix R'_k×kAnd reconstructing the coding matrix B_k×1Multiplying, i.e. recovering the information matrix M, M ═ R 'comprising the truncated data blocks'_k×k×B_k×1；

(7) And the data collector combines all the data blocks into original data by sending the original data blocks to the user, and data reconstruction is completed.

Furthermore, the nodes in the distributed storage system detect the survival states of each other through communication, and in the decoding process, if node failure occurs, namely the data block stored by the node is lost, and after other nodes detect the failed node, the distributed storage system generates a new node N for replacing the failed node to store data;

the newly born node N first generates a k × k identity matrix I_k×kRemoving the row vector corresponding to the failure node, adding the row vector corresponding to any one of the two check blocks, and placing the row vector in I_k×kThe bottom row of the matrix is obtained, and a repair matrix H is obtained_k×k(ii) a Verifying whether the added row enables repair matrix H_k×kFull rank, if not, another check block is replaced;

for the repair matrix H_k×kInverting to obtain an inverse matrix H'_k×k(ii) a H 'is taken out'_k×kRow direction of middle corresponding failure nodeMeasuring to obtain a data block required for repairing the failed node, and sending a new node N;

and the newly born node N performs exclusive-OR operation on the received data blocks to obtain the lost data blocks, and the repair is completed.

The coding matrix adopted by the method is a binary coefficient matrix, and the elements in the obtained inverse matrix are also binary, so that multiplication and addition operation are not needed when the matrix is multiplied, and only simple exclusive-or operation is needed. After any row is removed by configuration, the newly added row can still enable the full rank of the reconstructed matrix array of the k rows, thereby ensuring the success of the repair decoding. Overall, the coding and decoding process of the binary sparse coding matrix only involves the exclusive or operation between data blocks without multiplication, and the coding matrix has a low density characteristic, i.e. the number of 1 s is small. Therefore, the invention has the advantages of less exclusive or times and high efficiency in the encoding and decoding process, and the encoding and decoding efficiency is superior to the scheme based on the Van der Menu encoding matrix.

Drawings

FIG. 1 is a schematic diagram of information bit truncation according to the present invention;

FIG. 2 is a diagram illustrating the structure of an encoding matrix according to the present invention;

fig. 3 is a schematic diagram of a truncated code based on a binary sparse matrix.

Detailed Description

The present invention will be described in further detail with reference to the following specific examples and the accompanying drawings.

A coding and decoding method of erasure codes in distributed storage comprises a coding method, a decoding method and data block repair.

The method firstly carries out truncation operation on Erasure Codes (EC) and deletes some information bits, thereby achieving the purpose of shortening codes: deleting t information bits from an erasure code mother code (n, k) which has the length of n and comprises k information bits and n-k check bits to obtain a truncated code (n-t, k-t); the truncated code and the mother code have the same check bits which are n-k bits.

As shown in fig. 1, in this embodiment, n is 14, k is 10, and a truncation code of (14-2,10-2) with a truncation factor t being 2 is taken as an example, that is, two information bits are deleted, and a local parity block is generated in both the horizontal and vertical directions by using an encoding frame of a product-matrix (product-matrix). Each element in the information matrix M may be considered to be an equal-sized data block. In the encoding process, according to the distribution of 0 and 1 in the encoding matrix, the data blocks corresponding to 1 in the information matrix and the encoding matrix are subjected to exclusive OR to obtain the encoding block to be stored by each node. The purpose of shortening the code can be achieved through truncation, so that the cost of a storage space is reduced, and when the reliability and the calculation efficiency are considered, the maximum truncation factor t is not more than 2. And local check blocks, namely horizontal redundancy and vertical redundancy in fig. 1, are generated in the horizontal direction and the vertical direction, so that the cost of data repair is further reduced. The method can flexibly balance storage space and reconstruction time, and can reduce the value of a truncation factor aiming at a system requiring high data reconstruction speed; for a system with strict requirement on storage space, the value of the truncation factor can be properly increased on the premise of certain fault tolerance.

The coding method specifically comprises the following steps:

(1) setting a coding matrix consisting of binary matrices that are sparse as a whole

Wherein, I_k×kIs a k × k identity matrix, phi_(n-k)×kThe check matrix is (n-k) × k, is a binary matrix corresponding to Ψ_n×kThe coded vector of line i is Ψ_i，i＝1,2,…,n；

The truncated coding matrix is:

wherein Q is_k×kFrom I_k×kAnd setting all the two lines to zero to obtain the final product.

Fig. 2 is a schematic diagram of the structure of the coding matrix, in which black squares represent 1 and white squares represent 0. It can be seen that, overall, the coding matrix has a low density characteristic, that is, the number of 1 is small, and the coding and decoding process of the binary sparse coding matrix only involves the exclusive or operation between data blocks without multiplication, so that the exclusive or times in the coding and decoding process are small, the efficiency is high, and the coding and decoding efficiency is superior to the scheme based on the vandermonde coding matrix. This embodiment sets all of rows 1 and 6 to zero as shown in fig. 3. Fig. 3 is a specific example process of encoding, in order to implement distributed storage, each encoded block of the matrix C is sent to each node in the network for storage, and when data reconstruction or repair is involved, corresponding data is retrieved from each node and corresponding operation is performed.

(2) The codewords stored at n nodes in the distributed system are represented as an n x 1 matrix C, using truncated coding matrix Ψ'_n×kAnd obtaining a coded coding block matrix C ═ psi 'from the information matrix M'_n×kX M; the information matrix M is a k x 1 column vector,

in the present embodiment, the first and second electrodes are,

wherein, the data block d_jThe number of the symbols contained in the array is determined by the size of the erasure code file and the number of the blocks, j is 1,2, …, k, and the number of the symbols

L is the file size of the erasure code, and q is the block number of the erasure code.

The coded coding block matrix C comprises a data block matrix C₁And check block matrix C₂Matrix of data blocks C₁Is a column vector of k × 1, C₁＝Q_k×kX M, check block matrix C₂A column vector of (n-k) × 1, C₂＝φ_(n-k)×k×M。

Data C of ith row of coded block matrix C_iData c stored for the ith node_i＝Ψ′_i×M，Ψ′_iIs corresponding to Ψ'_n×kThe code vector of line i.

The decoding method specifically comprises the following steps: when a user needs to read Data, a Data Collector (DC) extracts a Data block matrix C₁All data in (1) needs extra due to the truncated codeFrom the check block matrix C₂Two check blocks are extracted for recovering the truncated data block d₁And d₆. The method comprises the following steps:

(3) the data collector arranges the extracted data blocks and check blocks into a reconstruction coding matrix B in a matrix form_k×1Wherein the check block is placed below the data block, reconstructing the coding matrix B_k×1A column vector of kx 1;

(4) will Q_k×kIn this embodiment, the rows 1 and 6 are removed, and two row vectors corresponding to the two extracted check blocks are added to Q_k×kIn the last two rows, a k × k reconstruction matrix R is formed_k×kIn this embodiment, the reconstruction matrix is R_10×10(ii) a Because the two extracted check blocks respectively cover the truncated data block d₁And d₆Therefore R is_k×kFull rank and invertible;

(5) for the reconstruction matrix R_k×kInverting to obtain an inverse matrix R'_k×k；

(6) Will reverse matrix R'_k×kAnd reconstructing the coding matrix B_k×1Multiplying, i.e. recovering the information matrix M, M ═ R 'comprising the truncated data blocks'_k′_×k×B_k×1. In the present embodiment, the first and second electrodes are,

u₁and u₆For the truncated data block, c_1,1And c_1,2For two check blocks extracted, the obtained u₁And u₆I.e. recovered d₁And d₆。

(7) And the data collector combines all the data blocks into original data by sending the original data blocks to the user, and data reconstruction is completed.

The data block repair specifically includes:

(8) the nodes in the distributed storage system detect the survival states of each other through communication, if node failure occurs in the decoding process, namely a data block stored by the node is lost, and after other nodes detect the failed node, the distributed storage system generates a new node N to replace the failed node for data storage, so that the availability of the system and the reliability of data are ensured;

(9) the newly born node N first generates a k × k identity matrix I_k×kRemoving the row vector corresponding to the failure node, adding the row vector corresponding to any one of the two check blocks, and placing the row vector in I_k×kThe bottom row of the matrix is obtained, and a repair matrix H is obtained_k×k(ii) a Verifying whether the added row enables repair matrix H_k×kFull rank, if not, another check block is replaced;

(10) for the repair matrix H_k×kInverting to obtain an inverse matrix H'_k×k(ii) a H 'is taken out'_k×kObtaining a data block required for repairing the failure node according to the row vector of the corresponding failure node, and sending a new node N;

(11) and the newly born node N performs exclusive-OR operation on the received data blocks to obtain the lost data blocks, and the repair is completed.

In this embodiment, the node c is used₂Failure is illustrated as an example. When other nodes detect node c₂After the system is failed, the system generates a new node N to replace the failed node for data storage, so that the availability of the system and the reliability of data are ensured.

Precision repair of c₂The specific process is as follows:

the new node N first generates a 10 × 10 identity matrix I and removes row 2 and adds the generated parity block c_2,1Or c_2,2The corresponding row vector obtains a reconstruction matrix R, and then whether the added row can ensure the full rank of the reconstruction matrix R is verified, for the example, only c is added_2,1Can make H_10×10Full rank, resulting in a 10 × 10 repair matrix H_10×10；

N to obtain H_10×10Of inverse matrix H'_10×10And taking out H'_10×10Row vector of row 2 to obtain repair node c₂The required data block and sends a solicitation request to the correspondent node. For benli, the newborn node N is according to H'_10×10Line 2 row vector knows that a request needs to be sent to c₃、c₇、c₈And c_2,1And obtain data thereof.

N will receiveThe obtained data is subjected to XOR operation, and the failure node c can be accurately repaired₂The accurate repair is completed.

Claims (2)

1. A coding and decoding method of erasure codes in distributed storage comprises a coding method and a decoding method; the method is characterized in that: firstly, truncating an erasure code, and deleting t information bits from an erasure code mother code (n, k) which has the length of n and comprises k information bits and n-k check bits to obtain a truncated code (n-t, k-t); the truncated code and the mother code have the same check bit;

the encoding method specifically comprises the following steps:

(1) setting a coding matrix consisting of binary matrices that are sparse as a whole

Wherein, I_k×kIs a k × k identity matrix, phi_(n-k)×kA check matrix of (n-k) × k corresponding to Ψ_n×kThe coded vector of line i is Ψ_iI ═ 1,2, …, n; truncated coding matrix

Wherein Q is_k×kFrom I_k×kSetting all the two lines to zero to obtain the product;

(2) the codewords stored at n nodes in the distributed system are represented as an n x 1 matrix C, using truncated coding matrix Ψ'_n×kAnd obtaining a coded coding block matrix C ═ psi 'from the information matrix M'_n×kX M; the information matrix M is a k x 1 column vector,

wherein, the data block d_jIs a one-dimensional array, j is 1,2, …, k;

the coded coding block matrix C comprises a data block matrix C₁And check block matrix C₂Matrix of data blocks C₁Is a column vector of k × 1, C₁＝Q_k×kX M, check block matrix C₂A column vector of (n-k) × 1, C₂＝φ_(n-k)×k×M；

Data C of ith row of coded block matrix C_iData c stored for the ith node_i＝Ψ′_i×M，Ψ′_iIs corresponding to Ψ'_n×kThe coded vector of the ith line;

the decoding method is data reconstruction, when a user needs to read data, a data collector extracts a data block matrix C₁All data in (2), and a secondary check block matrix C₂Two check blocks are extracted for recovering the truncated data block; the method comprises the following steps:

(3) the data collector arranges the extracted data blocks and check blocks into a reconstruction coding matrix B in a matrix form_k×1Wherein the check block is placed below the data block, reconstructing the coding matrix B_k×1A column vector of kx 1;

(4) will Q_k×kDeleting two rows with all zeroes, and adding two rows of row vectors corresponding to the two extracted check blocks into Q_k×kIn the last two rows, a k × k reconstruction matrix R is formed_k×k；

(5) For the reconstruction matrix R_k×kInverting to obtain an inverse matrix R'_k×k；

(6) Will reverse matrix R'_k×kAnd reconstructing the coding matrix B_k×1Multiplying, i.e. recovering the information matrix M, M ═ R 'comprising the truncated data blocks'_k×k×B_k×1；

(7) And the data collector combines all the data blocks into original data by sending the original data blocks to the user, and data reconstruction is completed.

2. The encoding and decoding method of erasure coding in distributed storage according to claim 1, wherein:

the nodes in the distributed storage system detect the survival states of each other through communication, if node failure occurs in the decoding process, namely the data block stored by the node is lost, and after other nodes detect the failed node, the distributed storage system generates a new node N to replace the failed node for data storage;

the new node N first generates k × kIdentity matrix I of_k×kRemoving the row vector corresponding to the failure node, adding the row vector corresponding to any one of the two check blocks, and placing the row vector in I_k×kThe bottom row of the matrix is obtained, and a repair matrix H is obtained_k×k(ii) a Verifying whether the added row enables repair matrix H_k×kFull rank, if not, another check block is replaced;

for the repair matrix H_k×kInverting to obtain an inverse matrix H'_k×k(ii) a H 'is taken out'_k×kObtaining a data block required for repairing the failure node according to the row vector of the corresponding failure node, and sending a new node N;

and the newly born node N performs exclusive-OR operation on the received data blocks to obtain the lost data blocks, and the repair is completed.

Images