WO2014012246A1

WO2014012246A1 - Encoding, reconstructing, and recovering methods used for self-repairing code stored by distributed network

Info

Publication number: WO2014012246A1
Application number: PCT/CN2012/078927
Authority: WO
Inventors: 李挥; 吉书龙; 侯韩旭; 张华宇; 韩元波; 张卫; 姜军
Original assignee: 北京大学深圳研究生院; 深圳报业集团
Priority date: 2012-07-20
Filing date: 2012-07-20
Publication date: 2014-01-23
Also published as: CN104782101A; CN104782101B

Abstract

The present invention relates to an encoding method used for a self-repairing code stored by a distributed network, comprising the following steps of: setting a basic finite field F_q _,and obtaining a first finite field being Equation (I); obtaining a second finite field being Equation (II), F_q⊆Equation (I)⊆Equation (II); dividing, in a form of a coset being Equation (III), a space represented by the second finite field being Equation (II) into sub-spaces with the number being Equation (IV); respectively selecting t+1 basic vectors from basic vectors represented by elements of the basic finite field of each sub-space to be used as encoding vectors of one storage node, the storage node being corresponding to the sub-space, and the encoding vector being corresponding to a position of an encoding data block in a storage file; and obtaining the encoding data block on the corresponding position in the storage file according to the encoding vector of each storage node, and storing the encoding data block in the storage node. The present invention further relates to methods for reconstructing data and recovering data in a storage system adopting the foregoing encoding method. The encoding, reconstructing, and recovering methods used for a self-repairing code stored by a distributed network of the present invention have the following beneficial effects that the operation is relatively simple, and the overhead is relatively small.

Description

Coding, reconstruction and recovery method for self-repairing codes for distributed network storage

Technical field

The present invention relates to the field of distributed network storage, and more particularly to a method for encoding, reconstructing and recovering self-healing codes for distributed network storage.

Background technique

Network storage systems have received much attention in recent years, and storage systems contain different types: such as P2P-based distributed storage systems and dedicated infrastructure systems based on data center and storage area networks. Since storage node failure or file transmission loss often occurs in distributed storage systems, redundancy must be present in the network storage system. Redundancy can be achieved by copying data from a single cartridge, but the storage effect is not high, and the error correction code provides an efficient storage scheme different from previous replication. An (n, k) MDS (Maximum Distance Separable) error correcting code needs to divide an original file into k equal-sized modules, and generate n mutually uncorrelated encoding modules by linear encoding, by n nodes. Store different modules and meet the MDS properties (any k of the n encoding modules can reconstruct the original file). This coding technology plays an important role in providing effective network storage redundancy, and is particularly suitable for storing large files and archive data backup applications.

Due to node failure or file loss, system redundancy is gradually lost over time, so an equipment is needed to ensure system redundancy. The EC code (Erasure Codes) proposed in the literature [R. Rodrigues and B. Liskov, "High Availability in DHTs: Erasure Coding vs. Replication", Workshop on Peer-to-Peer Systems (IPTPS) 2005. It is more efficient in terms of storage overhead, but the communication overhead required to support redundant recovery is also relatively large. Figure 1 shows that the original file can be obtained from the existing node as long as the number of valid nodes in the system d ≥ ; Figure 2 shows the process of restoring the contents stored in the failed node. It can be seen from Figure 1 and Figure 2 that the entire recovery process is: 1) First download data from the k storage nodes in the system and reconstruct the original file; 2) Re-encode the new module from the original file, store it in the new On the node. This recovery process indicates that the network load required to repair any failed node is at least the content stored by k nodes.

There are two measures to compensate for the high communication load required for the EC code repair process: 1) Using a hybrid strategy requires an additional backup of the entire original file, so that the network load required for the repair process is equal to the lost The amount of data, but this strategy increases the storage load and makes the system complex, and the node load Can not be balanced; 2) using lazy repair (the repair process is delayed until several nodes fail to repair together) can effectively avoid the extra repair load caused by temporary failure, but delay repair may make the system vulnerable, so the system A larger amount of redundancy is required, and the repair process may be blocked during the use of limited resources on the network.

It is worth noting that the EC code was originally designed to make the communication robust, that is, the failure of some modules can be tolerated in one communication channel. Network storage treats the EC code as a black box, providing an efficient distributed data storage and a data recovery device via EC code. However, the different challenges that are not addressed in the EC code faced in network storage, especially the repair problem. In a vulnerable network, nodes may fail or go online frequently. There must be new nodes to provide coding modules to compensate for the situation when a node leaves the system (failure) and ensure system redundancy (in order to Tolerate additional node failures afterwards).

In the case where the information stored in any one module is obtained by XORing the information of two other modules, any two module information can be used to repair the third module, in the literature [A. Duminuco, E. Biersack, " Hierarchical Codes: How to Make Erasure Codes Attractive for Peer-to-Peer Storage Systems", Peer-to-Peer Computing (P2P), 2008.] proposes a HC code (Hierarchical Codes). The HC code is an iterative construct that gradually forms a large code starting from a small EC code, generated by a submodule constructed by an EC code or by an EC code. The main idea is: Consider a file of size s x k, divide the file into s subgroups, each subgroup contains k uncoded modules. An (n, k) EC code is used in each subgroup to generate n-k partial redundancy coding modules. Further, r global redundancy coding modules are generated by all s k uncoded modules by the coding scheme. Therefore, a coding group is formed, and s k uncoded modules are coded into s n+r coding modules. The local redundancy module can be used to repair the failure of the nodes in the subgroup, so only need to access the module with less than the entire file size to repair; and the global redundancy module provides further repair guarantee, that is, the module that fails in a subgroup It can be fixed by the global redundancy module when there are too many to fix itself.

Due to the asymmetry of the system structure in the HC code, the status of some modules may be higher than that of other modules, making it difficult to make an in-depth resilience analysis (affecting the understanding of coding effectiveness); Encoding requires more complex algorithms (whether refactoring or repairing); different encoding modules have different status in HC code, so the number of modules needed to repair a lost module depends not only on the number of modules lost, but also on which Module loss is related; likewise, the number of modules needed to reconstruct the original file may also be different The missing module is different.

An RGC code (Regenerating Codes) is proposed in the patent PCT/CN2012/071177, which requires only a small amount of data to be repaired for a lost coding module without first reconstructing the entire file. The RGC code uses a linear network coding technique to improve the overhead required to repair an encoding module through the NC (Network Coding Network Coding) attribute (ie, maximum stream minimum cut). The network information theory can prove the same amount of data as the lost module. Network overhead can repair the original lost module. The main idea of the RGC code is to use the MDS attribute. When some storage nodes fail, it is equivalent to storing data loss. It is necessary to download information from the existing valid nodes to regenerate the lost data and store it on the new node. Over time, many of the original nodes may fail, and some of the regenerated new nodes can re-execute the regeneration process on their own, and then generate more new nodes. Therefore, the regeneration process needs to ensure two points: 1) The failed nodes are independent of each other, and the regeneration process can be cyclically recursive; 2) Any k nodes are enough to recover the original file.

Figure 3 depicts the regeneration process when a node fails. In the distributed system, n storage nodes each store "data. When one node fails, the new node is regenerated by downloading data from other ≥ surviving nodes. The download amount of each node is a pair for each storage node i. The nodes ^X '«, ^X '.w indicate that the pair of nodes are connected by an edge whose capacity is the storage amount of the node (ie, " ). The regeneration process is described by an information flow graph, from any of the available nodes in the system. Collect each of the beta data by ^χ « ^χ ."'in ^Χ . "" stores a data, any one of the recipients can access ^X. "'. The maximum information flow from the source to the sink is determined by the minimum cut set in the graph. When the sink is to reconstruct the original file, the size of the stream cannot be smaller than the size of the original file.

There is a trade-off between the amount of storage per node and the bandwidth required to regenerate a node, thus introducing Minimum-bandwidth Regenerating (MBR) and Minimum-Storage Regenerating (MSR). For the minimum storage point, you can know that each node stores at least M/k bits, so you can derive the MSR code ( _ss , ^ _s ) = ( ― ^ ), when d takes the maximum value, that is, a newcomer and kk (d - k + Y) min ₌ M_ W _ l

When all the surviving n-1 nodes communicate, the repair bandwidth ^^ is the smallest, ie ^MSS_T '^. The MBR code has a minimum repair bandwidth, and it can be introduced to obtain a minimum repair load when d=nl. For node failure repair problems, there are usually three fixes: exact repair: the failed module needs to be constructed correctly, the recovered information is the same as the lost one (the core technology is interference queue and NC); function repair: the newly generated module can contain different from the missing Node data, as long as the repaired system supports MDS code attributes (core technology is NC); system part exact repair: is a hybrid repair model between exact repair and function repair, in this hybrid model, for system nodes ( The storage of unencoded data requires accurate recovery, ie the recovered information is the same as the information stored by the failed node. For non-system nodes (storage encoding module), no exact repair is required, only functional repair is required to make the recovered information full. MDS code attributes (core technology is interference queue and NC).

In order to apply the RGC code to the actual distributed system, even if it is not optimal, at least the data needs to be downloaded from the k nodes to repair the lost module. Therefore, even if the data transmission amount required for the repair process is relatively low, the RGC code needs to be high. The protocol load and system design (NC technology) complexity is achieved. In addition, engineering solutions are not considered in the RGC code, such as the lazy repair process, so the repair load caused by temporary failure cannot be avoided. Finally, the computational cost of the codec implementation of the NC-based RGC code is relatively large, which is one order higher than the traditional EC code.

An HSRC code (Homomorphic Self-Repairing Codes) is proposed in the patent PCT/CN2012/074837. The HSRC code mainly has the following two attributes: 1) The missing encoding module can download less than the entire file data from other encoding modules for repair; 2) The missing encoding module is repaired from a given number of modules, the given number is only It is related to how many modules are lost, and it is not related to which modules are lost. These attributes make the load of repairing a lost module relatively low. In addition, because the nodes in the system have the same status and load balancing, different lost modules can be repaired independently and concurrently in different locations of the network. In addition to the above conditions, the codeword has the following characteristics: 1) When a node fails, there may be (n-1)/2 pairs of repair nodes available for selection; 2) when there are (n-1)/2 nodes At the same time, we can still use the remaining 2 nodes of (n+1) / nodes to repair the failed nodes.

However, the encoding of HSRC codes requires computational polynomials to be relatively complex. Secondly, in HSRC codes, the coding modules are not subdividable, so the repair coding modules must also be inseparable; in addition, in order to reproduce a specific storage Node, once a node is randomly selected as a help node, and for the HSRC code, There is only one node left to choose from.

Summary of the invention

The technical problem to be solved by the present invention is to provide a computing operation with a relatively simple operation and a small overhead for distributed network storage in view of the above-mentioned problems of repairing data or reconstructing data in the prior art. Self-healing code encoding, reconstruction and recovery methods.

The technical solution adopted by the present invention to solve the technical problem is: Constructing a coding method for a self-repairing code for distributed network storage, comprising the following steps:

A) setting a basic finite field; obtaining, according to the number m of encoded data blocks in the storage file, and setting a t-extension set S formed by the t-dimensional subspace of the m-dimensional space of the second finite field F, where t+1 I m Obtaining a first finite field using the t-stretching according to the t-stretching of the basic finite field; wherein, F^FF,

B) dividing the space represented by the second finite field F into its coset form M F "

^TTTJ subspaces; where i=0, 1, ..., , w is the multiplicative group F d of the second finite field

a generator element, which is a multiplicative group of the first finite field;

C) respectively selecting t+1 encoding vectors as one storage node among the elements represented by the basic finite field elements of each subspace; the one storage node corresponding to the one subspace; the encoding a vector corresponding to a location of the encoded data block in the stored file;

D) an encoded data block corresponding to a location in the storage file obtained according to each storage node coding vector and stored in the storage node.

Further, the step B) further includes the following steps:

B1) obtaining the first finite field multiplicative group F ⁺¹ , let V be the generator of the first finite field multiplicative group F ⁺¹ ; obtaining the multiplicative group / τ of the second finite field, and setting w to be a generator of the second finite field multiplicative group /

Β 2) using the coset M F " to divide the space of the multiplicative group of the second finite field, ie

F = Ljw - , where the symbol υ denotes the division of the finite field;

Β 3) According to the above division, the elements in the second finite field F _m are represented as m-tuples. Further, the division of the second finite field multiplicative group / represents the second finite field F as a form of multiplication of elements of the first finite field multiplicative group / and the second finite field multiplicative group / τ.

Further, the step C) further includes:

C1) respectively obtaining ⁺¹ elements in each of the subspaces;

C2) were selected coding vector t + 1 th sub-space as the corresponding storage node of each of the elements in ^a sub-space.

Further, the step D) further includes:

And in the t+1 coding vectors corresponding to each of the storage nodes, sequentially acquiring the stored file coded data blocks corresponding to the positions corresponding to the items of the elements in each code vector and storing the coded data determined as the code vector The block is stored in the storage node; each storage node stores t+1 stored coded data blocks determined by t+1 coding modules.

The present invention also relates to a method of reconstructing data in a storage system employing the self-healing code encoding method described above, comprising the steps of:

I) arbitrarily select k among n storage nodes; wherein, ≥ m / (i + l);

J) downloading data of the selected node and reconstructing the data according to its coding vector;

K) judging whether the data reconstruction is completed, and if so, exiting the data reconstruction; otherwise, performing the next step;

L) arbitrarily selecting one of the storage nodes that have not been selected, so that the selected storage node is incremented by one, and returns to step J).

Further, the step J) further comprises: respectively obtaining, by the server, a coding vector of the selected storage node or obtaining the coding vector by the selected storage node.

The present invention also relates to a method for repairing a failed storage node in a storage system employing the self-healing code encoding method described above, comprising the steps of:

M) confirming that a storage node has failed and obtaining a code vector of the storage node by the server;

N) opt-in a non-failed storage node and get its code vector;

0) obtaining at least one storage node associated with the selected storage node;

P) downloading data of the selected storage node and its associated storage node, and based on these numbers According to the data of the failed storage node, it is stored in a new storage node to complete data recovery.

Further, in the step 0), the coding vector of the relevant node is obtained by performing operation on the coding vector of the failed storage node and the selected storage node, and then the related node is found; or.

Further, in the step P), the data stored by the failed storage node is obtained by reorganizing data stored by the selected storage node and the associated storage node.

The method for encoding, reconstructing and restoring the self-repairing code for distributed network storage of the present invention has the following beneficial effects: the second finite field obtained according to the number of stored file encoding modules is divided into a plurality of subspaces, and Each subspace corresponds to a storage node, and determines the location of the encoded data module stored by the storage node. When the failed node is repaired, only one storage node is selected, and the storage node corresponding to the selected storage node is found, and the storage is downloaded. The data of the node is reorganized to obtain the data stored by the failed storage node. Therefore, the calculation is relatively simple and the overhead is small.

DRAWINGS

1 is a schematic diagram of data reconstruction of an EC code in the prior art;

2 is a schematic diagram of data repair of an EC code in the prior art;

3 is a schematic diagram of data reconstruction of an RGC code in the prior art;

4 is a coding flow chart of an embodiment of a method for encoding, reconstructing, and recovering a self-healing code for distributed network storage according to the present invention;

Figure 5 is a flow chart of a method for data reconstruction in the embodiment;

6 is a flow chart of a method for data repair in the embodiment;

Figure 7 is a graphical illustration of the comparison of the static restoring force encoded in the embodiment and the static restoring force of the EC code. detailed description

The embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

As shown in FIG. 4, in the embodiment of the encoding, reconstructing and restoring method for the self-repairing code of the distributed network storage of the present invention, the encoding process includes the following steps:

Step S41: setting a basic finite field having an inclusion relationship, a first finite field, and a second finite field: In this step, first setting a basic finite field of q order, and then dividing the projective space of the basic finite field into t Dimension space, that is, do t-stretch; then get a first finite field whose order is ¹ . For the q-order finite field, q is the power of a prime p, and the m-dimensional vector on the finite field is represented as PG ( m-1 , q ), which is called the projective space. In this embodiment, the storage file to be stored in each storage node is composed of a plurality of encoded data modules, and Q^ ⁷ ^ is actually the number of encoded data modules included in the storage file. The vectors in this implementation are all row vectors. Projective space is the most unique type of geometric object in algebraic geometry. It is defined as: in the n-dimensional affine space k ⁿ on the field k, the set of all the straight lines of the origin is called the projective space on the field k. . Here the field k can take the complex field and so on. From a basic mathematical concept, a coordinate system corresponds to an affine space (Affine Space). When a vector is transformed from one coordinate system to another, linear transformation is performed. Affine Transformation „ Let P be the projective space, the t-stretch of the projective space P is the t-dimensional subspace of the projective space P, and the set of t-dimensional subspaces is S, which divides the projective space P into several t In the dimension space, each point in the projective space P belongs to only one t-dimensional subspace in the set S. If P=PG(ml, q) is a finite projective space, then the condition of t-extension exists: the number of points in the t-dimensional subspace divides the number of points in the whole space, and satisfies the charge of the equation.

The necessary condition is (i + l) lm. That is, in the projective space P = PG (ml, q), the condition that there is t-stretching is if and only if (l) lm.

Thus, a second finite field with a step of ^ is set. That is, + get 3 finite fields, the basic finite field F0 = F, the first finite field F1 = F ^ and the second finite field Ρ 2 = ^. The relationship between finite fields is

Step S42 divides the second finite field into a plurality of subspaces in a coset manner: In this step, the second finite field F2 is divided into a plurality of subspaces in a coset manner. The second finite field F2 is an m-dimensional space V operated on the basic finite field F0, and the subspace of the space V may constitute a projective space P = PG(m, q). So first The finite field Fl is the (t+1)-dimensional subspace of the space V, that is, the t-dimensional shadow space of the projective space P. The coset in the finite field is a special case of the projective space. For the second finite field F2 and the first finite field F1 of its subset, the coset is ^1, 3 2 (ie, a is the element of the second finite field F2 ), the coset divides the multiplicative group in the second finite field F2 into parts. This constitutes a t-extension of the space P. In this embodiment, the step specifically includes: obtaining a first finite field multiplicative group F ⁺¹ , let V be a generator of the first finite field multiplicative group F ⁺¹ ; obtaining a second finite field multiplicative group /, and let _w be Generator of the second finite field multiplicative group; using cosets

Μ 划分 dividing the space of the multiplicative group of the second finite field, that is, F = MAF ⁺¹ , where the symbol U represents the division of the finite field; according to the above division, the element in the second finite field F is represented as M-tuple.

Step S43: Obtain a basic vector of each subspace, and select t+1 of the coding vectors as the storage nodes corresponding to the subspace: In this step, respectively obtain the basics of each of the subspaces (ie, one of the cosets) A vector, and in which a linearly independent t+1 basic vector is selected as the coding vector of the storage node. In this embodiment, the one subspace corresponds to a storage node, and the basic vector selected by the subspace is used as the coding vector of the storage node. In the present embodiment, this step comprises: each subspace separately acquire the element ^1; t + 1 respectively take any storage node as a sub-space corresponding to the m elements in each of the subspace Coding vector.

Step S44: Obtain an encoding module of the file according to the encoding vector of each storage node and store: In this step, the encoding data module of each storage node obtained according to the above steps obtains the corresponding encoded data module, and is stored on the storage node. . Specifically, in the t+1 coding vectors corresponding to each storage node, the storage file coded data blocks corresponding to the positions of the elements having the element of 1 in each coding vector are sequentially obtained and added as the storage determined by the coding vector. The coded data block is stored in the storage node; each storage node stores t+1 stored coded data blocks determined by t+1 coding modules.

An example of a single order is: Let the basic finite field F0 = F ₂ with elements 0 and 1. Consider 1-stretching, so that the plane can be obtained, t = l, so the first finite field F1 = F ₄ , then m = 4, that is, the second finite field F2 = F ₁₆ . Take! ^; expressed as a finite field! The multiplication group of ^ is a cyclic group. Let w and V be the generators of the second finite field multiplicative group and the first finite field multiplicative group, respectively. Since the order of the element V in the second finite field F2 is 3, _v = w ⁵ , so the first finite field multiplicative group can be expressed as Fl* = {1, ^ ⁵ , ^. } , and the second finite field multiplicative group is expressed as ^2* =^^ = ' ⁺ ' . ⁺ = ^', we use the symbol U to represent the division of the second finite field F2, and the second finite field F2 into the form of the coset H'F ₄ *, i = l, ..., 5. These five cosets define five different planes. More specifically, the second finite field F2 can be directly decomposed into several basic finite field F0 addition forms: F ₁₆ = F ₄ ten vF ₄ = F ₂ ten vF ₂ ten wF ₂ ten wvF ₂ , so, the second limited The elements in the field F2 can be written as a 4-tuple. For example, the set wF ₄ * contains the elements w, wv, wv ² , since _v ² = v + l , so wv ² can be represented by the other two elements. So let w = (0, 0, 1, 0), wv = (0, 0, 0, 1), so that the plane defined by the coset wF ₄ * is {(0010), (0001), (0011)}. In a distributed storage system, the size of the file is B, and the file B needs to be stored in n storage nodes. The storage size of each storage node is ", when there is a storage node failure, the remaining connections need to be connected (n -1) d of the storage nodes and download data from each of the d nodes, represented by PSRC(n, k) as a projective self-repair code, where parameters n and k are parameters in the stretch of the construct.

For all practical configuration codeword, it is usually substantially finite field F ₂ (i.e., q = 2). First let m=B, that is, the operation domain of the element is F _2B , which is the B-dimensional vector on the finite field F ₂ . The finite field F _2B is determined by the file size B. A set of t-stretches S formed by t-dimensional subspaces of space P, where t+llB. Specifically, let the first finite field F1=F, ₊₁ . Since each subspace of the space P is a (t+1)-dimensional vector space on the finite field F ₂ , it can be represented by (t+1) vectors on the finite ^. Here t + l = «, and the number n of (t + 1) vector storage nodes stored in each storage node on the finite field F ₂ is n = ^ at the maximum. Because αΙΒ, let B=b«,

-1 So: n= (2^1) = (2 ^ba _ = i + 2« + ₍₂ « )2 +... + ₍₂ « )w. ( 1 )

(2"-1) (2"-1) In order to satisfy the client's ability to recover the original data B from k storage nodes, we need b to satisfy b ≤ k. When b = k, this is equivalent to the minimum storage of each storage node. (The first finite field F1 can have multiple choices but must satisfy t+llB). Let ^ denote the vector set of n« stored by n storage nodes, where _Vl , ..., ^ are the vectors stored by the first storage node, _Va+1 ,..., the vector stored for the second storage node, Other vectors stored by other storage nodes can be obtained. The amount of data that the i-th storage node will store is }.

In this example, there are w ⁴ = w + l, IF 1 = 15, w ¹⁵ = l, v ² = v + l, IF I = 3, v ³ = l, v = w ⁵ = w ² + w. So the space is eventually divided into

F;= {( 1000), (0110), ( 1110)}

VF;= {(0100), (0011), (0111 )}

_V ² F:= {(0010), ( 1101 ), ( 1111 )}

v ³ F:= {(0001 ), ( 1010), ( 1011 )}

v ⁴ F:= {( 1100), (0101 ), ( 1001 )}

In this example, the parameters for constructing the codeword can be obtained according to equation (1) as B = 4, a = 2, n = l + 2 ² = 5. The five storage nodes are represented by Ν,, i=l, ..., 5, each storage node stores the amount of data = 2, and the original data to be stored is used o = ( _0l , o ₂ , o ₃ , o ₄ ) said. The data stored by each storage node is as follows: Node base vector stores data

Ni Vi=( 1000 ),v ₂ =( 0110) {θι, o ₂ +o ₃ }

N ₂ v ₃ =( 0100 ), v ₄ =( 0011 ) {o ₂ , O3+O4 }

N ₃ v ₅ =( 0010),v ₆ =( 1101 ) { 03, O1+O2+O4 }

N ₄ v ₇ = ( 0001 ), v ₈ = ( 1010) {o ₄ , 01+03}

N ₅ v ₉ = ( 1100 ) , v ₁₀ = { O1+O25 O2+O4 }

(0101)

In the present embodiment, it also relates to a method of reconstructing data from a storage module obtained by storing the above method. Including the following steps:

Step S51: Select k among n storage nodes: In this step, k are randomly selected from n storage nodes storing stored file encoded data, where k≥mt + V) , where m and t It has the same meaning as in the encoding step described above.

Step S52: Download the data in the selected storage node and reconstruct the data: In this step, the data of the selected storage node is separately downloaded and the storage file is reconstructed according to the coding vectors of the storage nodes. In this implementation In the example, the server obtains the code vector of the selected storage node. In some cases, the code vector can also be obtained by the selected storage node.

Step S53 Is the reconstruction completed? It is judged whether the file reconstruction is completed, that is, whether the file is reconstructed, and if so, step S54 is executed to exit the file data reconstruction; otherwise, the process goes to step S55.

Step S54 Exit this data reconstruction: In this step, the stored file has been obtained and exited.

Step S55: Select one of the unselected storage nodes: In this step, since the data downloaded by the selected storage node does not reconstruct the file data, one of the unselected storage nodes is selected, so that The number of selected storage nodes is increased by one, and the flow jumps to step S52.

In this embodiment, if the client connects any k storage nodes, it can obtain up to 1^« data blocks and attempt to reconstruct the original data B from these data blocks. So they must satisfy k ≥ B / «. Arbitrarily select the data of k storage nodes, obviously k≥B/«, first download the data of any B/« storage nodes. If B can be decoded, the reconstruction process ends, otherwise download the data of a storage node until The original data B is decoded. In this embodiment, B is equal to m and α is equal to _t+ i. If k = 2, then the original data B can be reconstructed from any k = 2 storage nodes, in which case the PSRC (n, k) code is an MDS code. When k = 2, each storage node stores = B/2 mutually independent vectors. The vector stored by the two storage nodes N and N is arbitrarily selected as (Vi, . . . , v _a ), and the vector stored by the node N′ is ( _Ul , ... , _M J . It is assumed that there is a vector in the storage node N V , vector V is linearly related to some vectors in node N', that is, V can be written as:

^¥ =∑^ +∑^.

!=1 j=l

Because VeN and Z^ cm e N , there must be A e N , which contradicts the nature of N and N' that do not intersect in the definition of stretching. Note that when k=2, the MDS code cannot reconstruct the original data B by downloading the data of the d=2 node.

In this implementation, the method further relates to a data recovery method for recovering the code obtained by the foregoing method, including the following Steps:

Step S61: Confirming that the storage node is invalid and obtaining the coding vector of the storage node: In this step, it is confirmed that one storage node has failed, and the stored data needs to be repaired and stored on another storage node; meanwhile, obtained by the server The encoding vector of the storage node.

Step S62 selects an un-failed storage node and obtains its coding vector: arbitrarily selects a node among the non-failed storage nodes, and obtains the coding vector of the storage node from the server.

Step S63: Finding a storage node associated with the selected storage node: In this step, performing at least one storage node related to the selected storage node by performing operation on the coded vector of the failed storage node and the selected storage node The node coding vector, and then the storage node corresponding to the coding vector is found on the server; in this step, the operation taken is an exclusive OR operation.

Step S64: downloading the selected storage node and its associated storage node data, and obtaining data stored by the failed node and saving: In this step, downloading data stored by the selected storage node and its associated storage node, and according to the data Corresponding coding vector (including the coding vector of the failed storage node, selecting the coding vector of the storage node and the coding vector of the above-mentioned storage node), reorganizing the data, obtaining the data stored by the failed node, and storing it on a new storage node .

In the case of a storage node failure, for a HSRC code, when a storage node fails, data of two storage nodes needs to be downloaded to repair the data of the failed node, and (n-1)/2 pairs of repair nodes are available for selection. The PSRC code still has this repair feature.

In the PSRC (n, k) code, there are a total of n storage nodes, and each storage node stores "the amount of encoded data. When a storage node ^ fails, we can recover the data stored by the failed node by connecting and downloading d=2 storage nodes. Specifically, among the existing (n-1) storage nodes, arbitrarily selecting one storage node, at least one storage node Nj can recover the data stored by the failed node by downloading the data of the storage node and Nj.

The subspace of the first storage node ^ is of the form v'F , 1 = 1, ..., n. Assuming the storage node fails, a new node replaces the failed node Ni. The new node selects any storage node. For example, the data stored by the N _l storage node is νΤ. We need to prove that there is at least one storage node Ν" so that ^V ' ^F ₂ : U ^VF ; can repair the data stored in the node Ni. Because (v' + v')F ₂ *„c _V 'F ₂ *„Lk'F ₂ *„ , so we You can choose j to make =(!'+. By storing the data stored in the storage node ^ and 相互 in combination with each other, we can get _V 'F ₂ U(v'+v')F ₂ , which is the invalid data v' F ₂ *.

That is to say, when any one storage node ^ fails, only one storage node and another corresponding node need to be arbitrarily selected to recover the invalid data.

The self-repair capability of the PSRC code is stronger than the self-repair capability of the HSRC code. Suppose the number of storage nodes is n=21, the amount of data stored in each storage node is =2, the size of the original data is B=6, and the codeword we construct is PSRC (21, 3). If the node fails, the rest is left. Any one of the 20 storage nodes selects one storage node, and there will be another three storage nodes N., N. and N., by downloading the storage node and or the data in N ₂ or ^ and ₃ Restore the data stored in the failed node.

Note that w is the generator of the loop group F:, the data stored by the storage node is v'F:, and the data stored by the storage node is v'F:. So have

i . i , I , I i i i , i I I I , I

V + V w + v +v w,v ,v w,v +v w,v ,v w,v +v w,

V + v ^l w, V + v ^l w + v ^l , v ^l w + v ^l + v ^l w, v ^l w + v ^l ,

V +v ^l w + v ^l w,v ^l +v ^l +v ^l w]

Let jl, j ₂ , j ₃ satisfy the following formula respectively

ν ^Λ =ν ^ι +ν ^ι , v ^Jl = V +v ^l w , V ^h = V +v'w

So have

(N _t , Ν _Α ) νΤ ₄ * (J (ν' + ν ^ι )F ₄ *] v'F ₄ *, (N _; , N ) =>V'F (J (ν' + V'W) F 3 v'F ₄ *, (N _; , N ) =>v'F U (ν' + V'W)F 3 v'F ₄ *. This method is actually an algorithm for finding different pairs of repair nodes. .

For the example listed in the above encoding step, if the node ^ fails, then the data block (equivalent to the basic vector (1000)) and o ₂ + o _{3 are} lost (equivalent to the basic vector (0110)) . The new node will join the storage system and connect the storage nodes Ν ₃ and Ν ₄ , from which the basics can be obtained separately. Vector v ₅ = (0010), v ₆ = ( 1101 ) and v ₇ = (0001 ), v ₈ = ( 1010). Further, (1000) can be obtained by v ₈ + v ₅ , and (0110) can be obtained by v ₈ + ( v ₆ + v ₇ ). On the other hand, assuming node ^ fails, the new node connects to node N ₄ and downloads _V ³ F: , there is

1 , ii , 3 21 11 _τ

v+v=l+v=v v =^·

1 . i . 3 21 9

v w+v =w+v =vv J ₁₀

1 i 3 21

V + VW = 1 + V W = V N5.

Therefore, the new node can repair the data stored by the failed node (N ₄ , N ₁₂ ), (N ₄ , N ₁₀ ) by connecting and downloading any of the following three pairs of nodes (the three pairs of nodes all contain the node N ₄ ), ( N ₄ , N ₅ ).

In addition, in a distributed storage system, static resilience refers to the probability that once the data is stored in the system, the stored original data can be recovered without further repairing the failed node. Let p _{n be} . _De is the effective probability of any given node. Since there are no two different data modules stored in the same node in the system, we can assume that the validity of the module stored by any node is p _n . _De . The probability p that the original data can be recovered. The probability of _b j is =^ _x 3⁄4^(l-;^y", where there is only one conditional probability, which is the probability that the original data can be recovered by downloading data from any of the n storage nodes.

For the (n, k) MDS erasure code, it is deterministic and its value is equal to 1 when x ≥ and its value equals 0 in other cases. But for self-healing codes, the value is undefined. In the example of the PSRC (21, 3) code constructed in this paper, the probability value can be calculated. For the case of x ≥, 1-A can be calculated by the exhaustive method.

For the specific calculation method, we can exhaust a unique group of five storage nodes, in which 10 basic vectors produce a matrix with a rank less than 6, and a unique group of 5 storage nodes is =20349. That is

5) Say, if we choose 5 storage nodes arbitrarily, the probability that the original data still cannot be recovered is 0.00083. Similarly, if we choose any of the 3 storage nodes, the probability of not recovering the original data is 0.150375. Conversely, for the MDS code, downloading the data of any three storage nodes can restore the original data. However, the inferiority of self-healing codes in recovering data has resulted in efficient self-healing capabilities. For any one storage node, if the node fails, arbitrarily choose to multiply one of the existing 20 storage nodes, and regenerate the invalid data by selecting any one of the three storage nodes corresponding thereto. . Figure 7 compares the probability of static restoring forces for PSRC (21, 3) codes and MDS (21, 3) codes. The values in the figure are calculated by the computer by evaluating the value of ^. From this figure we can see that the MDS code may not have any (n, k) characteristics. More importantly, although the PSRC (21, 3) code loses a bit of static resilience, it has more self-healing capabilities than the MDS code. It is not to be understood as limiting the scope of the invention. It should be noted that a number of variations and modifications may be made by those skilled in the art without departing from the spirit and scope of the invention. Therefore, the scope of the invention should be determined by the appended claims.

Claims

claims

1. A self-healing code encoding method for distributed network storage, characterized by including the following steps:

A) Set the basic finite field ^; According to the number m of encoded data blocks in the storage file, obtain the second finite field F, and set the t-stretch set S formed by the t-dimensional subspace of the m space of the second finite field F , where t+1 I m; according to the t-stretch of the basic finite field, the first finite field is obtained by using the t-stretch; where,

B) Divide the space represented by the second finite field F into m 1 mi using the form of its coset νι P ⁺¹

^TTTJ subspace; where, i=0, 1, ..., , w is the generation of the multiplicative group of the second finite field

In yuan, F ⁺¹ is the multiplicative group of the first finite field;

C) Select t+1 encoding vectors as one storage node among the elements represented by the basic finite field elements in each subspace respectively; the one storage node corresponds to the one subspace; the encoding a vector corresponding to the location of the encoded data block in the storage file;

D) Obtain the encoded data block corresponding to the position in the storage file according to the encoding vector of each storage node and store it in the storage node.

2. The encoding method of self-healing code for distributed network storage according to claim 1, characterized in that the step B) further includes the following steps:

B1) Obtain the first finite field multiplicative group, let V be the generator of the first finite field multiplicative group F ⁺¹ ; Obtain the multiplicative group F » of the second finite field, let w be the second The generator of "finite field multiplicative group";

B2) Use the coset νι P ⁺¹ to divide the space of the multiplicative group of the second finite field, that is

F - = LJw - , where the symbol υ represents the division of the finite field;

B3) Based on the above division, represent the elements in the second finite field F as m-tuples.

3. The encoding method of self-healing codes for distributed network storage according to claim 2, characterized in that the division of the second finite field multiplicative group F represents the second finite field F _QM as The first finite The form in which the elements of the field multiplicative group and the second finite field multiplicative group / are multiplied respectively.

4. The encoding method of self-healing code for distributed network storage according to claim 3, characterized in that the step C) further includes:

C1) Obtain t+1 basic vectors in each subspace respectively; The subspace includes ¹ element;

C2) respectively uses t+1 basic vectors as the encoding vectors of the storage nodes corresponding to the subspace.

5. The encoding method of self-healing code for distributed network storage according to claim 4, characterized in that the step D) further includes:

Among the t+1 coding vectors corresponding to each storage node, the storage file coding data blocks corresponding to the items whose elements are 1 in each coding vector are sequentially obtained and added together as the storage coding data determined by the coding vector. The blocks are stored in the storage node; each storage node stores t+1 storage coded data blocks determined by t+1 coding modules.

6. A method for reconstructing data in a storage system using the self-healing code encoding method as claimed in claim 1, characterized in that it includes the following steps:

I) Randomly select k storage nodes among n; among them, ≥m/(i + l);

J) Download the data of the selected node and reconstruct the data based on its encoding vector;

K) Determine whether the data reconstruction is completed, if so, exit this data reconstruction; otherwise, perform the next step;

L) Select any storage node that has not yet been selected, increasing the number of selected storage nodes by one, and return to step J).

7. The method of reconstructing data according to claim 5, wherein step J) further includes: obtaining the encoding vector of the selected storage node by the server or obtaining its encoding by the selected storage node. vector.

8. A method for repairing failed storage nodes in a storage system using the self-healing code encoding method as claimed in claim 1, characterized in that it includes the following steps:

M) Confirm that a storage node has failed and obtain the encoding vector of the storage node from the server; N) Select a non-failed storage node and obtain its encoding vector; O) Obtain at least one storage node related to the selected storage node;

P) Download the data of the selected storage node and its related storage nodes, and obtain the data of the failed storage node based on these data, and store it in a new storage node to complete data recovery.

9. The method according to claim 8, characterized in that, in step 0), the coding vector of the relevant node is obtained by operating the coding vector of the failed storage node and the selected storage node, and then the coding vector is searched. to the relevant node; the operation is XOR.

10. The method according to claim 9, characterized in that in step P), the data stored in the failed storage node is obtained by reorganizing the data stored in the selected storage node and related storage nodes. .