CN110781163A - Heterogeneous part repeated code construction and fault node repairing method based on complete graph - Google Patents
Heterogeneous part repeated code construction and fault node repairing method based on complete graph Download PDFInfo
- Publication number
- CN110781163A CN110781163A CN201910930841.5A CN201910930841A CN110781163A CN 110781163 A CN110781163 A CN 110781163A CN 201910930841 A CN201910930841 A CN 201910930841A CN 110781163 A CN110781163 A CN 110781163A
- Authority
- CN
- China
- Prior art keywords
- node
- coding blocks
- nodes
- blocks
- complete graph
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention belongs to the field of computers, and discloses a heterogeneous partial repeat code construction method and a fault node repair method based on a complete graph. The method mainly comprises the steps of carrying out MDS coding on data blocks of an original file to generate coding blocks, and constructing heterogeneous part repeated codes with different coding block repetition degrees according to the relation of vertexes, edges and angles of a complete graph. In the single-node fault or two continuous node fault repairing process, only the coding blocks are collected from m-2 surviving nodes, and the collected coding blocks are transmitted to a new node, so that the repairing of the fault node can be realized. In the process of repairing the fault node, the number of the connected active nodes and the disk I/O expense and repair time for repairing the fault node are reduced. Various repair schemes can be selected for repairing single node faults or any two continuous node faults, and the availability and the reliability of the system are effectively improved.
Description
Technical Field
The invention belongs to the field of computers, and particularly relates to a heterogeneous part repeated code construction and fault node repairing method based on a complete graph.
Background
Currently, internet technology has been rapidly developed, information technology has penetrated into various fields of social life, and is stepping from the computing age to the storage age, which has led to a dramatic increase in data. The traditional data storage scheme cannot be adapted to the storage of the current mass data, and the distributed storage system gradually becomes the mainstream data storage mode. Typically, distributed storage systems employ "replication" and "erasure coding" techniques to generate some redundant data, but the storage cost generated by the "replication" redundancy strategy is too high. The erasure code can optimize the storage overhead of the distributed storage system, but the whole file must be restored by repairing a single fault node, and the bandwidth overhead is too large. In response to the limitations of the "copy" and "erasure code" redundancy strategies, Dimakis et al propose regeneration codes. The regeneration code has a large number of connected nodes in the fault node repairing process and high disk I/O overhead. The partial repeated codes integrate the copying and regeneration code technology, so that only a few data blocks need to be downloaded from partial survival nodes when the failed nodes are repaired, and the downloaded data blocks are transmitted to new nodes without other operation. However, most conventional partial repetition code constructions are based on homogeneous distributed storage systems. In fact, in distributed storage systems, users tend to have unbalanced access to data, with "hot" data often being accessed and "cold" data being rarely accessed. If the conventional partial repetition code with the same data repetition degree is adopted, not only the storage overhead is increased, but also bottleneck problems such as 'hot data' access congestion and the like are caused.
Disclosure of Invention
The invention aims to provide a complete graph-based heterogeneous part repeated code construction method and a fault node repair method, which are used for solving the problems that in the prior art, the storage overhead of a homogeneous part repeated code is high, hot data access congestion is easily caused, the code construction method is complex, the repair locality is high in the node repair process, the node repair selectivity is low, and the like.
In order to realize the task, the invention adopts the following technical scheme:
the heterogeneous partial repetition code construction method based on the complete graph comprises the following steps:
step 1: dividing an original file into k original data blocks, and performing (n, k) MDS coding on the k original data blocks to obtain n coding blocks C
1,C
2,…，C
nWherein n and k are positive integers;
step 2: establishing a complete graph comprising m vertexes, numbering the m vertexes of the complete graph in sequence by using 1, 2, …, wherein m is the number of the m vertexes of the complete graph in the clockwise direction, numbering the m edges at the outermost layer of the complete graph in sequence by using 1, 2, … and m in the clockwise direction from the No. 1 vertex, and then numbering the edges inside the complete graph in sequence by using m +1, m +2, … and n in the clockwise direction from the vertex 1, wherein n is m (m-1)/2, and m is not less than 5;
and step 3: divide the complete map into m sets S
iI 1, 2, …, m, set S
iCorresponding to the vertexes i and S
iThe three opposite sides are the sides of the outermost layer of the complete graph corresponding to three internal included angles, and the three internal included angles are included angles which are formed by the first four sides which are connected with the vertex in the clockwise direction from the vertex and are adjacent in pairs;
and 4, step 4: divide the complete graph into m sets E
iSet E of
iCorresponding vertices i and E
iContaining m-3 sides, said E
iThe side in (1) is formed by connecting m-3 vertexes which are not adjacent to the vertex i with the vertex i;
and 5: 1 to m vertexes are respectively corresponding to 1 to m nodes, for a node i, three coding blocks with the repetition degree of 3 are stored firstly, and the numbers of the three coding blocks sequentially correspond to S
iThe serial numbers of three opposite sides in the code block are stored, then m-3 code blocks with the repetition degree of 2 are stored, and the serial numbers of the m-3 code blocks sequentially correspond to E
iAnd numbering m-3 edges in the code block to finally obtain m nodes, wherein m coding blocks are stored in each node, and the structure of the heterogeneous part repeated code is completed.
Further, step (ii)Set S in 3
iThe construction method comprises the following steps:
for vertex 1, construct S
1，S
1The three opposite sides of the first four sides L are respectively in the counterclockwise direction from the vertex and share the vertex
m、L
m+1、L
m+2And L
m+3In two adjacent edges L
mAnd L
m+1、L
m+1And L
m+2And L
m+2And L
m+3The outermost side L of the complete picture corresponding to the included angle
m-1、L
m-2And L
m-3To obtain S
1＝{L
m-1、L
m-2、L
m-3And f, successively analogizing to obtain a set S corresponding to all vertexes
i。
The fault node repairing method comprises the following steps:
according to any one of the complete graph-based heterogeneous partial repetition code construction methods, an original file containing n coding blocks is stored in m nodes of a distributed storage system, each node stores m coding blocks, and each node stores 3 coding blocks with the repetition degree of 3 and m-3 coding blocks with the repetition degree of 2;
when a single node fails, acquiring coding blocks from m-2 surviving nodes, and transmitting the acquired coding blocks to a new node to complete the repair of the single node failure;
when any two continuous nodes have faults, the coding blocks are collected from m-2 surviving nodes, and the collected coding blocks are transmitted to two continuous new nodes, so that the fault repair of any two continuous nodes is completed.
Compared with the prior art, the invention has the following technical characteristics:
1. given that distributed storage systems tend to be unbalanced to access data. The coding blocks with different access frequencies correspond to different repetition degrees, and the parallel access speed of the hot data can be improved by adopting the copy with higher repetition degree for the hot data. For the cold data, the storage efficiency of the distributed storage system can be improved by adopting the replication with lower repetition degree.
2. When a single node fails, four fault repairing schemes are provided, and only the coding blocks are collected from m-2 surviving nodes and the collected coding blocks are transmitted to a new node, so that the repairing of the failed node can be rapidly realized. The repairing process is simple in operation and small in repairing locality.
3. Two node failures can be tolerated. When any two continuous nodes have faults, two repair schemes are available, and the repair of the fault node can be quickly realized by only acquiring the coding blocks from m-2 surviving nodes and transmitting the acquired coding blocks to a new node.
4. The storage rate of the system is high. In addition, the part of repeated codes is simple in construction process, easy to popularize and strong in operability.
Drawings
Fig. 1 is a schematic configuration diagram of a storage system having 5 nodes and 10 coding blocks in an embodiment, each node storing a heterogeneous partial repetition code of 5 coding blocks;
FIG. 2 is a single node v containing 5 nodes and 10 coding blocks in an embodiment
1A schematic of fault repair;
fig. 3 is a schematic configuration diagram of a storage system having 6 nodes and 15 encoded blocks in an embodiment, each node storing a heterogeneous partial repetition code of the 6 encoded blocks;
FIG. 4 is a node v including 6 nodes and 15 coding blocks in an embodiment
1And node v
2Schematic diagram of fault repair.
Detailed Description
Example 1
The embodiment discloses a heterogeneous partial repeat code construction method based on a complete graph, which comprises the following steps:
step 1: dividing an original file into k original data blocks, and performing (n, k) MDS coding on the k original data blocks to obtain n coding blocks C
1,C
2,…，C
nWherein n and k are positive integers;
step 2: establishing a complete graph comprising m vertexes, numbering the m vertexes of the complete graph in sequence by using 1, 2, …, wherein m is the number of the m vertexes of the complete graph in the clockwise direction, numbering the m edges at the outermost layer of the complete graph in sequence by using 1, 2, … and m in the clockwise direction from the No. 1 vertex, and then numbering the edges inside the complete graph in sequence by using m +1, m +2, … and n in the clockwise direction from the vertex 1, wherein n is m (m-1)/2, and m is not less than 5;
and step 3: divide the complete map into m sets S
iI 1, 2, …, m, set S
iCorresponding to the vertexes i and S
iThe three opposite sides are the sides of the outermost layer of the complete graph corresponding to three internal included angles, and the three internal included angles are included angles which are formed by the first four sides which are connected with the vertex in the clockwise direction from the vertex and are adjacent in pairs;
and 4, step 4: divide the complete graph into m sets E
iSet E of
iCorresponding vertices i and E
iContaining m-3 sides, said E
iThe side in (1) is formed by connecting m-3 vertexes which are not adjacent to the vertex i with the vertex i;
specifically, two edges formed by the node m and the node 2 connected to the node 1 are removed from the node 1, and the number of the edge formed by the node 1 and the other nodes is recorded as a set E
1Will be set E
1The coding blocks corresponding to the numbers are stored in the node 1; similarly, two edges formed by the node 1 and the node 3 connected with the node 2 are removed, and the number of the edge formed by the node 2 and other nodes is recorded as a set E
2Will be set E
2The coding blocks corresponding to the numbers are stored in the node 2; according to the rule, two edges formed by the node m-1 and the node 1 connected with the node m are removed, and the number of the edge formed by the node m and other nodes is recorded as a set E
m。
And 5: 1 to m vertexes are respectively corresponding to 1 to m nodes, for a node i, three coding blocks with the repetition degree of 3 are stored firstly, and the numbers of the three coding blocks sequentially correspond to S
iThe serial numbers of three opposite sides in the code block are stored, then m-3 code blocks with the repetition degree of 2 are stored, and the serial numbers of the m-3 code blocks sequentially correspond to E
iAnd numbering m-3 edges in the code block to finally obtain m nodes, wherein m coding blocks are stored in each node, and the structure of the heterogeneous part repeated code is completed.
Specifically, set S in step 3
iThe construction method comprises the following steps:
for vertex1, construction of S
1，S
1The three opposite sides of the first four sides L are respectively in the counterclockwise direction from the vertex and share the vertex
m、L
m+1、L
m+2And L
m+3In two adjacent edges L
mAnd L
m+1、L
m+1And L
m+2And L
m+2And L
m+3The outermost side L of the complete picture corresponding to the included angle
m-1、L
m-2And L
m-3To obtain S
1＝{L
m-1、L
m-2、L
m-3And f, successively analogizing to obtain a set S corresponding to all vertexes
i。
The invention aims to provide a heterogeneous partial repetition code construction method based on a complete graph, and proposes that coding blocks with different access frequencies correspond to different repetition degrees, namely, the coding blocks adopt the copy with higher repetition degree for 'hot data' and adopt the copy with lower repetition degree for 'cold data'. Network resources are reasonably utilized to improve the parallel access speed of the hot data and the storage efficiency of the storage system. In addition, the system based on the heterogeneous part repeated codes of the polygon structure has high storage utilization rate, and the code structure process is simple, easy to popularize and strong in operability.
The embodiment also discloses a method for repairing a failed node, which comprises the following steps:
according to any one of the complete graph-based heterogeneous partial repetition code construction methods, an original file containing n coding blocks is stored in m nodes of a distributed storage system, each node stores m coding blocks, and each node stores 3 coding blocks with the repetition degree of 3 and m-3 coding blocks with the repetition degree of 2;
when a single node fails, acquiring coding blocks from m-2 surviving nodes, and transmitting the acquired coding blocks to a new node to complete the repair of the single node failure;
when any two continuous nodes have faults, the coding blocks are collected from m-2 surviving nodes, and the collected coding blocks are transmitted to two continuous new nodes, so that the fault repair of any two continuous nodes is completed.
In a distributed system storing mass data, a node failure has become a normal state. If the traditional partial repetition code is adopted, such as the partial repetition code constructed based on the regular graph, when the node fails, the repair scheme of the failed node is single, the I/O overhead of the repaired disk is high, meanwhile, the fault-tolerant capability of the system is low, and the requirement of mass data storage on the node is not met. In addition, the conventional partial repetition code with the same data repetition degree not only increases the storage overhead, but also causes bottleneck problems such as access congestion of "hot data". For this purpose, the invention is based on the heterogeneous partial repetition code of the complete graph, and the repetition degree of the coding blocks of different access frequencies is different. In the single-node fault and continuous two-node fault repairing processes, only the coding blocks are collected from m-2 surviving nodes, and the collected coding blocks are transmitted to a new node, so that the repairing of the fault node can be realized. To a certain extent, the number of connected and stored movable nodes is reduced, and the I/O overhead and the repair time of a repair disk for repairing a fault node are reduced. And various repair schemes are provided for repairing the single-node fault and any two continuous fault nodes, so that the system pressure is effectively relieved. In addition, the part of repeated codes requires fewer check blocks, and the storage utilization rate of the system is high.
Example 2
The embodiment discloses a method for constructing a heterogeneous partial repetition code based on a complete graph, and further discloses the following technical characteristics on the basis of the embodiment 1, wherein fig. 1 shows a storage scheme of a distributed storage system comprising 10 coding blocks, and each node stores the heterogeneous partial repetition codes of 5 coding blocks.
Step 1: storing a file with the size of 9M in a distributed storage system, firstly carrying out (10,9) MDS encoding on an original file, and then representing the 10 encoding blocks by 1, 2, 3 and … 10;
step 2: drawing a complete graph containing 5 vertexes, taking the 5 vertexes of the complete graph as 5 vertexes of the distributed storage system, sequentially numbering the 5 vertexes of the complete graph and 5 edges of the outermost layer by 1, 2, … and 5, and sequentially numbering the other edges inside the complete graph by 6,7, … and 10;
and step 3: an edge connecting the node 1 and the node 5 in the clockwise direction is taken as a starting edge, andthe number of the edge and the adjacent three edges at the right side is taken as a set s
1I.e. s
11, {5,6,7,1 }. The four edges and the node 1 form three included angles, the three included angles are respectively read and correspond to the edge numbers of the outermost layer of the complete graph, namely {2, 3, 4}, the numbers correspond to the numbers of partial coding blocks stored in the node 1, and then the partial coding blocks stored in the first node are: c
2、C
3、C
4. Similarly, the edge connecting node 2 and node 1 is taken as the starting edge, and the number of the edge and the three adjacent edges on the right side of the edge are taken as the set s
2I.e. s
21,8,9, 2. The four edges and the node 2 form three included angles, the three included angles are respectively read and correspond to the edge numbers of the outermost layer of the complete graph, namely {3, 4, 5}, the numbers correspond to the numbers of the coding blocks stored in the node 2, and then the partial coding blocks stored in the second node are as follows: c
3、C
4、C
5. According to the rule, the part of the coding blocks stored by the third node is as follows: c
1、C
4、C
5(ii) a The fourth node stores part of the coding blocks as follows: c
1、C
2、C
5(ii) a The fifth node stores part of the coding blocks as follows: c
1、C
2、C
3；
And 4, step 4: starting from node 1, two edges formed by node 5 and node 2 connected to node 1 are removed, and the number of the edge formed by node 1 and other nodes is recorded as a set E
1I.e. E
1Set E to {6,7}, set E
1The code blocks corresponding to the number of each edge are stored in the node 1, namely the code block C
6、C
7Stored in the first node. Similarly, two edges formed by the node 1 and the node 3 connected with the node 2 are removed, and the number of the edge formed by the node 2 and other nodes is recorded as a set E
2I.e. E
2Set E to {8,9}, set E
2The code blocks corresponding to the number of each edge are stored in the node 2, namely the code block C
8、C
9And storing the data in the second node. According to the rule, the code block C is coded
7、C
10Storing the data into a third node, and coding a block C
6、C
9Storing the data into a fourth node to encode a block C
8、C
10And storing into the fifth node.
And 5: finally, 5 heterogeneous partial repetition codes with the node storage capacity of 5 can be obtained, and each node stores 3 coding blocks with the repetition degree of 3 and 2 coding blocks with the repetition degree of 2. The coding block stored by the first node is: c
2、C
3、C
4、C
6、C
7(ii) a The second node stores the coding blocks as follows: c
3、C
4、C
5、C
8、C
9(ii) a The third node stores the coding blocks as follows: c
1、C
4、C
5、C
7、C
10(ii) a The fourth node stores the coding blocks as follows: c
1、C
2、C
5、C
6、C
9(ii) a The coding block stored by the fifth node is: c
1、C
2、C
3、C
8、C
10。
The embodiment also discloses a single-node fault repairing method, and fig. 2 shows a single node v in the embodiment
1Schematic diagram of fault repair. Scheme 1, Slave node v
2Upper collected data C
3And C
4From node v
3Collecting data C
7From node v
4Upper collected data C
2And C
6And encode the block C
2、C
3、C
4、C
6、C
7Transmitted to the new node to complete the node v
1And (4) repairing. Scheme 2, Slave node v
3、v
5Respectively collecting data C
4、C
7And C
2、C
3From node v
4Upper collected data C
6And encode the block C
2、C
3、C
4、C
6、C
7Transmitted to the new node to complete the node v
1And (4) repairing. Scheme 3, Slave node v
3Collecting data C
4And C
7From node v
4Upper collected data C
2And C
6From node v
5Upper collected data C
3Encode the block C
2、C
3、C
4、C
6、C
7Transmitted to the new node to complete the node v
1And (4) repairing. Scheme(s)4, slave node v
3、v
4Respectively collecting data C
4，C
7And C
2，C
6From node v
2Upper collected data C
3And encode the block C
2、C
3、C
4、C
6、C
7Transmitted to the new node to complete the node v
1And (4) repairing.
Example 2
The embodiment discloses a method for constructing a heterogeneous partial repetition code based on a complete graph, and further discloses the following technical characteristics on the basis of the embodiment 1, and fig. 3 shows a storage scheme of a heterogeneous partial repetition code which has 15 coding blocks and stores 6 coding blocks in each node in a distributed storage system.
Step 1: storing a file with the size of 13M in a distributed storage system, firstly carrying out (15,13) MDS encoding on an original file, and then representing the 15 encoding blocks by 1, 2, 3 and … 15;
step 2: drawing a complete graph containing 6 vertexes, taking the 6 vertexes of the complete graph as 6 vertexes of the distributed storage system, numbering the 6 vertexes of the complete graph and 6 edges of the outermost layer of the complete graph in sequence by 1, 2, … and 6, and numbering the other edges of the complete graph in sequence by 7 … and 15 from a node 1;
and step 3: taking the edge connecting the node 1 and the node 6 as a starting edge in the clockwise direction, and taking the number of the edge and the three adjacent edges at the right side as a set s
1I.e. s
16,7,8, 9. The four edges and the node 1 form three corners, the three corners are respectively read to correspond to the edge numbers {3, 4, 5} of the outermost layer of the complete graph, the numbers correspond to the numbers of the coding blocks stored in the node 1, and then the partial coding blocks stored in the first node are: c
3、C
4、C
5. Similarly, the edge connecting node 2 and node 1 is taken as the starting edge, and the number of the edge and the three adjacent edges on the right side of the edge are taken as the set s
2I.e. s
21,10,11, 12. The four edges and the node 2 form three corners, the three corners are respectively read to correspond to the edge numbers {4, 5, 6} of the outermost layer of the complete graph, and the numbers correspond to the coding blocks stored in the node 2The number of (2), the part of the coding blocks stored by the second node is: c
4、C
5、C
6. According to the rule, the part of the coding blocks stored by the third node is as follows: c
1、C
5、C
6(ii) a The fourth node stores part of the coding blocks as follows: c
1、C
2、C
6(ii) a The fifth node stores part of the coding blocks as follows: c
1、C
2、C
3(ii) a The sixth node stores part of the coding blocks as follows: c
2、C
3、C
4；
And 4, step 4: starting from node 1, two edges formed by node 6 and node 2 connected to node 1 are removed, and the number of the edge formed by node 1 and other nodes is recorded as a set E
1I.e. E
1Set E {7,8,9}
1The code blocks corresponding to the number of each edge are stored in the node 1, namely the code block C
7、C
8、C
9Stored in the first node. Similarly, two edges formed by the node 1 and the node 3 connected with the node 2 are removed, and the number of the edge formed by the node 2 and other nodes is recorded as a set E
2I.e. E
210,11, 12. Set E
2The code blocks corresponding to the number of each edge are stored in the node 2, namely the code block C
10、C
11、C
12And storing the data in the second node. According to the rule, the code block C is coded
9、C
13、C
14Storing the data in a third node; coding block C
8、C
12、C
15Storing the data into a fourth node; coding block C
7、C
11、C
14Storing the data into a fifth node; coding block C
10、C
13、C
15And storing the data into the sixth node.
And 5: finally, 6 heterogeneous partial repetition codes with the node storage capacity of 6 can be obtained, and each node stores 3 coding blocks with the repetition degree of 3 and 3 coding blocks with the repetition degree of 2. The coding block stored by the first node is: c
3、C
4、C
5、C
7、C
8、C
9(ii) a The second node stores the coding blocks as follows: c
4、C
5、C
6、C
10、C
11、C
12(ii) a The third node stores the coding blocks as follows: c
1、C
5、C
6、C
9、C
13、C
14(ii) a The fourth node stores the coding blocks as follows: c
1、C
2、C
6、C
8、C
12、C
15(ii) a The coding block stored by the fifth node is: c
1、C
2、C
3、C
7、C
11、C
14(ii) a The coding block stored by the sixth node is: c
2、C
3、C
4、C
10、C
13、C
15。
The embodiment also discloses a method for repairing two continuous fault nodes, and fig. 4 shows a node v in the embodiment
1And node v
2Schematic diagram of fault repair. Scheme 1, Slave node v
3And node v
5Respectively collecting data C
5、C
6、C
9And C
3、C
7、C
11From node v
4、v
6Respectively collecting data C
8、C
12And C
4、C
10. Coding block C
3、C
4、C
5、C
7、C
8、C
9Transmitted to the new node to complete the node v
1Repair of (2) coding the block C
4、C
5、C
6、C
10、C
11、C
12Transmitted to the new node to complete the node v
2And (4) repairing. Scheme 2, Slave node v
3、v
5Respectively collecting data C
5、C
9And C
7、C
11From node v
4And node v
6Respectively collecting data C
6、C
8、C
12And C
3、C
4、C
10. Coding block C
3、C
4、C
5、C
7、C
8、C
9Transmitted to the new node to complete the node v
1Repair of (2) coding the block C
4、C
5、C
6、C
10、C
11、C
12Transmitted to the new node to complete the node v
2And (4) repairing.
Claims (3)
1. The heterogeneous partial repetition code construction method based on the complete graph is characterized by comprising the following steps of:
step 1: dividing an original file into k original data blocks, and performing (n, k) MDS coding on the k original data blocks to obtain n coding blocks C
1,C
2,…，C
nWherein n and k are positive integers;
step 2: establishing a complete graph comprising m vertexes, numbering the m vertexes of the complete graph in sequence by using 1, 2, …, wherein m is the number of the m vertexes of the complete graph in the clockwise direction, numbering the m edges at the outermost layer of the complete graph in sequence by using 1, 2, … and m in the clockwise direction from the No. 1 vertex, and then numbering the edges inside the complete graph in sequence by using m +1, m +2, … and n in the clockwise direction from the vertex 1, wherein n is m (m-1)/2, and m is not less than 5;
and step 3: divide the complete map into m sets S
iI 1, 2, …, m, set S
iCorresponding to the vertexes i and S
iThe three opposite sides are the sides of the outermost layer of the complete graph corresponding to three internal included angles, and the three internal included angles are included angles which are formed by the first four sides which are connected with the vertex in the clockwise direction from the vertex and are adjacent in pairs;
and 4, step 4: divide the complete graph into m sets E
iSet E of
iCorresponding vertices i and E
iContaining m-3 sides, said E
iThe side in (1) is formed by connecting m-3 vertexes which are not adjacent to the vertex i with the vertex i;
and 5: 1 to m vertexes are respectively corresponding to 1 to m nodes, for a node i, three coding blocks with the repetition degree of 3 are stored firstly, and the numbers of the three coding blocks sequentially correspond to S
iThe serial numbers of three opposite sides in the code block are stored, then m-3 code blocks with the repetition degree of 2 are stored, and the serial numbers of the m-3 code blocks sequentially correspond to E
iAnd numbering m-3 edges in the code block to finally obtain m nodes, wherein m coding blocks are stored in each node, and the structure of the heterogeneous part repeated code is completed.
2. The full graph-based heterogeneous partial repeat of claim 1Code construction method, characterized in that in step 3 set S
iThe construction method comprises the following steps:
for vertex 1, construct S
1，S
1The three opposite sides of the first four sides L are respectively in the counterclockwise direction from the vertex and share the vertex
m、L
m+1、L
m+2And L
m+3In two adjacent edges L
mAnd L
m+1、L
m+1And L
m+2And L
m+2And L
m+3The outermost side L of the complete picture corresponding to the included angle
m-1、L
m-2And L
m-3To obtain S
1＝{L
m-1、L
m-2、L
m-3And f, successively analogizing to obtain a set S corresponding to all vertexes
i。
3. The method for repairing the fault node is characterized by comprising the following steps:
the method for constructing the heterogeneous partial repetition code based on the full graph according to any one of claims 1 or 2, wherein an original file containing n coding blocks is stored in m nodes of a distributed storage system, each node stores m coding blocks, and each node stores 3 coding blocks with repetition degree of 3 and m-3 coding blocks with repetition degree of 2;
when a single node fails, acquiring coding blocks from m-2 surviving nodes, and transmitting the acquired coding blocks to a new node to complete the repair of the single node failure;
when any two continuous nodes have faults, the coding blocks are collected from m-2 surviving nodes, and the collected coding blocks are transmitted to two continuous new nodes, so that the fault repair of any two continuous nodes is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910930841.5A CN110781163B (en) | 2019-09-29 | 2019-09-29 | Heterogeneous part repeated code construction and fault node repairing method based on complete graph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910930841.5A CN110781163B (en) | 2019-09-29 | 2019-09-29 | Heterogeneous part repeated code construction and fault node repairing method based on complete graph |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781163A true CN110781163A (en) | 2020-02-11 |
| CN110781163B CN110781163B (en) | 2023-01-17 |
Family
ID=69384617
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910930841.5A Active CN110781163B (en) | 2019-09-29 | 2019-09-29 | Heterogeneous part repeated code construction and fault node repairing method based on complete graph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781163B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113347026A (en) * | 2021-05-21 | 2021-09-03 | 长安大学 | Cube network-based partial repeated code construction and fault node repair method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2495879A2 (en) * | 2007-12-24 | 2012-09-05 | LG Electronics Inc. | Channel coding method of variable length information using block code |
| CN105260259A (en) * | 2015-09-16 | 2016-01-20 | 长安大学 | System minimum storage regeneration code based local repair encoding method |
| CN108540520A (en) * | 2018-02-06 | 2018-09-14 | 长安大学 | Locality reparation coding based on part duplication code and node failure restorative procedure |
| CN109684127A (en) * | 2018-12-29 | 2019-04-26 | 西安电子科技大学 | Locality node restorative procedure based on complete graph minimum bandwidth regeneration code |
| CN109828723A (en) * | 2019-02-13 | 2019-05-31 | 山东大学 | A kind of distributed memory system and its precise information restorative procedure and device |
-
2019
- 2019-09-29 CN CN201910930841.5A patent/CN110781163B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2495879A2 (en) * | 2007-12-24 | 2012-09-05 | LG Electronics Inc. | Channel coding method of variable length information using block code |
| CN105260259A (en) * | 2015-09-16 | 2016-01-20 | 长安大学 | System minimum storage regeneration code based local repair encoding method |
| CN108540520A (en) * | 2018-02-06 | 2018-09-14 | 长安大学 | Locality reparation coding based on part duplication code and node failure restorative procedure |
| CN109684127A (en) * | 2018-12-29 | 2019-04-26 | 西安电子科技大学 | Locality node restorative procedure based on complete graph minimum bandwidth regeneration code |
| CN109828723A (en) * | 2019-02-13 | 2019-05-31 | 山东大学 | A kind of distributed memory system and its precise information restorative procedure and device |
Non-Patent Citations (4)
| Title |
|---|
| JING WANG 等: "《Distributed scheme based on network coding to construct k -redundant multicast network》", 《2010 IEEE INTERNATIONAL CONFERENCE ON INFORMATION THEORY AND INFORMATION SECURITY》 * |
| 余春雷 等: "《异构分布式存储系统中部分重复码的构造》", 《信息通信》 * |
| 张会娇: "《基于设计和图的部分重复码的构造》", 《CNKI优秀硕士学位论文全文库》 * |
| 王静等: "分布式存储系统中基于部分重复循环码的局部修复码构造", 《电子测量与仪器学报》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113347026A (en) * | 2021-05-21 | 2021-09-03 | 长安大学 | Cube network-based partial repeated code construction and fault node repair method |
| CN113347026B (en) * | 2021-05-21 | 2022-06-28 | 长安大学 | Part repeated code construction and fault node repairing method based on cube network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781163B (en) | 2023-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10067832B2 (en) | Imposter slices | |
| US9043548B2 (en) | Streaming content storage | |
| US9201732B2 (en) | Selective activation of memory to retrieve data in a dispersed storage network | |
| CN109643258B (en) | Multi-node repair using high-rate minimal storage erase code | |
| CN105956128B (en) | A kind of adaptive coding storage fault-tolerance approach based on simple regeneration code | |
| CN109814807B (en) | Data storage method and device | |
| CN109491835B (en) | Data fault-tolerant method based on dynamic block code | |
| CN106776129B (en) | A kind of restorative procedure of the multinode data file based on minimum memory regeneration code | |
| CN105393225A (en) | Erasure coding across multiple zones | |
| CN103746774B (en) | The fault-tolerant coding method that a kind of efficient data is read | |
| CN111831223B (en) | Fault-tolerant coding method, device and system for improving expandability of data deduplication system | |
| CN107003933B (en) | Method and device for constructing partial copy code and data restoration method thereof | |
| CN108132854B (en) | Erasure code decoding method capable of simultaneously recovering data elements and redundant elements | |
| CN110764950A (en) | Hybrid coding method, data restoration method and system based on RS (Reed-Solomon) code and regeneration code | |
| CN106484559A (en) | A kind of building method of check matrix and the building method of horizontal array correcting and eleting codes | |
| WO2023103213A1 (en) | Data storage method and device for distributed database | |
| CN108762978B (en) | Grouping construction method of local part repeated cyclic code | |
| CN112799605B (en) | Square part repeated code construction method, node repair method and capacity calculation method | |
| US20150227425A1 (en) | Method for encoding, data-restructuring and repairing projective self-repairing codes | |
| CN107153661A (en) | A kind of storage, read method and its device of the data based on HDFS systems | |
| CN113258936B (en) | Dual coding construction method based on cyclic shift | |
| Estrada-Galinanes et al. | Alpha entanglement codes: practical erasure codes to archive data in unreliable environments | |
| CN110781163B (en) | Heterogeneous part repeated code construction and fault node repairing method based on complete graph | |
| CN110781024B (en) | Matrix construction method of symmetrical partial repetition code and fault node repairing method | |
| CN112000278B (en) | Self-adaptive local reconstruction code design method for thermal data storage and cloud storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |