CN114785667B - Pipeline repairing method suitable for RS code data repairing - Google Patents

Abstract

A pipeline repair method suitable for RS code data repair, the method comprising repair path selection and a pipeline repair scheme. The whole pipeline repair includes the following steps: (1) Determining a graph G by the reachable nodes, edges and bandwidths; (2) Calculating an optimal restoration path of the node by using the graph G; (3) starting to construct a repair pipeline; (4) And (3) data repair, namely dividing repair sub-blocks according to a pipeline repair model. The invention can ensure that the nodes receive, send and calculate data are balanced when the data is repaired, and the repair route of the assembly line is not required to be replaced frequently. When k is very large, on the premise of ensuring load balance, the existing pipeline is difficult to obtain a reasonable value on the data segmentation quantity, and the invention can greatly reduce the selection difficulty because each group of repair is balanced.

Description

Pipeline repairing method suitable for RS code data repairing

Technical Field

The invention relates to a pipeline repairing method suitable for RS code data repairing, and belongs to the technical field of erasure codes.

Background

Network bandwidth, CUP, disk I/O and other resources in the distributed system are limited and scarce, the resources affect the service efficiency of the erasure codes at all times, the low repair performance of the erasure codes means that the availability of data is limited, and for the distributed system with node faults at any time, the repair rate directly affects the reliability of the whole system. The availability and timeliness of data in a distributed system is important, and it is desirable to repair failure data at the fastest rate to ensure reliability of the system. The data repair time of the erasure code comprises data transmission time, disk I/O time and calculation time, wherein the data transmission time occupies the maximum proportion of 94 percent.

The invention can be applied to the data restoration process of the failure of the RS code single node, and aims to improve the data restoration efficiency of the RS code. In order to fully utilize the uplink and downlink bandwidths of the nodes, the repair blocks are segmented, a plurality of repair sub-operations can be executed in parallel, and then a pipeline repair mode is adopted, so that the purpose of parallel repair is achieved, and the rapid data repair is achieved.

Disclosure of Invention

The invention aims to improve the data restoration efficiency of an RS code and achieve quick data restoration, and provides a pipeline restoration method suitable for the data restoration of the RS code.

The technical scheme of the invention is as follows, a pipeline repair method suitable for RS code data repair, the method comprises repair path selection and pipeline repair scheme;

the repair path is selected as follows:

the pipeline modification model is expressed as: g= { V, E, W };

wherein V represents a set of nodes; e represents node-to-node connection (edge); w represents the bandwidth of the edge; v= { V _i |0≤i≤n-1}，V _i When i=0, V is node ₀ Representing a new node, V ₁ -V _n-1 Representing a help node; e= { (V) _i ,V _j )|0≤i，j≤n-1}，(V _i ,V _j ) Indicating the presence of a slave V _i To V _j Is a side of (c).

Repair path selection to eliminate new node V ₀ Arbitrary node V _i For starting and ending, it is required to remove V ₀ The outer k nodes (including the start point) need to find a path that can maximize the total path bandwidth traversed; a path with the largest bandwidth can be found using the minimum spanning tree algorithm.

The pipeline repairing scheme is as follows:

dividing the repair block into s sub-blocks, wherein s is required to satisfy that s is divided by k to be equal to 0, and each repair block B _i (1.ltoreq.i.ltoreq.k) equally dividing into s sub-blocks B _i，1 、B _i，2 、…、B _i,s Denoted as B _i,j (i is less than or equal to j is less than or equal to s), and the transmission of the first sub-block is regarded as a time slot; suppose the repair route is: v (V) ₁ -＞V ₂ -＞…-＞V _k ，V ₁ -＞V ₂ Represents V ₁ To V ₂ Transmission data, V _i To help repair node (1. Ltoreq.i.ltoreq.k), node V is regenerated ₀ The data to be repaired is B ₀ ＝(B _0,1 B _0,2 …B _0,s )。

The whole pipeline repair includes the following steps:

(1) Determining a graph G by basic data including reachable nodes, edges and bandwidths;

(2) Using a graph G, selecting according to the repair paths, and calculating an optimal repair path of k nodes; assuming that the optimal repair path is V ₁ -＞V ₂ -＞…-＞V _k K pipelines can be constructed, assuming k=4, then there are 4 pipeline repair routes:

V ₁ -＞V ₂ -＞V ₃ -＞V ₄

V ₂ -＞V ₃ -＞V ₄ -＞V ₃

V ₃ -＞V ₄ -＞V ₁ -＞V ₂

V ₄ -＞V ₁ -＞V ₂ -＞V ₃ ；

(3) Taking k=4 as an example, the construction of a repair pipeline begins

Construction of type 1 pipeline repair model M ₁ Directly using k pipelines in the step (2) as repair routes, wherein the first group is completed only by k-1 transmissions as the result of the first group does not need to be processed by the previous round, and is used for repairing k sub-blocks before processing;

construction of model M for pipeline repair of type 2 ₂ The k production lines are still used as repair routes, and the first group of production line schemes are improved in a staggered placement and one-time transmission adding mode to obtain a second group of production line routes for processing the repair of all sub-blocks except the first k sub-blocks;

construction of the 3 rd pipelineThe repair model only needs to send the last group of repair results to the new node V ₀ The method comprises the steps of carrying out a first treatment on the surface of the At this time, the pipeline model is very simple; for processing the results of the last group;

(4) Data repair, which is to use the different repair schemes in the above 3, constructs a pipeline repair model M ₁ 、M ₂ 、M ₃ The repair model to which the repair sub-blocks belong is divided according to the pipeline repair model;

assuming that the data slice is s, when s=k, all the sliced molecular blocks use the model M ₁ Repair is carried out, and the model M can be used only by processing the result of the previous round ₃ Repairing the sub-block cutting result; then when s > k, the first k sub-blocks, in the first set of repairs, follow model M ₁ The mode is as follows; residual of

Group compliance model M ₂ The method comprises the steps of carrying out a first treatment on the surface of the Finally, the last group of sub-block repair results are processed, and a model M is used ₃ 。

Compared with the prior art, the method has the advantages that the method can ensure that the nodes receive, send and calculate the data in a balanced way when the data is repaired, and the repair route of the assembly line is not required to be replaced frequently. When k is very large, on the premise of ensuring load balance, the existing pipeline is difficult to obtain a reasonable value on the data segmentation quantity, and the invention can greatly reduce the selection difficulty because each group of repair is balanced.

Drawings

FIG. 1 is a schematic diagram of a repair path with maximum bandwidth allocation;

FIG. 2 is a basic repair flow for pipeline repair;

FIG. 3 is M ₁ A pipeline model;

FIG. 4 is M ₂ A pipeline model;

FIG. 5 is M ₃ A pipeline model;

fig. 6 is a schematic diagram of a pipeline repair scheme for k=4, s=8, where the black part indicates that the final result of the computation is transmitted to the new node;

FIG. 7 is a schematic diagram of the main portion of the pipeline repair of the present invention;

FIG. 8 is a schematic diagram of the entire pipeline repair process of the present invention.

Detailed Description

Specific embodiments of the invention are shown in the drawings.

The embodiment of the pipeline repair method suitable for RS code data repair comprises two parts of repair path selection and a pipeline repair model.

The repair path is selected as follows:

setting a pipeline repair model to be represented by G= { V, E, W } and V represents a node set; e represents node-to-node connection (edge); w represents the bandwidth of the edge; v= { V _i |0≤i≤n-1}，V _i When i=0, V is node ₀ Representing a new node, V ₁ -V _n-1 Representing a help node; e= { (V) _i ,V _j )|0≤i，j≤n-1}，(V _i ,V _j ) Indicating the presence of a slave V _i To V _j Edges (representing V) _i 、V _j Network connection).

Repair path selection to eliminate new node V ₀ Arbitrary node V _i For starting and ending, it is required to remove V ₀ The outer k nodes (including the start point) need to find a path that maximizes the path bandwidth traversed. A path with the largest bandwidth can be found using the minimum spanning tree algorithm.

When k=4, as shown in fig. 1: the repair path is V ₁ -＞V ₃ -＞V ₄ -＞V ₂ -＞V ₁ 。

The pipeline repairing scheme comprises the following steps:

pipeline repair, namely firstly dividing a repair block into s subblocks, wherein s needs to satisfy that s integer division k is equal to 0, and each repair block B _i (1.ltoreq.i.ltoreq.k) equally dividing into s sub-blocks B _i，1 、B _i，2 、…、B _i,s Denoted as B _i,j (i.ltoreq.j.ltoreq.s) and the transmission of a sub-block is regarded as one slot.

Suppose the repair route is: v (V) ₁ -＞V ₂ -＞…-＞V _k ，V ₁ -＞V ₂ Represents V ₁ To V ₂ Transmission data, V _i To help repair node (1. Ltoreq.i.ltoreq.k), node V is regenerated ₀ The data to be repaired is B ₀ ＝(B _0,1 B _0,2 …B _0,s ) Then there is:

B _0，1 ＝a ₁ B _1,1 +a ₂ B _2,1 +…+a _k-1 B _k-1,1 +a _k B _k,1

B _0,2 ＝a ₁ B _1,1 +a ₂ B _2,2 +…+a _k-1 B _k-1,1 +a _k B _k,1

…

B _0，s ＝a ₁ B _1,s +a ₂ B _2,s +…+a _k-1 B _K-1,s +a _k B _k,s

then taking k=4 as an example, the basic repair flow is as shown in fig. 2:

V ₁ read B _1，1 Transmitting a ₁ B _1,1 Give V to ₂ ，

V ₂ Read B _2，1 Combine with V ₁ Will be a ₁ B _1,1 +a ₂ B _2,1 Give V to ₃ ，

V ₃ Read B _3，1 Combine with V ₂ Will be a ₁ B _1,1 +a ₂ B _2,1 +a ₃ B _3,1 Give V to ₄ ，

V ₄ Read B _4，1 Combine with V ₂ Will be a ₁ B _1,1 +a ₂ B _2,1 +a ₃ B _3,1 +a ₄ B _4,1 Give V to ₀ ，

B can be obtained by repairing in the above manner _0，1 And will B _0，1 Placement V ₀ In the same way, B can be repaired _0，2 、…、B _0,s B can be repaired by combining the sub-blocks ₀ 。

As shown in fig. 8, the whole pipeline repair of the present embodiment can be divided into the following 4 steps:

step 1, determining the graph G by some basic data, such as reachable nodes, edges and bandwidths.

And 2, selecting and calculating an optimal repair path of k nodes according to the repair path of the first part of the invention content by using the graph G. Assuming that the optimal repair path is V ₁ -＞V ₂ -＞…-＞V _k K pipelines can be constructed as follows, assuming k=4, then there are 4 pipeline repair routes as follows:

V ₁ -＞V ₂ -＞V ₃ -＞V ₄

V ₂ -＞V ₃ -＞V ₄ -＞V ₃

V ₃ -＞V ₄ -＞V ₁ -＞V ₂

V ₄ -＞V ₁ -＞V ₂ -＞V ₃

step 3, taking k=4 as an example, starting to construct a repair pipeline

Model M of type 1 pipeline ₁ As shown in fig. 3:

construction of type 1 pipeline repair model M ₁ And (3) directly using k pipelines in the step (2) as repair routes, wherein the first group can be completed only by k-1 transmissions as the first group does not need to be processed by the last round of results, and is used for repairing the k sub-blocks before processing.

Model M of type 2 pipeline ₂ As shown in fig. 4:

the second group, also based on the k pipelines, requires proper modification of the pipeline paths because the results from the first group need to be processed. Construction of model M for pipeline repair of type 2 ₂ The first group of pipeline schemes are improved by misplacement and one-time transmission, so that a second group of pipeline routes are obtained and are used for processing the repair of all the sub-blocks except the first k sub-blocks.

3 rd pipeline model M ₃ As shown in fig. 5:

V ₄ -＞V ₀ V ₁ -＞V ₀ V ₂ -＞V ₀ V ₃ -＞V ₀

construction of 3 rd pipeline repair model M ₃ Only the last group of repair results are required to be sent to the new node V ₀ The method comprises the steps of carrying out a first treatment on the surface of the The pipeline model is simple at this time. For processing the results of the last group.

The previous 3 steps can be calculated in advance, and the calculation is performed when the repair is not needed.

Step 4, data repair, wherein the data repair needs to be performed by using the 3 different repair schemes, and a pipeline repair model M is constructed ₁ 、M ₂ 、M ₃ The repair model to which the repair sub-block belongs is divided according to the pipeline repair model.

Assuming that the data slice is s, when s=k, all the sliced molecular blocks use the model M ₁ Repair is carried out, and the model M can be used only by processing the result of the previous round ₃ Repairing the sub-block results. Then when s > k, the first k sub-blocks, in the first set of repairs, follow model M ₁ The method is just needed. Residual of

Group compliance model M ₂ And (3) obtaining the product. Finally, the last group of sub-block repair results are processed, and a model M is used ₃ 。

Taking the parameter k=4 as an example, the pipeline repair scheme is as follows:

the repair whole data block is split first into sub-operations to repair s data slices, assuming the split s=8, as shown in fig. 6.

(1)t ₀ V when pipeline repair is started at moment ₁ 、V ₂ 、V ₃ 、V ₄ Starting to execute respective pipeline repair circuitsLine t ₃ The first set of repairs is completed at the instant. Through t ₀ -t ₃ At this time, a sub-block result B can be obtained _0，1 、B _0，2 、B _0,3 、B _0,4 Results are respectively stored in the node V ₄ 、V ₁ 、V ₂ 、V ₃ The repair results for each group are borne by the next group.

(2) The second group has k pipeline paths as the first group, t ₃ Starting to execute the second group of repairs at the moment, but being different from the first group in that the first round of calculation results need to be transmitted to V ₀ A single transfer process is added so that k pipeline paths can be made at the same time without collision. The second group needs to time-share the first group calculation result to V ₀ The new node, because of the need to transmit the result of the last group, the pipeline path also needs to be changed; the changes were as follows:

1)t ₃ -t ₄ time V ₄ Will B _0，1 Sent to V ₀ Due to V at this time ₄ Transmission is occupied, so B _0,8 Is delayed to t ₄ -t ₅ Starting.

2)t ₄ -t ₅ Time V ₁ Will B _0，2 Sent to V ₀ 。

3)t ₅ -t ₆ Time V ₂ Will B _0,3 Sent to V ₀ 。

4)t ₆ -t ₇ Time V ₃ Will B _0,4 Sent to V ₀ 。

The second group was newly added 4 transmission processes compared to the first group, with the specific change shown in fig. 7.

Through t ₃ -t ₇ First set of results B _0，1 、B _0，2 、B _0,3 、B _0,4 Has been sent to the new node V ₀ While the second group obtains sub-block result B _0,5 、B _0,6 、B _0,7 、B _0,8 The calculation results are respectively placed with V ₄ 、V ₁ 、V ₂ 、V ₃ 。

(3) Third group, will B _0,5 、B _0,6 、B _0,7 、B _0,8 Send to V ₀ The repair has been completed so far.

The embodiment has no special requirement on language tools, and can be realized in the C language, the C++ language, the Java language and the like. The system has no special requirement on an operating system platform, and a Microsoft Windows system, various Linux systems, a Mac system and the like can be used as the operating system running platform.

When the method is implemented, data is read from a disk to a memory, then the data is segmented, the segmentation number can be selected according to the working environment, the segmented data is transmitted through a pipeline repair route, and then the data is calculated, wherein each repair route in each group is different, but the repair mode is consistent with the RS code repair logic.

Claims (1)

1. A pipeline repair method suitable for RS code data repair, the method comprising repair path selection and pipeline repair scheme;

the repair path is selected as follows:

the pipeline repair model is expressed as: g= { V, E, W };

wherein V represents a set of nodes; e represents a node-to-node connection; w represents the bandwidth of the edge; v= { V _i |0≤i≤n-1}，V _i When i=0, V is node ₀ Representing a new node, V ₁ -V _n-1 Representing a help node; e= { (V) _i ,V _j )|0≤i，j≤n-1}，(V _i ,V _j ) Indicating the presence of a slave V _i To V _j Is a side of (2);

repair path selection to eliminate new node V ₀ Arbitrary node V _i For starting and ending, it is required to remove V ₀ The k nodes outside need to find out the route bandwidth and the maximum route that can be made to pass through; a minimum spanning tree algorithm is used to find a bandwidth and a maximum path;

the pipeline repairing scheme is as follows:

dividing the repair block into s sub-blocks, wherein s is required to satisfy that s is divided by k to be equal to 0, and each repair block B _i Equal amount of i is more than or equal to 1 and less than or equal to k is divided into s sub-blocks B _i ， ₁ 、B _i ， ₂ 、…、B _i,s Denoted as B _i，j I is less than or equal to j is less than or equal to s, and the transmission of the first sub-block is regarded as a time slot; suppose the repair route is: v (V) ₁ -＞V ₂ -＞…-＞V _k ，V ₁ -＞V ₂ Represents V ₁ To V ₂ Transmission data, V _i To help repair node 1.ltoreq.i.ltoreq.k, node V is regenerated ₀ The data to be repaired is B ₀ ＝(B _0,1 B _0,2 …B _0,s )；

The whole pipeline repair includes the following steps:

(1) Determining a graph G by the reachable nodes, edges and bandwidths;

(2) Using a graph G, selecting according to the repair paths, and calculating an optimal repair path of k nodes; assuming that the optimal repair path is V ₁ -＞V ₂ -＞…-＞V _k K pipelines can be constructed, assuming k=4, then there are 4 pipeline repair routes:

V ₁ -＞V ₂ -＞V ₃ -＞V ₄

V ₂ -＞V ₃ -＞V ₄ -＞V ₃

V ₃ -＞V ₄ -＞V ₁ -＞V ₂

V ₄ -＞V ₁ -＞V ₂ -＞V ₃ ；

(3) Taking k=4 as an example, the construction of a repair pipeline begins

Construction of type 1 pipeline repair model M ₁ Directly using k pipelines in the step (2) as repair routes, wherein the first group is completed only by k-1 transmissions as the result of the first group does not need to be processed by the previous round, and is used for repairing k sub-blocks before processing;

construction of model M for pipeline repair of type 2 ₂ The k pipelines are still used as repair routes, and the first group of pipeline schemes are improved in a staggered placement and one-time transmission adding mode to obtain a second group of pipeline routes for processing the repair of all sub-blocks except the first k sub-blocksRepeating;

construction of 3 rd pipeline repair model M ₃ Only the last group of repair results are required to be sent to the new node V ₀ The method comprises the steps of carrying out a first treatment on the surface of the At this time, the pipeline model is very simple; for processing the results of the last group;

(4) Data repair, which is to use the above 3 different repair schemes to construct a pipeline repair model M ₁ 、M ₂ 、M ₃ The repair model to which the repair sub-blocks belong is divided according to the pipeline repair model;

assuming that the data slice is s, when s=k, all the sliced molecular blocks use the model M ₁ Repair is carried out, and the model M can be used only by processing the result of the previous round ₃ Repairing the sub-block cutting result; then when s > k, the first k sub-blocks, in the first set of repairs, follow model M ₁ The mode is as follows; residual of

Group compliance model M ₂ The method comprises the steps of carrying out a first treatment on the surface of the Finally, the last group of sub-block repair results are processed, and a model M is used ₃ 。

Images