CN113285985A

CN113285985A - RS code node repairing method based on genetic algorithm under multi-data center background

Info

Publication number: CN113285985A
Application number: CN202110482403.4A
Authority: CN
Inventors: 王勇; 锁欣; 叶苗; 蔡月
Original assignee: Guilin University of Electronic Technology
Current assignee: Guilin University of Electronic Technology
Priority date: 2021-04-30
Filing date: 2021-04-30
Publication date: 2021-08-20

Abstract

The invention discloses a RS code node repairing method based on a genetic algorithm under the background of multiple data centers. The invention aims to solve the problem that a global optimal bottleneck bandwidth repair scheme cannot be obtained in a traditional data repair mode under the background of multiple data centers, and provides an optimal bottleneck bandwidth path selection method based on a genetic algorithm. The RS code node repairing method based on the genetic algorithm in the multi-data center background overcomes the problems of long repairing time delay and large bandwidth consumption of the traditional star repairing scheme and the traditional assembly line repairing scheme, reduces redundant data transmission, improves repairing efficiency and reduces repairing time.

Description

RS code node repairing method based on genetic algorithm under multi-data center background

Technical Field

The invention belongs to the field of distributed erasure code storage systems, and particularly relates to a RS code node repairing method based on a genetic algorithm under the background of multiple data centers.

Background

The distributed storage system becomes a mainstream storage system in the current large-scale data storage field by virtue of excellent performance and low construction cost, but node failure becomes a normal state because the underlying equipment of the distributed storage system generally adopts cheap commercial hardware.

In order to prevent service loss caused by data failure due to node failure, an erasure code and multiple copies of redundant data are often used in a distributed storage system to ensure the integrity and reliability of data, wherein the erasure code is widely applied because of low extra storage overhead, but when data failure occurs due to failure of a part of nodes, the erasure code needs to read other data blocks and perform coding and decoding to recover the failed data, and a large amount of recovery traffic overhead is generated in the process, and the recovery speed is low.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention provides a RS code node repairing method based on a genetic algorithm under the background of multiple data centers, which reduces the repairing flow overhead and accelerates the repairing efficiency.

Taking (n, k) RS code as an example, n data blocks of the set of codes are stored in 2 data centers, respectively, and the network topology of the two data centers can be represented as G ═ V, E, W, V ═ 1,2, …, n₁,n₁+1, …, n } represents a set of n nodes with (n, k) codes stored in the cluster, where cloud 1 contains n1 repair nodes, cloud 2 contains n2 repair nodes, n₁+n₂＝n；E＝{e₁₁,e₁₂,…,e_ijDenotes the link connections inside the cluster, e_ijRepresenting an optimal link between nodes; w_ijIndicates the available bandwidth between nodes and, when i ═ j, indicates the processing power of the nodes. When a node failure occurs in one cloud center, the cloud center C1 and the cloud center C2 respectively provide k1 and k2 data blocks as provider nodes to repair the failed node, and k is used as a provider node to repair the failed node₁+k₂＝k。

When erasure code data is repaired, data blocks transmitted by all provider nodes can be recovered after all data blocks reach the repaired nodesAnd repeating the failed data blocks, wherein the link with the minimum bandwidth in the tree-shaped repair topology directly affects the efficiency of the whole data transmission process, the link is called a bottleneck link, the available bandwidth of the link is called bottleneck bandwidth, and the larger the bottleneck bandwidth of the repair tree is, the higher the efficiency of the repair tree in transmitting the repair data is. The problem of building a repair tree can therefore be generalized to finding one in graph G rooted at the reconstruction point and C₁、C₂Each of which includes k₁、k₂The repair tree of a provider node maximizes the bottleneck bandwidth, which can be expressed as the following formula, where w (i, j) is the available bandwidth of the current link, D is the data block transmitted on the link, and the quotient of the two is the transmission delay:

T＝min(max{D/w(i,j)})

meanwhile, the tree topology has extra traffic to transmit compared with other topologies, so that each node in the repair tree performs merging operation on the child nodes and the coding blocks of the node, thereby saving the traffic transmission overhead, reducing the network load as much as possible and ensuring that the repair process does not cause great influence on the network. The calculated delay across the link can be expressed as the following equation, where the process_iRepresenting the computational power of node i:

t(v_i,v_j)＝D_i/process_i+D_j/process_j

when the traditional repair tree is added with the condition of node processing capability, the objective function in this chapter can be expressed as the following formula:

T_repair＝min(max(D/wij+D_i/process_i+D_j/process_j))

s.t k₁+k₂＝k

count(C₁<k₁),count(C₂<k₂)

the technical scheme adopted by the invention is as follows:

compared with the prior art, the method has the advantages that the repair bandwidth overhead is effectively reduced by introducing the factor of heterogeneous node processing capacity, and the repair time delay of the reconstruction tree is effectively reduced by designing a genetic algorithm.

Drawings

FIG. 1 is a flow chart of an algorithm;

FIG. 2 shows a coding scheme;

FIG. 3 is a crossover operation;

FIG. 4 shows a mutation operation;

fig. 5 shows the results of the experiment.

Detailed Description

The present invention will be further described with reference to the accompanying drawings, in which the described embodiments are merely illustrative and are not to be construed as limiting the present invention.

The invention relates to the problem of data recovery in a distributed storage system, and a node repair path is selected by utilizing an RS code node repair method based on a genetic algorithm under the background of multiple data centers, and lost data is repaired by transmitting data of a survival point through the path. The invention mainly comprises the following steps:

step 1: constructing a multi-center cluster which comprises a plurality of nodes, wherein the bandwidth distribution of the nodes among the data centers is 1G, the connection among the nodes in the data centers selects a partial topological structure in Internet 2OS3E topology, the network bandwidth is distributed according to data obtained by a planetlab, and the performance parameter x of the nodes_kSelecting I/O, CPU, memory, and chip, each parameter corresponding to weight r_k40%, 30%, 20%, 10%, corresponding value range is [21,265 ]]，[1,90]，[0.3,76]，[0.5,23]The processing capacity of each node is:

step 2: starting to run a genetic algorithm by taking an optional point in the data center as a failure reconstruction node, wherein the algorithm steps are shown in figure 1;

and step 3: constructing a repair tree which takes the failure node as a root and meets the node repair constraint by taking the initialization function of the genetic algorithm as a basis, and repeating the popsize times to form a population; assuming that the data center is C1/C2, the specific repair tree generation algorithm is as follows:

(1) all the alternative points are divided into two sets: selected points and unselected points, wherein the subset of the selected points are C1_ provider and C2_ provider respectively, the selected points are a node set in the reconstruction tree, the unselected points are C1_ alive and C2_ alive, and the code T is [0,0,0, …,0 ];

(2) randomly selecting a point i from a union set of C1_ provider and C2_ provider, finding a node set S directly connected with the point i, then selecting a point j from an intersection of the S and C1_ alive and C2_ alive, taking the point as a child node of the point i, adding the j into a C1_ provider or C2_ provider set, and deleting the j from C1_ alive or C2_ alive;

(3) detecting the length of the queue of the C1_ provider and the C2_ provider, checking whether the constraint is met simultaneously, if not, repeatedly executing the step 2, and if so, outputting the code T, as shown in FIG. 2;

and 4, step 4: and randomly selecting two repair trees from the population according to the crossing probability f1 to cross, and generating new individuals. The crossover operation firstly compares the coding sequences of two parents, then reserves the common edge of the parents in the coding sequence of a new offspring, then detects whether the reserved part is a tree with one root as a reconstruction point, if not, expands the tree into a tree with one root as a reconstruction point through an upper and a lower adding points and a rotating direction, and finally carries out constraint detection on the tree, when the points existing in the coding sequence do not meet the constraint conditions of C1 and C2 on the nodes, randomly selects points from a gene pool for removing all the nodes in the expanded tree, adds the points into the coding sequence until the constraint is met, and generates a new crossover individual, as shown in FIG. 3;

and 5: and randomly selecting a repair tree from the population according to the mutation probability f2 to carry out mutation, and generating new individuals. In the mutation operation, a node is divided into two sets, namely a selection pool and a node set, one point is selected from the selection pool as a mutation point, then a subtree of the mutation point is detected, one point which is not in the mutation subtree and can be directly connected with the mutation point is selected from a reconstruction tree, and the point is used as a father node of the mutation point, so that a new mutation individual is generated, as shown in fig. 4;

step 6: and (4) selecting the population after cross variation to generate a new population, and then circulating the

steps

4 and 5 again until the specified algebra is circulated, and outputting the optimal repair tree. The selection operation adopts a mechanism of 10% elite selection and 90% random selection, so that the convergence of the algorithm is ensured, and the diversity of the population is kept.

The experimental result is shown in fig. 5, and it can be seen that after the factor of node processing capacity is introduced, the solution solved by the primMST algorithm based on the greedy mechanism is not a local optimal solution. Meanwhile, compared with the traditional pipeline repairing scheme, the tree repairing scheme belongs to a plurality of nodes to concurrently execute repairing, so that the repairing time delay is obviously shorter than that of the pipeline repairing scheme.

Claims

1. A RS code node repairing method based on genetic algorithm in the background of multiple data centers is characterized by comprising the following steps:

step 2: optionally selecting one point from the data center as a failure reconstruction node;

and step 3: constructing a repair tree which takes the failure node as a root and meets the node repair constraint by taking the initialization function of the genetic algorithm as a basis, and repeating the popsize times to form a population;

and 4, step 4: randomly selecting two repair trees from the population according to the crossing probability f1 to carry out crossing to generate new individuals;

and 5: randomly selecting a repair tree from the population according to the mutation probability f2 to carry out mutation so as to generate new individuals;

step 6: and (4) selecting the population after cross variation to generate a new population, and then circulating the steps 4 and 5 again until the specified algebra is circulated, and outputting the optimal repair tree.

2. The node repair algorithm according to claim 1, wherein: step 4, a parent public link is reserved, which shows that the link has excellent performance and good transmission delay and calculation delay; in step 5, one node is selected to change its parent node in order to prevent the repair tree from falling into local optimum and being unable to jump out of the area.