CN109067852A - Cross-center cooperative repair method based on erasure codes - Google Patents
Cross-center cooperative repair method based on erasure codes Download PDFInfo
- Publication number
- CN109067852A CN109067852A CN201810773745.XA CN201810773745A CN109067852A CN 109067852 A CN109067852 A CN 109067852A CN 201810773745 A CN201810773745 A CN 201810773745A CN 109067852 A CN109067852 A CN 109067852A
- Authority
- CN
- China
- Prior art keywords
- memory node
- node
- data block
- block
- network delay
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a cross-center collaborative repair method based on erasure codes, which adopts the technical scheme that a distributed storage system consisting of a data center and control nodes is firstly constructed, the data center comprises N storage nodes, and the control nodes are failure data blocks DiSelecting a new storage node L N, sending a network delay test request and a storage node two-dimensional array NSet to L N, testing the network delay between storage nodes in L N and NSet by L N, sending a network delay test result array N L to a control node, selecting k available data blocks and storage nodes where the k available data blocks are located by the control node according to N L, carrying out decoding calculation on the storage nodes where the k available data blocks are located, merging the decoding calculation results by a data center where the selected storage nodes are located, sending the merged results to L N, further merging the results by L N to obtain recovered Di(ii) a And the control node returns the repair success information to the user. The invention can improve the data restoration efficiency and fault-tolerant capability of the large-scale distributed storage system.
Description
Technical field
The present invention relates to the fault tolerant of distributed storage system, the span centre heart collaboration for being based especially on correcting and eleting codes is repaired
Method.
Background technique
Data fault-tolerant is one of the key technology of mass data distribution storage, availability of the data fault-tolerant for raising system
It is most important with reliability.Currently, fault tolerant mainly includes the fault-toleranr technique based on duplication and the appearance based on correcting and eleting codes
Wrong technology.
Fault-toleranr technique simple, intuitive based on duplication, it is easy to accomplish and deployment, it needs to create for each data object several
An equal amount of copy, memory space expense are larger.It is multiple by the way that data object to be divided into based on the fault-toleranr technique of correcting and eleting codes
Data block encodes original data block using correcting and eleting codes algorithm to obtain the data block of redundancy, the letter of multiple data blocks
Breath is fused in less redundancy, effectively saving memory space.Compared with the fault-toleranr technique based on duplication, it is based on correcting and eleting codes
Fault-toleranr technique have many advantages, such as that redundancy is low, storage space utilization is high, gradually become big data era data fault-tolerant
Mainstream technology.
The basic thought of fault-toleranr technique based on correcting and eleting codes is to carry out coding to k original data block that m is calculated is superfluous
Remaining data block；Only need any k number in k+m data block according to block, so that it may calculate by decoding and restore to obtain original k number
According to block.K and m is positive integer.
Specifically, redundant data block Cj(1≤j≤m) is k original data block i.e. D1, D2... ..., Di... ..., Dk(1
≤ i≤k) linear combination, coding calculate be represented by formula (1), wherein GjiIt is data block code coefficient.That is, right
In redundant data block CjFor, firstly, code coefficient GjiWith data block DiWise multiplication is carried out, new data block T is obtainedji=Gji×
Di；Then, k new data block is added by turn, obtains redundant data block Cj,
Gj1×D1+…+Gji×Di+…+Gjk×Dk=Cj (1)
Each original data block DiIt is all other any k available block Er(1≤r≤k) (original data block is superfluous
Remaining data block) linear combination, decoding calculate be represented by formula (2), wherein HirIt is data block desorption coefficient.That is,
For original data block DiFor, firstly, desorption coefficient HirWith available block ErWise multiplication is carried out, new data block S is obtainedir
=Hir×Er；Then, k new data block is added by turn, obtains original data block
Hi1×E1+…+Hir×Er+…+Hik×Ek=Di (2)
Although as the progress of technology, the reliability of various software and hardware resources is significantly improved, still, with storage
The continuous expansion of resource extent, the failure probability of resource obviously rise.In extensive distributed storage system, resource failed is one
Kind normal behavior.In data block failure, the fault-toleranr technique based on correcting and eleting codes needs multiple data blocks that could decode recovery failure
Original data block, need to occupy more Internet resources.
With the rapid development of cloud computing technology, the distributed storage system based on multiple data centers is become increasingly popular.Same number
According to the network delay between two memory nodes of central interior and between two memory nodes at different data center
Network delay differs greatly, and the network delay between different data center can also have larger difference.Existing correcting and eleting codes data
Recovery technique mainly randomly chooses available block and carries out fail data reparation, does not fully take into account available block place
Memory node between network delay between difference give remediation efficiency bring influence.
Therefore, it how to be directed to the difference of network delay between memory node, the span centre heart collaboration based on correcting and eleting codes is designed and repairs
Compound method, effectively improves the data remediation efficiency of extensive distributed storage system, to improve extensive distributed storage system
Fault-tolerant ability has become the hot research problem of distribution field of storage.
Summary of the invention
The technical problem to be solved by the present invention is proposing that one kind is based on for the difference of network delay between memory node
The span centre heart cooperative repair method of correcting and eleting codes, effectively improves the data remediation efficiency of extensive distributed storage system, to improve
The fault-tolerant ability of extensive distributed storage system.
Technical solution of the present invention the following steps are included:
The first step constructs a distributed storage system, it is made of M data center and a control node, every number
Include N number of memory node according to center, each node (including control node and memory node) be one can be independently operated
Computer, each node are interconnected by the network equipment.M and N is positive integer.
It is mounted on operating system, ICP/IP protocol software in control node and memory node, is configured with network environment.
Control node is responsible for interacting with user, receives the fail data block reparation request that user submits；It is responsible for failure number
New memory node is selected according to block, and sends network delay test request to new memory node；It is responsible for being selected according to network delay
Select available block and its place memory node；It is responsible for storage desorption coefficient, distributes the reparation of fail data block to each memory node
Successful information is repaired in task and reception, is returned to user and is repaired successful information.
Memory node is responsible for the original data block and redundant data block of storing data object, executes network delay test and appoints
Business, and network delay test result is returned to control node；It executes fail data block and repairs task, and repaired to control node return
Multiple successful information.If data object DO is divided into k original data block i.e. D1, D2... ..., Di..., Dk(1≤i≤k), k is
Positive integer carries out coding to k original data block and m redundant data block is calculated, and m is positive integer, this k+m (k+m < N) is a
Data block is respectively stored on k+m memory node.In distributed storage system, each data block possesses unique data block coding
Number.
Task management program and result reclaimer are installed in control node.Task management program receives what user submitted
Fail data block reparation request selects a new memory node for fail data block, and sends network delay to new memory node
Test request；According to network delay test result from the k+m-1 of data object DO (removing fail data block) a available block
Middle k available block of selection and its place memory node；Fail data is sent to the memory node where k available block
Relevant information needed for block reparation request and reparation.As a result reclaimer is responsible for receiving the reparation successful information of memory node and be returned
Back to user.
Network delay test program and repair procedure are installed on memory node.Network delay test program, which is responsible for receiving, to be come
From the network delay test request of control node, network delay test is carried out, sends network delay test result to control node.
Repair procedure is responsible for receiving the request of the fail data block reparation from control node, and to the data available stored on memory node
Block is decoded calculating, completes to send after repairing to control node and repairs successful information.
Second step, control node are fail data block D to be repairediNew memory node LN is selected, specific method is:
The task management program of 2.1 control nodes receives the fail data block D that user submitsiRepair request.
The task management program of 2.2 control nodes selects a memory node as fail data from available memory node
Block DiNew memory node LN, selection principle is any data block of the not stored data object DO of memory node LN.
Third step, control node send network delay test request and memory node two-dimensional array to memory node LN
NSet, specific method are:
The task management program of 3.1 control nodes sends network delay test request to memory node LN.
The task management program of 3.2 control nodes is by the removing fail data block D of data object DOiIn addition k+m-1
Memory node where available block, which constitutes memory node two-dimensional array NSet, NSet [s] [1], indicates s-th of data available
Memory node where block, the data center where memory node where NSet [s] [2] s-th of available block of expression, 1
≤ s≤k+m-1 sends memory node two-dimensional array NSet to memory node LN.
4th step, memory node LN receive network delay test request and memory node two-dimensional array from control node
NSet successively tests the network delay in LN and NSet between each memory node, sends network delay test knot to control node
Fruit array NL, specific method is:
The network delay test program of 4.1 memory node LN receives the network delay test request from control node and deposits
Store up node two-dimensional array NSet.
The network delay test program of 4.2 memory node LN successively tests the network in LN and NSet between each memory node
Delay stores test result using network delay test result array NL, and NL [s] indicates memory node LN and memory node NSet
Network delay between [s] [1], the specific steps are as follows:
4.2.1 memory node serial number s=1 is initialized.
4.2.2 test memory node LN and memory node NSet [s] [1] between network delay Lat, enable NL [s]=
Lat。
4.2.3 node ID, s=s+1 are updated storage.
4.2.4 if s=k+m, network delay test terminates, and turns the 4.3rd step；Otherwise, 4.2.2 step is gone to, after
Continuous test network delay.
The network delay test program of 4.3 memory node LN sends network delay test result array NL to control node.
5th step, control node receive the network delay test result array NL from memory node LN；According to NL from number
It is saved according to selecting k available block and its place to store in k+m-1 (removing fail data block) a available block of object DO
Point, specific method are:
The task management program of 5.1 control nodes receives the network delay test result array NL from memory node LN.
The task management program of 5.2 control nodes is a according to the k+m-1 (removing fail data block) of NL from data object DO
K available block is selected in available block, it can using k that selected memory node two-dimensional array RNSet storage is selected
With where data block memory node and its place data center, RNSet [t] [1] indicate t-th selected of available block
Place memory node, RNSet [t] [2] are indicated in the place data of selected t-th of available block place memory node
The heart, 1≤t≤k, the specific steps are as follows:
5.2.1 selected memory node serial number t=1 is initialized.
5.2.2 according to network delay test result array NL, from k+m-1 (removing fail data block) a available block
Select t-th of available block and its place memory node, the specific steps are as follows:
5.2.2.1 the memory node serial number s=k+m-1 where available block is initialized.
5.2.2.2 if NL [s]=0, enables s=s-1, turn 5.2.2.2 step；Otherwise, minimum network delay MinL=
NL [s], the corresponding memory node serial number ms=s of minimum network delay.
5.2.2.3 if MinL > NL [s-1] and NL [s-1] > 0, enable MinL=NL [s-1], ms=s-1.
5.2.2.4 s=s-1 is enabled.
5.2.2.5 if s=0, find t-th of available block and its place memory node, RNSet [t]
[1]=NSet [ms] [1], RNSet [t] [2]=NSet [ms] [2], NL [ms]=0 turn 5.2.3 step；Otherwise, turn
5.2.2.3 step continues comparing cell delay.
5.2.3 selected memory node serial number, t=t+1 are updated.
5.2.4 if t > k, k available block and its place memory node have been found, the 6th step is turned；It is no
Then, turn 5.2.2 step, continue to select available block.
6th step, control node execute task management program, and the place of the k available block selected to the 5th step stores
Node sends fail data block DiRepair request, data available block number and its desorption coefficient Hir(1≤i≤k, 1≤r≤k), quilt
Select memory node two-dimensional array RNSet, fail data block DiNew memory node LN number.
The place memory node of 7th step, k available block of the 5th step selection executes repair procedure, receives and carrys out automatic control
The fail data block D of node processedi(1≤i≤k) repairs request, data available block number and its desorption coefficient Hir(1≤i≤k, 1
≤ r≤k), selected memory node two-dimensional array RNSet, fail data block DiNew memory node LN number；Based on available number
According to block and its desorption coefficient HirIt is decoded calculating, each data center in RNSet first merges decoding calculated result,
Then the intermediate decoded results after merging are sent to LN.
The repair procedure of the place memory node of 7.1 k selected available blocks is decoded calculating, obtains k
New data block, i.e. Si1..., Sir..., Sik, the specific steps are as follows:
7.1.1 selected memory node serial number r=1 is initialized.
7.1.2 the repair procedure of memory node RNSet [r] [1] receives the fail data block D from control nodeiIt repairs
Request, data available block number and its desorption coefficient Hir, selected memory node two-dimensional array RNSet, fail data block DiIt is new
The number of memory node LN.
7.1.3 the repair procedure of memory node RNSet [r] [1] according to data available block number obtain be locally stored can
With data block Er, carry out local decoding and calculate, by desorption coefficient HirWith available block ErIt carries out wise multiplication and obtains new data
Block Sir, that is, Sir=Hir×Er。
7.1.4 if r < k, enables r=r+1, turn 7.1.2 step；Otherwise, decoding calculates and terminates, and turns the 7.2nd step.
The repair procedure of the place memory node of 7.2 k selected available blocks carries out the solution inside data center
Code result merges, the intermediate decoded results after being merged, the specific steps are as follows:
7.2.1 it is using the decoding calculated result of k selected available block of data block status array CSet record
No to be transmitted and merge, initializing each element in CSet is 0, i.e. CSet [1] ... ..., CSet [r] ... ..., CSet
[k] is 0 (1≤r≤k).
7.2.2 selected memory node serial number r=1 is initialized.
7.2.3 follow-up storage node ID x=r+1 is initialized.
7.2.4 if CSet [x]=0 and RNSet [r] [2]=RNSet [x] [2], memory node RNSet [x]
[1] the data block S that decoding is calculated repair procedureixIt is sent to memory node RNSet [r] [1], memory node RNSet
The repair procedure of [r] [1] receives the decoding calculation result data block S that memory node RNSet [x] [1] is sentix, by SixWith this
Ground decodes calculation result data block SirIt merges, that is, Sir=Sir+Six(i.e. by data block SirAnd SixIt is added, is enabled by turn
CSet [x]=1 turns 7.2.5 step；Otherwise, directly turn 7.2.5 step.
7.2.5 if x < k, enables x=x+1, turn 7.2.4 step；Otherwise, r=r+1 turns 7.2.6 step.
7.2.6 if r < k, turn 7.2.3 step；Otherwise, data center's inner decoding result merging terminates, and turns the 7.3rd
Step.
Intermediate decoded results after merging are sent to fail data block D by 7.3 each data centersiNew memory node LN,
Specific step is as follows:
7.3.1 selected memory node serial number r=1 is initialized.
7.3.2 if CSet [r]=0, the repair procedure of memory node RNSet [r] [1] is by data block SirIt is sent to mistake
Imitate data block DiNew memory node LN, turn 7.3.3；If CSet [r] ≠ 0, directly turns 7.3.3.
7.3.3 if r < k, enables r=r+1, turn 7.3.2 step；Otherwise, intermediate decoded results transmission terminates, and turns the 8th
Step.
8th step, fail data block DiNew memory node LN receive the intermediate decoded results that each data center sends, will
The intermediate decoded results received, which merge, finally to be decoded as a result, the D restoredi, to control node send repair at
Function information.
The result reclaimer of 9th step, control node receives the reparation successful information from memory node LN, to user
It returns and repairs successful information.
Compared with prior art, the present invention has following technological merit:
1. 4.2 step fail data block D of the present inventioniNew memory node LN receive from control node network delay test
The network delay between the memory node of each available block place is successively tested after request, and network delay test has been effectively ensured
Accuracy.
2. 5.2 step control nodes of the invention select available block and its place according to the network delay between memory node
Memory node effectively reduces the network transmission expense of repair data, establishes base to effectively improve the remediation efficiency of fail data
Plinth.
3. being carried out inside data center between the memory node where k selected available block of 7.2 steps of the invention
Decoding result merge, the intermediate decoded results after merging are then sent to fail data block DiNew memory node LN, fill
Divide the bandwidth resources being utilized inside data center, reduces the transport overhead between data center, effectively increase failure number
According to remediation efficiency.
Detailed description of the invention
Fig. 1 is the physical structure of the distributed storage system of first step building of the present invention.
Fig. 2 is the software deployment figure of distributed storage system of the present invention.
Fig. 3 is overview flow chart of the present invention.
Specific embodiment
Fig. 3 is general flow chart of the present invention.Detailed process is as follows:
The first step constructs a distributed storage system as shown in Figure 1, it is by M data center and a control node
Composition, each data center includes N number of memory node, each node (including control node and memory node) be one can
Independently operated computer, each node are interconnected by the network equipment.M and N is positive integer.The node of distributed storage system is by controlling
Node and memory node composition processed.Control node and memory node are comprising processor, memory, disk and network interface
Computer.It is connected between control node and memory node by interference networks.Behaviour is mounted in control node and memory node
Make system, ICP/IP protocol software, is configured with network environment.Control node is responsible for interacting with user, receives the mistake that user submits
Imitate data block reparation request；It is responsible for fail data block and selects new memory node, and send network to new memory node and prolong
Slow test request；It is responsible for selecting available block and its place memory node according to network delay；It is responsible for storage desorption coefficient, to
Each memory node distribution fail data block, which is repaired task and received, repairs successful information, returns to user and repairs successful information.It deposits
Storage node is responsible for the original data block and redundant data block of storing data object, executes network delay test assignment, and to control
Node returns to network delay test result；It executes fail data block and repairs task, and returned to control node and repair successful information.
As shown in Fig. 2, being equipped with task management program and result reclaimer in control node.Task management program receives
The fail data block reparation request that user submits selects a new memory node for fail data block, and sends out to new memory node
The request of SCN Space Cable Network delayed test；K+m-1 (removing fail data block) according to network delay test result from data object DO is a
K available block and its place memory node are selected in available block；To the memory node hair where k available block
Relevant information needed for sending fail data block reparation to request and repair.As a result reclaimer be responsible for receive memory node reparation at
Function information simultaneously returns to user.Network delay test program and repair procedure are installed on memory node.Network delay tests journey
Sequence is responsible for receiving the network delay test request from control node, carries out network delay test, sends network to control node
Delayed test result.Repair procedure is responsible for receiving the request of the fail data block reparation from control node, and on memory node
The available block of storage is decoded calculating, completes to send after repairing to control node and repairs successful information.
Second step, control node execute task management program, are fail data block D to be repairediSelect new memory node
LN。
Third step, control node execute task management program, send network delay test request to memory node LN and deposit
Store up node two-dimensional array NSet.
4th step, memory node LN execute network delay test program, receive the network delay test from control node
Request and memory node two-dimensional array NSet, successively the network delay in test and NSet between each memory node, is saved to control
Point sends network delay test result array NL.
5th step, control node execute task management program, receive the network delay test result from memory node LN
Array NL；It selection k can from k+m-1 (removing fail data block) a available block of data object DO according to network delay
With data block and its place memory node.
6th step, control node execute task management program, and the place of the k available block selected to the 5th step stores
Node sends fail data block DiRepair request, data available block number and its desorption coefficient Hir, selected memory node two-dimemsional number
Group RNSet, fail data block DiNew memory node LN number.
The place memory node of 7th step, k available block of the 5th step selection executes repair procedure, receives and carrys out automatic control
The fail data block D of node processediRepair request, data available block number and its desorption coefficient Hir, selected memory node two-dimemsional number
Group RNSet, fail data block DiNew memory node LN number；Based on available block and its desorption coefficient HirIt is decoded
It calculates, first merges decoding calculated result inside each data center in RNSet, then by the intermediate decoded after merging
As a result it is sent to LN.
8th step, fail data block DiNew memory node LN receive the intermediate decoded results that each data center sends, will
The intermediate decoded results received, which merge, finally to be decoded as a result, the D restoredi, to control node send repair at
Function information.
The result reclaimer of 9th step, control node receives the reparation successful information from memory node LN, to user
It returns and repairs successful information.
Claims (6)
1. a kind of span centre heart cooperative repair method based on correcting and eleting codes, it is characterised in that the following steps are included:
The first step constructs a distributed storage system, it is made of M data center and a control node, in each data
The heart includes N number of memory node, each node be one can independently operated computer, each node is mutual by the network equipment
Even, M and N are positive integer；
It is mounted on operating system, ICP/IP protocol software in control node and memory node, is configured with network environment；
Control node is responsible for interacting with user, receives the fail data block reparation request that user submits；It is responsible for fail data block
New memory node is selected, and sends network delay test request to new memory node；Being responsible for can according to network delay selection
With data block and its place memory node；It is responsible for storage desorption coefficient, repairs task to each memory node distribution fail data block
Successful information is repaired with receiving, is returned to user and repairs successful information；
Memory node is responsible for the original data block and redundant data block of storing data object, executes network delay test assignment, and
Network delay test result is returned to control node；Execute fail data block repair task, and to control node return repair at
Function information；Data object DO is divided into k original data block i.e. D1, D2... ..., Di..., Dk, 1≤i≤k, k are positive whole
Number carries out coding to k original data block and m redundant data block is calculated, and m is positive integer, this k+m data block is deposited respectively
Storage is on k+m memory node, k+m < N；In distributed storage system, each data block possesses unique data block number；
Task management program and result reclaimer are installed in control node；Task management program receives the failure that user submits
Data block reparation request selects a new memory node for fail data block, and sends network delay test to new memory node
Request；It is selected from k+m-1 available block of the removing fail data block of data object DO according to network delay test result
Select k available block and its place memory node；Fail data block is sent to the memory node where k available block to repair
Relevant information needed for multiple request and reparation；As a result reclaimer is responsible for receiving the reparation successful information of memory node and be returned to
User；
Network delay test program and repair procedure are installed on memory node；Network delay test program, which is responsible for receiving, carrys out automatic control
The network delay test request of node processed carries out network delay test, sends network delay test result to control node；It repairs
Program be responsible for receiving the fail data block reparation from control node request, and to the available block stored on memory node into
Row decoding calculates, and completes to send after repairing to control node and repairs successful information；
Second step, control node are fail data block D to be repairediSelect new memory node LN；
Third step, control node send network delay test request and memory node two-dimensional array NSet, tool to memory node LN
Body method is:
The task management program of 3.1 control nodes sends network delay test request to memory node LN；
The task management program of 3.2 control nodes is by the removing fail data block D of data object DOiThe available number of k+m-1 in addition
Constituting memory node two-dimensional array NSet, NSet [s] [1] according to the memory node where block indicates s-th of available block place
Memory node, NSet [s] [2] indicate s-th of available block where memory node where data center, 1≤s≤k+
M-1 sends memory node array NSet to memory node LN；
4th step, memory node LN receive network delay test request and memory node two-dimensional array from control node
NSet successively tests the network delay in LN and NSet between each memory node, sends network delay test knot to control node
Fruit array NL, specific method is:
The network delay test program of 4.1 memory node LN receives network delay test request and storage section from control node
Point two-dimensional array NSet；
The network that the network delay test program of 4.2 memory node LN is successively tested in LN and NSet between each memory node prolongs
Late, test result is stored using network delay test result array NL, NL [s] indicates memory node LN and memory node NSet
Network delay between [s] [1]；
The network delay test program of 4.3 memory node LN sends network delay test result array NL to control node；
5th step, control node receive the network delay test result array NL from memory node LN；According to NL from data pair
As the removing fail data block of DO k+m-1 available block in select k available block and its place memory node, have
Body method is:
The task management program of 5.1 control nodes receives the network delay test result array NL from memory node LN；
The task management program of 5.2 control nodes is a available according to the k+m-1 of NL from the removing fail data block of data object DO
K available block is selected in data block, using the k available numbers that selected memory node two-dimensional array RNSet storage is selected
According to where block memory node and its place data center, RNSet [t] [1] indicate where t-th selected of available block
Memory node, the place data center of memory node where RNSet [t] [2] indicates t-th of available block being selected, 1≤
t≤k；
6th step, control node execute task management program, the place memory node of the k available block selected to the 5th step
Send fail data block DiRepair request, data available block number and its desorption coefficient Hir, selected memory node two-dimensional array
RNSet, fail data block DiNew memory node LN number, 1≤r≤k；
The place memory node of 7th step, k available block of the 5th step selection executes repair procedure, receives from control section
The fail data block D of pointiRepair request, data available block number and its desorption coefficient Hir, selected memory node two-dimensional array
RNSet, fail data block DiNew memory node LN number；Based on available block and its desorption coefficient HirIt is decoded meter
It calculates, each data center in RNSet first merges decoding calculated result, then sends out the intermediate decoded results after merging
Give LN:
The repair procedure of the place memory node of 7.1 k selected available blocks is decoded calculating, obtains new data block
Si1..., Sir..., Sik；
The repair procedure of the place memory node of 7.2 k selected available blocks carries out the decoding knot inside data center
Fruit merges, and the intermediate decoded results after being merged, method is:
7.2.1 using the decoding calculated result of k selected available block of data block status array CSet record whether into
Row transmission and merging, initializing each element in CSet is 0, i.e. CSet [1] ... ..., CSet [r] ... ..., and CSet [k] is equal
=0；
7.2.2 selected memory node serial number r=1 is initialized；
7.2.3 follow-up storage node ID x=r+1 is initialized；
7.2.4 if CSet [x]=0 and RNSet [r] [2]=RNSet [x] [2], the reparation of memory node RNSet [x] [1]
The data block S that decoding is calculated programixIt is sent to memory node RNSet [r] [1], memory node RNSet [r] [1] is repaired
Multiple program receives the decoding calculation result data block S that memory node RNSet [x] [1] is sentix, by SixIt is calculated with local decoding
Result data block SirIt merges, i.e. Sir=Sir+Six, CSet [x]=1 is enabled, 7.2.5 step is turned；Otherwise, directly turn 7.2.5
Step；
7.2.5 if x < k, enables x=x+1, turn 7.2.4 step；Otherwise, r=r+1 turns 7.2.6 step；
7.2.6 if r < k, turn 7.2.3 step；Otherwise, data center's inner decoding result merging terminates；
Intermediate decoded results after merging are sent to fail data block D by 7.3 each data centersiNew memory node LN；
8th step, fail data block DiNew memory node LN receive the intermediate decoded results that each data center sends, will receive
Intermediate decoded results merge finally decoded as a result, i.e. restore Di, sent to control node and repair successful information；
The result reclaimer of 9th step, control node receives the reparation successful information from memory node LN, returns to user
Repair successful information.
2. a kind of span centre heart cooperative repair method based on correcting and eleting codes as described in claim 1, it is characterised in that second step institute
Stating control node is fail data block D to be repairediSelecting the specific method of new memory node LN is:
The task management program of 2.1 control nodes receives the fail data block D that user submitsiRepair request；
The task management program of 2.2 control nodes selects a memory node as fail data block D from available memory nodei
New memory node LN, selection principle is any data block of the not stored data object DO of memory node LN.
3. a kind of span centre heart cooperative repair method based on correcting and eleting codes as described in claim 1, it is characterised in that the 4th step by step
The network that the network delay test program of the rapid 4.2 memory node LN is successively tested in LN and NSet between each memory node prolongs
Slow method is:
4.2.1 memory node serial number s=1 is initialized；
4.2.2 the network delay Lat between memory node LN and memory node NSet [s] [1] is tested, NL [s]=Lat is enabled；
4.2.3 node ID, s=s+1 are updated storage；
4.2.4 if s=k+m, terminate；Otherwise, 4.2.2 step is gone to.
4. a kind of span centre heart cooperative repair method based on correcting and eleting codes as described in claim 1, it is characterised in that the 5th step by step
The task management program of rapid 5.2 control node can from k+m-1 of the removing fail data block of data object DO according to NL
Specific method with k available block of selection in data block is:
5.2.1 selected memory node serial number t=1 is initialized；
5.2.2 it according to network delay test result array NL, is selected from k+m-1 available block for removing fail data block
T-th of available block and its place memory node, the specific steps are as follows:
5.2.2.1 the memory node serial number s=k+m-1 where available block is initialized；
5.2.2.2 if NL [s]=0, enables s=s-1, turn 5.2.2.2 step；Otherwise, minimum network delay MinL=NL
[s], the corresponding memory node serial number ms=s of minimum network delay；
5.2.2.3 if MinL > NL [s-1] and NL [s-1] > 0, enable MinL=NL [s-1], ms=s-1；
5.2.2.4 s=s-1 is enabled；
5.2.2.5 t-th of available block and its place memory node have been found if s=0, RNSet [t] [1]=
NSet [ms] [1], RNSet [t] [2]=NSet [ms] [2], NL [ms]=0 turn 5.2.3 step；Otherwise, turn 5.2.2.3
Step；
5.2.3 selected memory node serial number, t=t+1 are updated；
5.2.4 if t > k, terminate；Otherwise, turn 5.2.2 step, continue to select available block.
5. a kind of span centre heart cooperative repair method based on correcting and eleting codes as described in claim 1, it is characterised in that the 7th step by step
The repair procedure of the place memory node of the rapid 7.1 k selected available block is decoded the specific method of calculating
It is:
7.1.1 selected memory node serial number r=1 is initialized；
7.1.2 the repair procedure of memory node RNSet [r] [1] receives the fail data block D from control nodeiReparation request,
Data available block number and its desorption coefficient Hir, selected memory node two-dimensional array RNSet, fail data block DiNew storage section
The number of point LN；
7.1.3 the repair procedure of memory node RNSet [r] [1] obtains the available number being locally stored according to data available block number
According to block Er, carry out local decoding and calculate, by desorption coefficient HirWith available block ErIt carries out wise multiplication and obtains new data block Sir,
That is Sir=Hir×Er；
7.1.4 if r < k, enables r=r+1, turn 7.1.2 step；Otherwise, decoding calculates and terminates, and obtains Si1..., Sir...,
Sik。
6. a kind of span centre heart cooperative repair method based on correcting and eleting codes as described in claim 1, it is characterised in that the 7th step by step
Intermediate decoded results after merging are sent to fail data block D by rapid 7.3 each data centeriNew memory node LN tool
Body method is:
7.3.1 initializing selected memory node serial number r=1；
7.3.2 if CSet [r]=0, the repair procedure of memory node RNSet [r] [1] is by data block SirIt is sent to failure number
According to block DiNew memory node LN, turn 7.3.3；If CSet [r] ≠ 0, directly turns 7.3.3；
7.3.3 if r < k, enables r=r+1, turn 7.3.2 step；Otherwise, intermediate decoded results transmission terminates.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810773745.XA CN109067852A (en) | 2018-07-15 | 2018-07-15 | Cross-center cooperative repair method based on erasure codes |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810773745.XA CN109067852A (en) | 2018-07-15 | 2018-07-15 | Cross-center cooperative repair method based on erasure codes |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109067852A true CN109067852A (en) | 2018-12-21 |
Family
ID=64816570
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810773745.XA Pending CN109067852A (en) | 2018-07-15 | 2018-07-15 | Cross-center cooperative repair method based on erasure codes |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109067852A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109889440A (en) * | 2019-02-20 | 2019-06-14 | 哈尔滨工程大学 | A kind of correcting and eleting codes failure node reconstruct routing resource based on maximum spanning tree |
| CN112256471A (en) * | 2020-10-19 | 2021-01-22 | 北京京航计算通讯研究所 | Erasure code repairing method based on separation of network data forwarding and control layer |
| CN112445656A (en) * | 2020-12-14 | 2021-03-05 | 北京京航计算通讯研究所 | Method and device for repairing data in distributed storage system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607304A (en) * | 2013-11-21 | 2014-02-26 | 中国人民解放军国防科学技术大学 | Erasure code based failure data linear restoration method |
| CN103647815A (en) * | 2013-12-03 | 2014-03-19 | 清华大学 | An optimal storage cloud construction method based on erasure codes and selection factors |
| CN103729151A (en) * | 2014-01-16 | 2014-04-16 | 中国人民解放军国防科学技术大学 | Failure data recovery method based on improved erasure codes |
| CN103944981A (en) * | 2014-04-14 | 2014-07-23 | 中国科学院计算技术研究所 | Cloud storage system and implement method based on erasure code technological improvement |
| CN104035732A (en) * | 2014-06-17 | 2014-09-10 | 中国人民解放军国防科学技术大学 | Data placing method aiming at erasure codes |
| CN104393960A (en) * | 2014-12-09 | 2015-03-04 | 南京邮电大学 | Erasure code based multi-path reliable satellite-ground transmission method for satellite networks |
| CN107436733A (en) * | 2017-06-29 | 2017-12-05 | 华为技术有限公司 | Management by district method and management by district device |
-
2018
- 2018-07-15 CN CN201810773745.XA patent/CN109067852A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607304A (en) * | 2013-11-21 | 2014-02-26 | 中国人民解放军国防科学技术大学 | Erasure code based failure data linear restoration method |
| CN103647815A (en) * | 2013-12-03 | 2014-03-19 | 清华大学 | An optimal storage cloud construction method based on erasure codes and selection factors |
| CN103729151A (en) * | 2014-01-16 | 2014-04-16 | 中国人民解放军国防科学技术大学 | Failure data recovery method based on improved erasure codes |
| CN103944981A (en) * | 2014-04-14 | 2014-07-23 | 中国科学院计算技术研究所 | Cloud storage system and implement method based on erasure code technological improvement |
| CN104035732A (en) * | 2014-06-17 | 2014-09-10 | 中国人民解放军国防科学技术大学 | Data placing method aiming at erasure codes |
| CN104393960A (en) * | 2014-12-09 | 2015-03-04 | 南京邮电大学 | Erasure code based multi-path reliable satellite-ground transmission method for satellite networks |
| CN107436733A (en) * | 2017-06-29 | 2017-12-05 | 华为技术有限公司 | Management by district method and management by district device |
Non-Patent Citations (2)
| Title |
|---|
| XIAOQIANG PEI，YIJIE WANG， XINGKONG MA ，FANGLIANG XU: "Repairing multiple failures adaptively with erasure codes in distributed storage systems", 《WILEY ONLINE LIBRARY (WILEYONLINELIBRARY.COM)》 * |
| 郑志蕴等: "基于纠删码的动态副本冗余存储研究", 《计算机工程与设计》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109889440A (en) * | 2019-02-20 | 2019-06-14 | 哈尔滨工程大学 | A kind of correcting and eleting codes failure node reconstruct routing resource based on maximum spanning tree |
| CN112256471A (en) * | 2020-10-19 | 2021-01-22 | 北京京航计算通讯研究所 | Erasure code repairing method based on separation of network data forwarding and control layer |
| CN112445656A (en) * | 2020-12-14 | 2021-03-05 | 北京京航计算通讯研究所 | Method and device for repairing data in distributed storage system |
| CN112445656B (en) * | 2020-12-14 | 2024-02-13 | 北京京航计算通讯研究所 | Method and device for repairing data in distributed storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103607304B (en) | A kind of linear restorative procedure of fail data based on correcting and eleting codes | |
| CN108540520B (en) | Partial repeated code based locality repairing coding and node fault repairing method | |
| CN110212923B (en) | Distributed erasure code storage system data restoration method based on simulated annealing | |
| CN103729151B (en) | A kind of fail data restorative procedure based on modified model correcting and eleting codes | |
| CN109067852A (en) | Cross-center cooperative repair method based on erasure codes | |
| CN109889440B (en) | Erasure code failure node reconstruction path selection method based on maximum spanning tree | |
| JP2011504269A (en) | Parallel Reed-Solomon RAID (RS-RAID) architecture, device, and method | |
| CN108370255A (en) | Polar code continuously eliminates memory management and paths ordering in list-decoding device | |
| CN107844272A (en) | A kind of cross-packet coding and decoding method for improving error correcting capability | |
| CN104035732B (en) | Data placing method aiming at erasure codes | |
| CN103106124B (en) | Intersection reconstruction method based on erasure code cluster memory system | |
| CN107665154A (en) | Authentic data analysis method based on RDMA and message transmission | |
| CN109194444A (en) | A kind of balanced binary tree restorative procedure based on network topology | |
| CN113190377B (en) | Reliable redundancy method and equipment based on distributed storage system | |
| CN106484559A (en) | A kind of building method of check matrix and the building method of horizontal array correcting and eleting codes | |
| US11645133B2 (en) | Modifying encoded data in a distributed storage network | |
| Tebbi et al. | A code design framework for multi-rack distributed storage | |
| CN108762978A (en) | A kind of constructed in groups method of Part portions repetitive cycling code | |
| CN109062724B (en) | A kind of correcting and eleting codes conversion method and terminal | |
| CN105610879B (en) | Data processing method and device | |
| CN104102558B (en) | Erasure code based file appending method | |
| CN109343998A (en) | Erasure code-based full-distribution restoration method | |
| CN104782101A (en) | Encoding, reconstructing, and recovering methods used for self-repairing code stored by distributed network | |
| CN108304264A (en) | A kind of correcting and eleting codes archiving method based on SPARK streaming computings | |
| US11953991B2 (en) | Method, device and computer program product for storage management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20181221 |