CN105959338B - A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula - Google Patents
A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula Download PDFInfo
- Publication number
- CN105959338B CN105959338B CN201610168643.6A CN201610168643A CN105959338B CN 105959338 B CN105959338 B CN 105959338B CN 201610168643 A CN201610168643 A CN 201610168643A CN 105959338 B CN105959338 B CN 105959338B
- Authority
- CN
- China
- Prior art keywords
- node
- face
- block
- data
- failure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention belongs to cloud storage and information security field, the cloud storage multinode data reconstruction method of mainly a kind of three-dimensional stereo model formula.The present invention includes: that the file division module of cloud storage system carries out linear partition to original document in finite field；Network code policy module receive by file division module transmit Lai set of data blocks；Encoding block memory module receives the set of data blocks after the coding sent by network code policy module.Data block after coding stores the respective nodes to three-dimensional stereo model；Failure node repair module receives the failure node information that nodal test module D1 is sended over.The present invention can not only the single-unit point failure to cloud storage system restore, additionally it is possible to more piece point failure is restored.This method guarantees the reliability of system, after detecting failure node, by restoring to failure node, improves the integrality and safety of system by carrying out periodic detection to memory node.
Description
Technical field
The invention belongs to cloud storage and information security field, mainly a kind of cloud storage multinode of three-dimensional stereo model formula
Data reconstruction method.
Background technique
As an emerging research hotspot, cloud storage is not only a kind of memory technology or equipment, even more a kind of innovation service
Form, there is easily extension, support the PB grade above dilatation, use as needed, the characteristics such as user's flexibly configurable, it is traditional in offer
Except access mode, the function of the management and external public service of mass data is additionally provided.However, quick with cloud service
Promote, illegal invasion, artificially divulge a secret, administrator right abuse etc. due to, cloud storage system leakage privacy of user event not
Disconnected to emerge in large numbers, safety issue is increasingly prominent, causes the extensive concern of industry.Therefore, cloud storage safety issue is solved to become
It obtains very urgent.
It is based on duplication, based on entangling respectively at present there are mainly three types of the mode of safety storage data in cloud storage field
Delete cloud storage scheme of the sum based on network code of code.Wherein, although the first scheme data access is convenient, communication overhead compared with
Greatly, safety and reliability is poor；And the cloud storage scheme based on correcting and eleting codes is compared with the cloud storage scheme based on duplication, although
It is improved to some extent on safety and reliability, but is still faced with high bandwidth consumption and computational load when data are repaired
The problems such as larger.Different from first two scheme, the cloud storage scheme based on network code not only has better safety and can
By property, but also has and save bandwidth consumption, realize load balancing, improve network overall transmission rate and the reconstruct of efficient original text
The features such as.
Article " Reducing Repair Traffic for Erasure Coding-Based Storage via
Interference Alignment ", which proposes the maximum distance based on (n, k), can divide (MDS) code, be directed within the storage system
Individual node failure is restored.Wherein, for the node of failure, system need to download the coded datas of other effective nodes into
Row decoding, regenerates a new node according to obtained linearly related data, to recover the node of failure, maintenance is same
Deng horizontal availability of data and integrality.(http://ieeexplore.ieee.org/stamp/stamp.jsp?
Arnumber=5205898).Article " Explicit Construction of Optimal Exact Regenerating
Exact-MBR codes method is proposed in Codes for Distributed Storage ", this method can be for independent
Memory node failure restored, i.e., the block retained in the block number for needing to be resumed in failure node and other complete nodes
Number is identical, the content of failure node can be recovered from complete node, which can subtract to avoid unwanted interference
Small reparation bandwidth (http://arxiv.org/abs/0906.4913).Article " Exact Regeneration Codes for
Distributed Storage Repair Using Interference Alignment " proposes accurate minimum memory
Code (EMSR codes) method is regenerated, this method can make to repair bandwidth reduction 25% in the case where storage size is constant
(http://arxiv.org/pdf/1001.0107.pdf).Article " NCCloud:A Network-Coding-Based
Storage System in a Cloud-of-Clouds " proposes function minimum memory regeneration code (FMSR codes) method,
This method is further improved on the basis of previous methods, in the case where original is very big, reparation bandwidth can be subtracted
It is small close to 50% (http://www.computer.org/csdl/trans/tc/2014/01/ttc2014010031-
abs.html).Patent " Distributed Storage and repair system and its method (CN104113562A) based on network code "
Bandwidth demand needed for data storage can be reduced and repaired, the load balancing of safeguards system, and have one in Information Security
Fixed raising.
In conclusion current method, is directed generally to reduce storage and repairs bandwidth, improve the peace of storage as much as possible
Quan Xing.But they are only that data recovery is carried out under for single node failure conditions, without considering more piece point failure
The case where.
The present invention is a kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula, and this method is first to file
It is divided, network code then is carried out to the data of division, to reduce storage overhead.Cloud storage system can be to memory node
Periodic detection is carried out, guarantees system reliability.After detecting failure node, failure node is restored, to improve and be
The integrality and safety of system.
Summary of the invention
The purpose of the present invention is to provide a kind of cloud storage multinode data reconstruction methods of three-dimensional stereo model formula.
The object of the present invention is achieved like this:
The cloud storage multinode data reconstruction method of three-dimensional stereo model formula, includes the following steps:
(1) the file division module of cloud storage system carries out linear partition to original document in finite field, marks off n block
The identical data block of size, and n=2k, k >=3, by data chunk at a set of data blocks M ', i.e. M '={ m1’,m2’,
...mn', set of data blocks M ' passes to network code policy module；
(2) network code policy module receive by file division module transmit Lai set of data blocks M ', and to each in M '
Block data block individually carries out network code, the set of data blocks M, i.e. M={ m after forming coding1,m2,...mn, network code strategy
Set of data blocks M after coding is transmitted to encoding block memory module by module；
(3) encoding block memory module receives the set of data blocks M after the coding sent by network code policy module, and
N block data block after coding is stored onto t initial data node, 0 < t < n, and t=2q, q >=2, wherein each node
Storage 2hA data block, constitutes a reference vector, and composition one includesThe reference vector collection SV-set of a reference vector,Non-referenced vector NSV is obtained by operation by reference vectorij And j ≠ i；It is full for any addition, subtraction operation
Sufficient law of communication；α, β are Arbitrary Coefficient, using a kind of three-dimensional of the cloud storage multinode data reconstruction method of three-dimensional stereo model formula
Three-dimensional distributed structure method constructs a three-dimensional stereo model；
(4) respective nodes of the data block storage to three-dimensional stereo model after coding, the base including Memory Reference vector
Quasi- node and after storing the non-referenced node of non-referenced vector, nodal test module D1 regularly detect on each node whether
There are data to be destroyed, whether node fails；If returning to the signal of a safety without node failure, if there is node failure,
Failure node information then is sent to failure node repair module；
(5) failure node repair module receives the failure node information that nodal test module D1 is sended over, and passes through three
The three-dimensional stereo data restoration methods for tieing up the cloud storage multinode data reconstruction method of three-dimensional model formula repair failure node
It is multiple；After failure node restores, step (6) are entered；
(6) after node recovered part restores the node of failure, nodal test module D2 is carried out for the content restored
Detection judges whether recovery data are correct, if correctly, terminating the running in the period；If incorrect, sound a warning, weight
It is new to repair.
The step (3) further include:
(3.1) selecting any one three-dimensional face is benchmark face face 1, the optional n/2 block number from the data block after n block coding
It stores according to block onto s datum node of datum level 1, whereinEach datum node stores 2 in vector formhBlock number evidence
Block, i.e., 2k-1=s*2h, h is the random natural number that sets according to the size of n；
(3.2) it is starting point with datum level face 1, defines principal surface face 2, datum level supplemented by its three-dimensional lap in a clockwise direction
Face 3 and secondary surface face 4, upper side are secondary surface face 5, and downside is secondary surface face 6；
(3.3) it will be stored in the same manner in addition to storing to the data block after n/2 block coding remaining on datum level face 1
Onto another datum level face 3；
(3.4) from the 2 of datum level face 1k-1T is randomly selected in block data block1=2k-2Block data block, remaining T2=2k-2Block number
According to block, from the 2 of datum level face 3k-1T is randomly selected in block data block3=2k-2Block data block, remaining T4=2k-2Block data block；?
T1And T3Composition 2k-1Block data block is stored on secondary surface face 2；T1And T4Composition 2k-1Block data block is stored in secondary surface face 4
On；T2And T3Composition 2k-1Block data block is stored on secondary surface face 5；T2And T4The 2 of compositionk-1Block data block is stored in auxiliary
On face face 6.
The step (5) further include:
(5.1) it sets on each face of three-dimensional stereo model and has p node respectively, whereinThen 3 D stereo
Model contains 6p node altogether, wherein each face includes a non-referenced nodes of s datum node and p-s, wherein I.e. three-dimensional stereo model includes 6s datum node and a non-referenced node of 6* (p-s)；
(5.2) when nodal test module D1 detects a node failure of certain one side in three-dimensional stereo model, pass through
Other nodes on common face are restored；When the failure of i-th of datum node in some face, by remaining p-1 node
Data carry out linear combination operation, recover fail data；
When nodal test module D1 detects that the failure node in three-dimensional stereo model in certain one side is more than one, by three
It ties up three-dimensional distributed structure method and learns that the three-dimensional stereo model shares 6 faces, wherein being respectively face 1 and face there are two datum level
3, four secondary surfaces are respectively face 2, face 4, face 5 and face 6；It obtains secondary surface face 2 and secondary surface face 4 includes in datum level face 1
Data block T1, secondary surface face 5 and secondary surface face 6 include the data block T in datum level face 12, secondary surface face 2 and secondary surface face 5 are wrapped
Data block T in face containing datum level 33, secondary surface face 4 and face secondary surface 6 include the data block T in datum level face 34；Pass through other
In face on node relevant to failure node data linear combination operation, recover the data in failure node；When three-dimensional is vertical
When datum node failure number is x=6s on body Model, this method loses recovery meaning；When datum node on three-dimensional stereo model
Failure number is x=6s-1, and when non-referenced node does not fail, the data of datum node all are resumed out；When three-dimensional is vertical
Datum node failure number is x < 6s-1 on body Model, and non-referenced node relevant to the datum node of failure does not fail
When, the data of datum node all are resumed out.
The beneficial effects of the present invention are:
The invention proposes a kind of cloud storage multinode data reconstruction methods of three-dimensional stereo model formula, and this method can not only
It is enough that the single-unit point failure of cloud storage system is restored, additionally it is possible to which that more piece point failure is restored.This method is by depositing
It stores up node and carries out periodic detection, guarantee the reliability of system, it is extensive by being carried out to failure node after detecting failure node
It is multiple, improve the integrality and safety of system.
Detailed description of the invention
A kind of schematic diagram of the cloud storage multinode data reconstruction method of three-dimensional stereo model formula of Fig. 1；
On a kind of three-dimensional stereo model face 1 of the cloud storage multinode data reconstruction method of three-dimensional stereo model formula of Fig. 2
Specific node repairs exemplary diagram；
A kind of flow chart of the cloud storage multinode data reconstruction method of three-dimensional stereo model formula of Fig. 3.
Specific embodiment
The present invention is described further with reference to the accompanying drawing.
The present invention be directed to cloud storage propose a kind of three-dimensional stereo model formula cloud storage multinode data reconstruction method,
Aiming to solve the problem that can only be to the deficiency that single-unit point failure is restored in existing method, and load is realized by the way of network code
Equilibrium improves the safety of data in cloud storage system.
A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula of the present invention includes following
Comprising modules:
(1) file division module: cloud storage system draws original document using file division module in finite field
Point, to guarantee there is lower redundancy than simple replication theme.
(2) network code policy module: cloud storage system using coding strategy module file division module mark off come
Data block carry out network code, to guarantee the reliability and safety of storage file.
(3) encoding block memory module: cloud storage system is using encoding block memory module the data storage after coding to phase
On the node answered, make these nodes that there is relevance.
(4) nodal test module: the module is divided into two functional modules, is nodal test module D1 and nodal test respectively
Module D2.The function of nodal test module D1 is data storage in encoded to after in respective nodes, and system can be examined periodically
The data on node are surveyed, to guarantee the integrality and safety of data；The function of nodal test module D2 is to work as to have node mistake
When effect, it can detecte whether the node data after repairing is consistent with initial data, to guarantee the consistency of data.
(5) failure node repair module: when there is node failure in system, it is ensured that the integrality of data file, Yun Cun
Storage system can be restored using node of the failure node repair module to failure.
A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula proposed by the present invention, main thought are
A three-dimensional stereo model is constructed, the node comprising data block is dispersed on each face of the three-dimensional model, and each
Data on node can be restored by a face on three-dimensional stereo model or the data on multiple faces, and then improve number
According to reliability and recovery correctness.
A kind of cloud storage multinode data reconstruction method of above-mentioned three-dimensional stereo model formula, comprising the following specific steps
(1) it firstly, the file division module of cloud storage system carries out linear partition to original document in finite field, divides
The identical data block of n (n is natural number) block size out, and n=2k(k >=3), by these data chunks at a set of data blocks
M ', i.e. M '={ m1’,m2’,...mn'}.Later, set of data blocks M ' is transferred to network code policy module.
(2) network code policy module receive by file division module transmit Lai set of data blocks M ', and to each in M '
Block data block individually carries out network code (such as Exact-MBR Codes method etc.), the set of data blocks M, i.e. M after forming coding
={ m1,m2,...mn}.Hereafter, the set of data blocks M after coding is transmitted to encoding block memory module by network code policy module.
(3) encoding block memory module receives the set of data blocks M after the coding sent by network code policy module, and
The n block data block storage after coding to t (0 < t < n, and t=2q, q >=2) and on a initial data node, wherein each node
Storage 2h(wherein h is the random natural number that sets according to the size of n) a data block, they constitute a reference vectors, by
This constitutes oneThe reference vector collection SV-set of a reference vector, ifBy
Reference vector obtains non-referenced vector NSV by operationij,And j ≠ i；For any addition, subtraction operation, meet
Law of communication；α, β are Arbitrary Coefficient).Using a kind of three-dimensional of the cloud storage multinode data reconstruction method of three-dimensional stereo model formula
Three-dimensional distributed structure method constructs a three-dimensional stereo model.
(4) respective nodes (including datum node (Memory Reference of data block storage after coding to three-dimensional stereo model
Vector) and non-referenced node (storing non-referenced vector)) on after, nodal test module D1 can regularly be detected on each node
Whether there are data to be destroyed, whether node fails.If the signal of a safety can be returned without node failure, if there is node
Failure then sends failure node information to failure node repair module.
(5) failure node repair module receives the failure node information that nodal test module D1 is sended over, and passes through one
The three-dimensional stereo data restoration methods of the cloud storage multinode data reconstruction method of kind of three-dimensional stereo model formula to failure node into
Row is repaired.After failure node restores, step (6) are entered.
(6) after node recovered part restores the node of failure, nodal test module D2 can for the content restored into
Row detection judges whether recovery data are correct, if correctly, terminating the running in the period；If incorrect, sound a warning,
Again it repairs.
The wherein three-dimensional of the cloud storage multinode data reconstruction method of one of above-mentioned steps (3) three-dimensional stereo model formula
Three-dimensional distributed structure method, specifically further include:
1) selecting any one three-dimensional face is benchmark face face 1, the optional n/2 block number evidence from the data block after n block coding
Block store to datum level 1 s (wherein) on a datum node, each datum node stores 2 in vector formh(h is root
The random natural number set according to the size of n) block data block, i.e., 2k-1=s*2h；
2) it is starting point with datum level face 1, defines principal surface face 2, datum level face 3 supplemented by its three-dimensional lap in a clockwise direction
With secondary surface face 4, upper side is secondary surface face 5, and downside is secondary surface face 6；
3) it will be arrived in addition to storage stores in the same manner to the data block after n/2 block coding remaining on datum level face 1
On another datum level face 3；
4) from the 2 of datum level face 1k-1T is randomly selected in block data block1=2k-2Block data block, remaining T2=2k-2Block number evidence
Block, from the 2 of datum level face 3k-1T is randomly selected in block data block3=2k-2Block data block, remaining T4=2k-2Block data block.T1
And T3Composition 2k-1Block data block is stored on secondary surface face 2.T1And T4Composition 2k-1Block data block is stored on secondary surface face 4.
T2And T3Composition 2k-1Block data block is stored on secondary surface face 5.T2And T4The 2 of compositionk-1Block data block is stored in secondary surface face
On 6.
The wherein 3 D stereo of the cloud storage multinode data reconstruction method of one of above-mentioned steps (5) three-dimensional
Data reconstruction method, specifically further include:
1) it sets on each face of three-dimensional stereo model and has p respectively (wherein) a node, then 3 D stereo mould
Type altogether contains 6p node, wherein each face include s (wherein) a datum node and p-s are (wherein) a
Non-referenced node, i.e. three-dimensional stereo model include 6s (i.e. 3t) a datum node and a non-referenced node of 6* (p-s).
2) when nodal test module D1 detects a node failure of certain one side in three-dimensional stereo model, can pass through
Other nodes on common face are restored.It, can be by remaining p-1 node when the failure of i-th of datum node in some face
Data carry out linear combination operation, recover fail data, non-referenced node failure do not influence entire data integrality and
Safety.
3) when nodal test module D1 detects that the failure node in three-dimensional stereo model in certain one side is more than one, by
3 D stereo distributed structure method learns that the three-dimensional stereo model shares 6 faces, wherein being respectively 1 He of face there are two datum level
Face 3, four secondary surfaces are respectively face 2, face 4, face 5 and face 6.It can also further obtain secondary surface face 2 and secondary surface face 4 includes
Data block T in datum level face 11, secondary surface face 5 and secondary surface face 6 include the data block T in datum level face 12, secondary surface face 2
With the data block T that secondary surface face 5 includes in datum level face 33, secondary surface face 4 and face secondary surface 6 include the number in datum level face 3
According to block T4.It can thus be appreciated that the data of node and the data of node on datum level have correlation on three-dimensional stereo model secondary surface,
So can be recovered in failure node by the linear combination operation of data on node relevant to failure node in other faces
Data.When datum node failure number is x (wherein x=6s) on three-dimensional stereo model, this method loses recovery meaning；When three
Tieing up datum node failure number on three-dimensional model is x (wherein x=6s-1), and when non-referenced node does not fail, datum node
Data all can be resumed out；When on three-dimensional stereo model datum node failure number be x (wherein x < 6s-1), and with failure
Datum node relevant non-referenced node when not failing, the data of datum node all can be resumed out.
Fig. 1 illustrates how file is handled in cloud storage system.It is stored in cloud storage system first
File will pass through file division module, file is divided into the identical data block of n block size, later by network code strategy mould
Block carries out coded treatment, and the data block after coding can be assigned to corresponding memory node and be stored, meanwhile, the node of system
Detection module D1 can regularly detect whether these nodes are destroyed, after detecting node i failure, the nodal test of system
Module D1 is notified that the failure node repair module of system restores failure node by three-dimensional stereo data restoration methods,
Meanwhile whether the data that nodal test module D2 can be detected after repairing are consistent with preprosthetic data, to guarantee data
Integrality.
Fig. 3 illustrates specific three-dimensional stereo model node in the present invention and repairs schematic diagram.Assuming that the text of cloud storage system
Original document has been divided into 16 (n=16) block data blocks, i.e. M '={ m by part division module1’,m2’,...,m16', after coding
Set of data blocks is M={ m1,m2,...,m16, select every 4 (2h=4, i.e., selection h=2) block storage on one node, then generate
4 initial data nodes, i.e. t=4, specific Node distribution are Node1:A={ m1,m2,m3,m4, Node2:B={ m5,m6,m7,
m8, Node3:C={ m9,m10,m11,m12, Node4:D={ m13,m14,m15,m16}.The three-dimensional stereo model one of composition is shared
18 nodes have 3 nodes on each face, wherein 2 datum nodes and 1 non-referenced node, i.e. p=3, s=2.Optional three
A face for tieing up three-dimensional model is benchmark face face 1, and three knot vectors distribution on the face is as follows:
Node11:A1={ m1,m2,m3,m4}
Node12:B1={ m5,m6,m7,m8}
Node 13:Wherein α1, β1For coefficient.
WhereinFor any addition, subtraction operation symbol, meet law of communication.
It is starting point with datum level face 1, defines principal surface face 2,3 and of datum level face supplemented by its three-dimensional lap in a clockwise direction
Secondary surface face 4, upper side are secondary surface face 5, and downside is secondary surface face 6.
The knot vector distribution in datum level face 3 is as follows:
Node31:A3={ m9,m10,m11,m12}
Node32:B3={ m13,m14,m15,m16}
Node33:Wherein α2, β2For coefficient.
WhereinFor any addition, subtraction operation symbol, meet law of communication.
The knot vector distribution in secondary surface face 2 is as follows:
Node21:A2={ m1,m2,m3,m4}
Node22:B2={ m9,m10,m11,m12}
Node23:Wherein α3, β3For coefficient.
WhereinFor any addition, subtraction operation symbol, meet law of communication.
The knot vector distribution in secondary surface face 4 is as follows:
Node41:A4={ m1,m2,m3,m4}
Node42:B4={ m13,m14,m15,m16}
Node43:Wherein α4, β4For coefficient.
WhereinFor any addition, subtraction operation symbol, meet law of communication.
The knot vector distribution in secondary surface face 5 is as follows:
Node51:A5={ m5,m6,m7,m8}
Node52:B5={ m9,m10,m11,m12}
Node53:Wherein α5, β5For coefficient.
WhereinFor any addition, subtraction operation symbol, meet law of communication.
The knot vector distribution in secondary surface face 6 is as follows:
Node61:A6={ m5,m6,m7,m8}
Node62:B6={ m13,m14,m15,m16}
Node63:Wherein α6, β6For coefficient.
WhereinFor any addition, subtraction operation symbol, meet law of communication.
1) it when some node failure in some face, can be total to by the data vector on other nodes of the same face
With completion repair content repair text.Assuming that the Node11 on datum level face 1 fails, the content to fail on node can be by Node12 and Node13
Plus and minus calculation is carried out to obtain；If failure node is non-datum node Node13, the integrality of file is not influenced.
2) when having multiple node failures in certain one side, the three-dimensional stereo model known to 3 D stereo distributed structure method is each
A face has correlation, so that multiple nodes of failure can be restored by the node of other faces.Assuming that the Node11 on face 1
It being all destroyed with the data of Node12, the data of Node11 are restored by the knot vector progress plus and minus calculation on face 2 and face 4,
The data of Node12 carry out plus and minus calculation by the knot vector on face 5 and face 6 to restore.
Fig. 3 illustrates main flow of the invention.Firstly, storage file is divided into the identical n of size by cloud storage system
(n is natural number to block, and n=2k, k >=3) and data block, form set of data blocks M '.Second step is each piece of data block list in M '
Network code is solely carried out, the set of data blocks M after forming coding.Third step is that the data block after n block coding is respectively stored into phase
On the memory node (including datum node and non-referenced node) answered, and a three-dimensional is constructed using 3 D stereo distributed structure method
Three-dimensional model.4th step is nodal test module D1, and whether the data in part meeting periodic detection node are complete, and whether node
Failure, if not detecting failure node, terminates the detection in this period, if detecting failure node, will enter
Next step node repair module.In node repair module, system carries out failure node by three-dimensional stereo data restoration methods
It repairs, later, nodal test module D2 can detect the node data after recovery, and whether the data for judging that it restores are just
True, if restoring correct, terminates the running in the period, otherwise, repaired again.
The beneficial effects of the present invention are embodied in:
(1) the invention proposes a kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula, this method is not
Only can the single-unit point failure to cloud storage system restore, additionally it is possible to more piece point failure is restored.
(2) this method guarantees the reliability of system, is detecting failure node by carrying out periodic detection to memory node
Afterwards, by restoring to failure node, the integrality and safety of system are improved.
Claims (1)
1. a kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula, which comprises the steps of:
(1) the file division module of cloud storage system carries out linear partition to original document in finite field, marks off n block size
Identical data block, and n=2k, k >=3, by data chunk at a set of data blocks M ', i.e. M '={ m1’,m2’,...mn', number
Network code policy module is passed to according to block collection M '；
(2) network code policy module receive by file division module transmit Lai set of data blocks M ', and to block number each in M '
Network code is individually carried out according to block, the set of data blocks M, i.e. M={ m after forming coding1,m2,...mn, network code policy module
Set of data blocks M after coding is transmitted to encoding block memory module；
(3) encoding block memory module receives the set of data blocks M after the coding sent by network code policy module, and volume
In n block data block storage to t initial data node after code, 0 < t < n, and t=2q, q >=2, wherein each node storage 2h
A data block, constitutes a reference vector, and composition one includesThe reference vector collection SV-set of a reference vector,H is the random natural number set according to the size of n, passes through operation by reference vector
Obtain non-referenced vector NSVij,And j ≠ i；For any addition, subtraction operation, meet law of communication；α, β are Arbitrary Coefficient, using a kind of cloud storage more piece of three-dimensional stereo model formula
The 3 D stereo distributed structure method of point data restoration methods constructs a three-dimensional stereo model；
(4) respective nodes of the data block storage to three-dimensional stereo model after coding, the benchmark section including Memory Reference vector
After the non-referenced node for putting and storing non-referenced vector, nodal test module D1 regularly detects on each node whether have number
According to being destroyed, whether node fails；If returning to the signal of a safety without node failure, if having node failure, to mistake
It imitates node repair module and sends failure node information；
(5) failure node repair module receives the failure node information that nodal test module D1 is sended over, vertical by three-dimensional
The three-dimensional stereo data restoration methods of the cloud storage multinode data reconstruction method of body Model formula repair failure node；It loses
After imitating node recovery, step (6) are entered；
(6) after node recovered part restores the node of failure, nodal test module D2 is detected for the content restored,
Judge whether recovery data are correct, if correctly, terminating the running in the period；If incorrect, sound a warning, repairs again
It is multiple；
The step (3) further include:
(3.1) selecting any one three-dimensional face is benchmark face face 1, the optional n/2 block data block from the data block after n block coding
It stores on s datum node of datum level 1, whereinEach datum node stores 2 in vector formhBlock data block,
I.e. 2k-1=s*2h, h is the random natural number that sets according to the size of n；
(3.2) it is starting point with datum level face 1, defines principal surface face 2, datum level face 3 supplemented by its three-dimensional lap in a clockwise direction
With secondary surface face 4, upper side is secondary surface face 5, and downside is secondary surface face 6；
(3.3) it will store in the same manner in addition to storing to the data block after n/2 block coding remaining on datum level face 1 to another
On one datum level face 3；
(3.4) from the 2 of datum level face 1k-1T is randomly selected in block data block1=2k-2Block data block, remaining T2=2k-2Block number evidence
Block, from the 2 of datum level face 3k-1T is randomly selected in block data block3=2k-2Block data block, remaining T4=2k-2Block data block；T1
And T3Composition 2k-1Block data block is stored on secondary surface face 2；T1And T4Composition 2k-1Block data block is stored on secondary surface face 4；
T2And T3Composition 2k-1Block data block is stored on secondary surface face 5；T2And T4The 2 of compositionk-1Block data block is stored in secondary surface face
On 6；
The step (5) further include:
(5.1) it sets on each face of three-dimensional stereo model and has p node respectively, whereinThen three-dimensional stereo model one
Contain 6p node altogether, wherein each face includes a non-referenced nodes of s datum node and p-s, wherein I.e. three-dimensional stereo model includes 6s datum node and a non-referenced node of 6* (p-s)；
(5.2) when nodal test module D1 detects a node failure of certain one side in three-dimensional stereo model, by common
Other nodes on face are restored；When the failure of i-th of datum node in some face, by the data on remaining p-1 node
Linear combination operation is carried out, fail data is recovered；
It is vertical by three-dimensional when nodal test module D1 detects that the failure node in three-dimensional stereo model in certain one side is more than one
Body distributed structure method learns that the three-dimensional stereo model shares 6 faces, wherein be respectively face 1 and face 3 there are two datum level, four
Secondary surface is respectively face 2, face 4, face 5 and face 6；It obtains secondary surface face 2 and secondary surface face 4 includes the data block in datum level face 1
T1, secondary surface face 5 and secondary surface face 6 include the data block T in datum level face 12, secondary surface face 2 and secondary surface face 5 include benchmark
Data block T in face face 33, secondary surface face 4 and face secondary surface 6 include the data block T in datum level face 34；By in other faces with
The linear combination operation of data, recovers the data in failure node on the relevant node of failure node；Work as three-dimensional stereo model
When upper datum node failure number is x=6s, this method loses recovery meaning；When datum node failure on three-dimensional stereo model
Number is x=6s-1, and when non-referenced node does not fail, the data of datum node all are resumed out；Work as three-dimensional stereo model
Upper datum node failure number is x < 6s-1, and when non-referenced node relevant to the datum node of failure does not fail, benchmark
The data of node all are resumed out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610168643.6A CN105959338B (en) | 2016-03-23 | 2016-03-23 | A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610168643.6A CN105959338B (en) | 2016-03-23 | 2016-03-23 | A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105959338A CN105959338A (en) | 2016-09-21 |
| CN105959338B true CN105959338B (en) | 2019-04-19 |
Family
ID=56917327
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610168643.6A Active CN105959338B (en) | 2016-03-23 | 2016-03-23 | A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105959338B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106685592B (en) * | 2016-12-26 | 2019-06-11 | 华中科技大学 | A kind of network transfer method encoded using three-dimensional space network |
| CN110401703B (en) * | 2019-07-10 | 2021-10-15 | 东华大学 | Cloud storage data recovery method based on multi-level network coding |
| CN111414345B (en) * | 2020-03-30 | 2022-03-25 | 杭州华望系统科技有限公司 | Model data recovery method based on operation log |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102938778A (en) * | 2012-10-19 | 2013-02-20 | 浪潮电子信息产业股份有限公司 | Method for realizing multi-node disaster tolerance in cloud storage |
| CN103533055A (en) * | 2013-10-16 | 2014-01-22 | 南京大学镇江高新技术研究院 | Low-redundancy cloud storage system and realization method thereof based on network encoding |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9164856B2 (en) * | 2013-11-11 | 2015-10-20 | International Business Machines Corporation | Persistent messaging mechanism |
-
2016
- 2016-03-23 CN CN201610168643.6A patent/CN105959338B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102938778A (en) * | 2012-10-19 | 2013-02-20 | 浪潮电子信息产业股份有限公司 | Method for realizing multi-node disaster tolerance in cloud storage |
| CN103533055A (en) * | 2013-10-16 | 2014-01-22 | 南京大学镇江高新技术研究院 | Low-redundancy cloud storage system and realization method thereof based on network encoding |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105959338A (en) | 2016-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105393225B (en) | Erasure code across multiple areas | |
| CN105993043B (en) | Safety device, its method and computer-readable recording medium | |
| CN106844098B (en) | Rapid data recovery method and system based on cross erasure coding | |
| CN105959338B (en) | A kind of cloud storage multinode data reconstruction method of three-dimensional stereo model formula | |
| US20180025670A1 (en) | Inconsistency detecting method, inconsistency detecting system, inconsistency detecting device, and program | |
| CN101834899A (en) | Distributed adaptive coding and storing method | |
| CN102170460A (en) | Cluster storage system and data storage method thereof | |
| US20150178170A1 (en) | Method and Apparatus for Recovering Data | |
| WO2005122630A3 (en) | Arrangement in a network node for secure storage and retrieval of encoded data distributed among multiple network nodes | |
| CN104935481B (en) | Data reconstruction method based on redundancy scheme under a kind of distributed storage | |
| CN110212923A (en) | A kind of distributed correcting and eleting codes memory system data restorative procedure based on simulated annealing | |
| CN102761596A (en) | Method for uploading a file in an on-line storage system and corresponding on-line storage system | |
| CN106230982A (en) | A kind of dynamic self-adapting secure cloud storage method considering node reliability | |
| CN102752402A (en) | Cloud storage method and cloud storage system | |
| WO2020035088A3 (en) | Prioritizing shared blockchain data storage | |
| CN103761171B (en) | A kind of low bandwidth data reconstructing method for binary coding redundant storage system | |
| CN105159603A (en) | Repair method for distributed data storage system | |
| CN102769673A (en) | Failure detection method suitable to large-scale storage cluster | |
| CN109714325A (en) | A kind of one-way optical gate data transmission method, system, electronic equipment and medium | |
| CN103561033B (en) | User remotely accesses the device and method of HDFS cluster | |
| CN103744620B (en) | A kind of method and apparatus for data storage | |
| CN103106124A (en) | Intersection reconstruction method based on erasure code cluster memory system | |
| CN109194444A (en) | A kind of balanced binary tree restorative procedure based on network topology | |
| CN104486438A (en) | Disaster-tolerant method and disaster-tolerant device of distributed storage system | |
| CN107153661A (en) | A kind of storage, read method and its device of the data based on HDFS systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |