CN106100801A - A kind of non-homogeneous erasure code method of cloud storage system - Google Patents

A kind of non-homogeneous erasure code method of cloud storage system Download PDF

Info

Publication number
CN106100801A
CN106100801A CN201610743775.7A CN201610743775A CN106100801A CN 106100801 A CN106100801 A CN 106100801A CN 201610743775 A CN201610743775 A CN 201610743775A CN 106100801 A CN106100801 A CN 106100801A
Authority
CN
China
Prior art keywords
data block
memory node
data
cold data
cold
Prior art date
Application number
CN201610743775.7A
Other languages
Chinese (zh)
Other versions
CN106100801B (en
Inventor
胡玉鹏
邓俊杰
伍麟珺
温冠超
皮玲
Original Assignee
湖南大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 湖南大学 filed Critical 湖南大学
Priority to CN201610743775.7A priority Critical patent/CN106100801B/en
Publication of CN106100801A publication Critical patent/CN106100801A/en
Application granted granted Critical
Publication of CN106100801B publication Critical patent/CN106100801B/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network-specific arrangements or communication protocols supporting networked applications
    • H04L67/10Network-specific arrangements or communication protocols supporting networked applications in which an application is distributed across nodes in the network
    • H04L67/1097Network-specific arrangements or communication protocols supporting networked applications in which an application is distributed across nodes in the network for distributed storage of data in a network, e.g. network file system [NFS], transport mechanisms for storage area networks [SAN] or network attached storage [NAS]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00Arrangements for detecting or preventing errors in the information received
    • H04L1/004Arrangements for detecting or preventing errors in the information received by using forward error control
    • H04L1/0056Systems characterized by the type of code used
    • H04L1/0061Error detection codes

Abstract

The invention discloses a kind of non-homogeneous erasure code method of cloud storage system; crash rate according to physical store node; the data block being stored on them is divided into the logic groups varied in size; and provide entangling of different stage to delete protection for different packets; and then give more to protect to the data on high failure rate memory node, allow their reparation expense lower simultaneously.So, the present invention can improve the overall reliability of system by strengthening the protection of data on high failure rate memory node, can reduce system further by the reparation expense of data on reduction high failure rate memory node and totally repair expense.

Description

A kind of non-homogeneous erasure code method of cloud storage system

Technical field

The present invention relates to a kind of non-homogeneous erasure code method of cloud storage system.

Background technology

In recent years, the cloud storage system development with large-scale data center as representative is swift and violent (such as Windows Azure Storage、Amazon S3、Google GFS], EMC ATMOS etc.), the data volume of accumulation from PB to EB even to ZB so that Cloud storage network size and number of nodes are explosive growth, but the crash rate of data is more and more higher (such as, One MapReduce operation of the often execution at Google cloud storage center averagely has about 5 node failures, about 4000 nodes Hadoop system within average 6 hours, just have a disk failure, and its crash rate increases with system scale and increases rapidly). Memory node (referring to store server or disk), after inefficacy, causes stored data inaccessible, especially critical data Lose, massive losses will be brought to user.It is therefore desirable to use data entangle delete or calamity for technology to ensure the reliability of data, After a part of data deletion (i.e. losing), it is possible to carry out entangling deleting operation, still can guarantee that storage system worked well.Many copies Technology (typically 3 copy) is the default storage standard of numerous cloud storage system, be conventional data calamity for technology, in original number After losing, it is replaced with residue copy.The cloud storage systems such as Google GFS, Hadoop HDFS and CFS all use many Replication technology, its elementary tactics is: when file object writes, and the data block (such as 64M) being split as fixed size is carried out Distributed storage, and each data block is multiple copy at copy-on-write, uses chain-like manner to be pushed to cloud storage net successively Different memory nodes in network, when some memory node lost efficacy, can carry out redundant copy replicating operation, recover data Block.As a whole, the major advantage of n Replication technology is embodied in: one is to support concurrent read operation well, in digital independent rate There is time high performance advantage;Two is that data reconstruction/recovery expense is little, loses a data block, as long as replicating a data block ?.But, its shortcoming is the most more apparent: the storage overhead of data trnascription is the hugest, and its storage overhead is the n of normal condition Times, especially when system is held deletion Capability Requirement height, this is particularly problematic.It is based on finite field operations that (n k) entangles and deletes N data block can be encoded to (n+k) individual correcting and eleting codes piecemeal (having more k redundancy check data block) by code (Erasure Codes), As long as obtaining wherein arbitrarily n correcting and eleting codes piecemeal just can recover all n original data blocks, can at most hold deletion k according to block. Representational Reed-Solomon encodes, i.e. (n, k) correcting and eleting codes is a kind of systematicness coding to RS, i.e. former n data block after coding Keeping constant, remaining k redundancy check data block is that original data block calculating gets.Compared with multi-duplicate technology, fault-tolerant ability Under the same terms, the storage efficiency of correcting and eleting codes technology is higher, cost is lower;Such as, if according to 100PB data (Windows About 2012 scales of Azure Storage), $ 1/GB/, calculate (if Replication technology and correcting and eleting codes are each by fault-tolerant ability 1 Need 2 times and 1.5 times of memory spaces), the most only storage overhead correcting and eleting codes the most just can save $ 52,428,800.

Therefore, in order to ensure the storage overhead reducing system while data high reliability as much as possible and run into This, the typical technology that each main flow cloud storage service supplier uses is that the data block after gradually cooling down is encoded to correcting and eleting codes and deletes Except unnecessary copy, i.e. use many copies redundancy (such as 3 copy mechanism of standard) to ensure concurrently to read in early days in the storage of dsc data block Operating characteristics, is converted into correcting and eleting codes when then using correcting and eleting codes technology after data block gradually cools down (when user's rate of people logging in is low) Piecemeal stores.Such as, the Windows Azure Storage of Microsoft uses LRC correcting and eleting codes to process cold data block, Google GFS II (colossus) system uses RS (6,3) correcting and eleting codes to process cold data block, and the f4 cloud storage system of Facebook is then adopted RS (10,6) or RS is then used by the ATMOS system of Xorbas and Hitchhiker correcting and eleting codes, DiskReduce system and EMC (9,3) correcting and eleting codes such as.But, the shortcoming of correcting and eleting codes technology is also apparent from: (n, k) defect that correcting and eleting codes is maximum is to need to obtain extremely Few n data block could recover initial data;When having only to recover a data block, it is still desirable to obtain n data block, its System I/O and the network bandwidth cost when data are recovered are the highest, as long as far above the expense of in multi-duplicate technology data block. Obviously, its data will be caused to recover/reconstruct efficiency under large-scale cloud storage environment low, network overhead is big, it tends to be difficult to adapt to Data-intensive process demand under cloud storage environment.At present correcting and eleting codes technology mainly research how in memory space expense and Data are repaired and are carried out balance process between traffic overhead, the problem big to solve correcting and eleting codes technical data reparation traffic overhead, closely Create new technologies and methods over Nian, such as regeneration code, locally reconstruct a yard LRC (Local Reconstruction Codes).

But, the correcting and eleting codes technology at current cloud storage center all follows the coded method of traditional communications field, does not has pin The feature of data failure mode is designed.In recent years, having had many research to show, the crash rate of each memory node is Inconsistent, affected (e.g., the tank temperature of memory node place rack, disk operating temperature, magnetic by various physical parameters The dish work age etc.), present the feature of a kind of non-uniform Distribution.Such as, Microsoft passes through substantial amounts of experiment statistics, draws Result of study show, the annual crash rate of different disk with average operating temperature change and have fluctuation greatly, annual Crash rate reaches as high as 12%, is then about 3% time minimum;And store the annual crash rate of server and the water of place rack Bath temperature degree has bigger association, fluctuates between 2%~14%, it is seen that usual maximum failure rate is 6-7 times of minimum crash rate.By Different in the crash rate of each storage device, by different for the crash rate causing its data stored, present a kind of non-homogeneous point Cloth.

Summary of the invention

The technical problem to be solved is, not enough for prior art, it is provided that a kind of cloud storage system non-all Even erasure code method.

For solving above-mentioned technical problem, the technical solution adopted in the present invention is: the non-homogeneous of a kind of cloud storage system is entangled Delete coded method, comprise the following steps:

1) find out cold data block to be encoded, and described cold data block is grouped, volatile data block is put into little point Group, the data block being difficult to lose efficacy puts into big packet;

2) structure non-uniform encoding generator matrix, and utilize code generator matrix that cold data block is encoded;

3) data block after coding and redundancy check block are disposed.

Step 1) the process that implements include:

1) use cold data judgment method find out need cold data block set C to be processed, in batches to cold data block at Reason, processes a subset every timeSC={D0,D1,...,Dz},DzRepresent cold data block;Z states the individual of cold data block Number;

2) data block comprising subset SC carries out Random Maps respectively, i.e. uses random Harsh function, by certain cold number According to being mapped on certain memory node, represent that this cold data block to move to this memory node, record storage after coded treatment The ID value of node;

3) utilize the crash rate of current acquired memory node that cold data are ranked up, high failure rate will be stored in On node, the easy inefficacy data block (if annual crash rate is higher than 10%) comes foremost, is stored in relatively low crash rate (such as year Mean failure rate is less than 10%) being difficult to after fail data block row on node;

4) by foremost k0Individual data block, i.e.Put into first group, by (k subsequently0+ 1) individual data block is put into Second group, then by (k subsequently0+ 2) individual data block puts into the 3rd group, by that analogy, until last (k0+ l-1) individual data Block puts into the l packet, and

Structure non-uniform encoding generator matrix, and utilize code generator matrix to implementing that cold data block encodes Process includes: build code generator matrix G, has local verification data block p by G coding relief (i+1) individual packeti, i =0,1 ..., l-1, additionally, r overall situation verification data block P is also shared in all packetsj, j=0,1 ..., r-1, piBy each group Interior cold data block XOR generates, PjBeing multiplied by after code coefficient XOR again by all of cold data block to generate, G meets following formula:

G · D 0 D 1 ... D | S C | - 1 = p 0 p 1 ... p l - 1 P 0 P 1 ... P r - 1 ;

It is expressed as with system of linear equations:

D 0 + D 1 + ... + D k 0 - 1 = p 0 D k 0 + D k 0 + 1 + ... + D 2 k 0 = p 1 ...... D | S C | - k 0 - l + 1 + ... + D | S C | - 1 = p l - 1 a 0 D 0 + a 1 D 1 + ... a k 0 - 1 D k 0 - 1 + b 0 D k 0 + ... + b k 0 D 2 k 0 + c 0 ... = P 0 a 0 2 D 0 + a 1 2 D 1 + ... a k 0 - 1 2 D k 0 - 1 + b 0 2 D k 0 + ... + b k 0 2 D 2 k 0 + c 0 2 ... = P 1 ...... a 0 r - 2 D 0 + a 1 r - 2 D 1 + ... a k 0 - 1 r - 2 D k 0 - 1 + b 0 r - 2 D k 0 + ... + b k 0 r - 2 D 2 k 0 + c 0 r - 2 ... P r - 1 .

The process that implements disposing the data block after coding and redundancy check block includes: utilize G to generate redundancy Verification data block, the cold data block set after being encoded They are deployed to corresponding memory node, and all cold data blocks are according to the memory node recorded ID value is disposed, and verification data block is disposed by Random Maps method.

Compared with prior art, the had the beneficial effect that present invention of the present invention is by strengthening high failure rate storage joint On point, the protection of data can improve the overall reliability of system, by reducing the reparation expense of data on high failure rate memory node System can be reduced further and totally repair expense, data can be effectively reduced on the premise of keeping identical appearance deletion ability Reconstruct expense, including magnetic disc i/o expense and network traffics expense, thus effectively promotes the performance of storage system.

Accompanying drawing explanation

Fig. 1 shows the schematic diagram of UFP-LRC (2,2,2) encoding scheme.

Fig. 2 shows that the average expense of recovering under theoretical numerical analysis contrasts.

Fig. 3 shows that the average expense of recovering under different storage overhead contrasts.

Fig. 4 shows that availability contrasts.

Fig. 5 shows that the average expense of recovering in real system experiment contrasts.

Fig. 6 shows that data recover mortality contrast.

Detailed description of the invention

Coded method UFP-LRC (the Unequal Failure Protection based Local of present invention design Reconstruction Codes) it is divided into following step: the first step is to find out the cold data block to be encoded, and to them Being grouped, volatile data block is put into smaller packet, the data block being difficult to lose efficacy puts into bigger packet;Second step is that structure is non- Uniform encoding generator matrix, and utilize code generator matrix that cold data block is encoded;3rd step is to the data after coding Block and redundancy check block are disposed.

For encoding scheme UFP-LRC (k0, l, r) for, specific coding implementation process is as follows:

The first step: use general cold data judgment method to find out and need cold data block set C to be processed, in batches to cold number Process according to block, process a subset every timeSC={D0,D1,...,Dz},DzRepresent cold data block.

Second step: the data block comprising subset SC carries out Random Maps respectively, i.e. uses general random Harsh function, Certain cold data is mapped on certain memory node and (generally can ensure that different data blocks is mapped to different storage joints Point, it is to avoid relevant failure), represent that this cold data block to move to this memory node after coded treatment, record memory node ID value.

3rd step: utilizing the crash rate of current acquired memory node to be ranked up cold data, those will store Easy inefficacy data block on high failure rate node comes foremost, and be stored on relatively low crash rate node is difficult to fail data After block row.

4th step: data block is divided into l packet.Will foremost k0Individual data block is (i.e.) put into first group, By (k subsequently0+ 1) individual data block is (i.e.) put into second group, then by (k subsequently0+ 2) individual data block puts into Three groups, by that analogy, until last (k0+ l-1) individual data block puts into the l packet.And

5th step: build check code generator matrix G, i.e.Coefficient matrix.(i+1) individual packet can be allowed by G after being encoded There is local verification data block pi(i=0,1 ..., l-1), additionally, r overall situation verification data block P is also shared in all packetsj (j=0,1 ..., r-1).piGenerated by the data block XOR in each group, PjAnd after being multiplied by code coefficient by all of data block XOR generates again.Under G is satisfied facial (1), computing is all carried out on finite field.

G · D 0 D 1 ... D | S C | - 1 = p 0 p 1 ... p l - 1 P 0 P 1 ... P r - 1 - - - ( 1 )

It is expressed as with system of linear equations:

D 0 + D 1 + ... + D k 0 - 1 = p 0 D k 0 + D k 0 + 1 + ... + D 2 k 0 = p 1 ...... D | S C | - k 0 - l + 1 + ... + D | S C | - 1 = p l - 1 a 0 D 0 + a 1 D 1 + ... a k 0 - 1 D k 0 - 1 + b 0 D k 0 + ... + b k 0 D 2 k 0 + c 0 ... = P 0 a 0 2 D 0 + a 1 2 D 1 + ... a k 0 - 1 2 D k 0 - 1 + b 0 2 D k 0 + ... + b k 0 2 D 2 k 0 + c 0 2 ... = P 1 ...... a 0 r - 2 D 0 + a 1 r - 2 D 1 + ... a k 0 - 1 r - 2 D k 0 - 1 + b 0 r - 2 D k 0 + ... + b k 0 r - 2 D 2 k 0 + c 0 r - 2 ... P r - 1 - - - ( 2 )

Wherein, a, b, c ..., it is all to need the code coefficient on the finite field of structure, what all of parameter a was corresponding is the The code coefficient of one group of data, what b was corresponding is the code coefficient of second group of data, and c is the code coefficient of the 3rd group, with this type of Push away.These code coefficients are all once to set, when next round data block subset processes, the most reusable.

These code coefficients need to meet following formula (3):

ax,ay,bs,bt,cv,cw...≠0

ax≠ay,bs≠bt,cv≠cw,...

ax,ay≠bs,bt≠cv,cw≠...

ax+ay≠bs+bt≠cv+cw≠...

ax≠bs+bt,bs≠ax+ay,cv≠ax+ay,... (3)

Formula (3) represents: all code coefficient a, b, c ..., it is not the most 0;Arbitrarily two coding systems in group and between group Number is the most unequal;Two coefficient sums that arbitrarily two coefficient sums in group are all not equal in another one group;Any one Any two coefficient sum that coefficient is all not equal in other group (notes: the XOR fortune being shown with in confinement with operation table here Calculate).

The code coefficient of formula (3), UFP-LRC (k is met by selection0, l, r) will have good appearance deletion ability:

(1) the UFP-LRC coding constructed has maximum restorability.As long as i.e. can decode on Theory of Linear Equations, Data recovery can be carried out.

(2) have non-homogeneous erasure code protection, i.e. before (r-k0+ 1) individual packet can tolerate that deletion arbitrarily (r+2) is individual original Data block;And packet below does not has this characteristic.

(3) block (including original data block and verification data block) of deletion arbitrarily (r+1) individual deletion can be tolerated.、

(4) deletion (l+r) individual block (including original data block and verification data block) is at most tolerated.

6th step: according to formula (1), utilizes G to generate the verification data block of redundancy.Finally obtain the data block set after codingThey are deployed to corresponding memory node, all Data block D is disposed according to the memory node ID value recorded, verification data block by Random Maps method or other conventional reflect Shooting method is disposed.

7th step: last, when some memory node lost efficacy, if when there being m data block, during n verification dropout of data block, Verification data block p that will lose efficacyiOr PjCorresponding equation is deleted from formula (2), after deleting n corresponding equation, then from remaining Linear equation is found out any m equation and builds a new equation group (as long as equation group has in theory determines unique solution, i.e. Corresponding linear matrix is reversible), solve equation group and can draw all | SC | individual original data block D0,D1,...,D|SC|-1, complete Become data reparation.

Be described in detail as a example by UFP-LRC (2,2,2) concrete coding and repair process below.

The first step: set the general cold data block determination methods of employing and (e.g., according to accessing the time recently, the most do not visit The data markers asked is cold data), find out 5 cold data block subsets SC={D that coding is presently required0,D1,...,D4}。

Second step: the data block comprising subset SC carries out Random Maps respectively, allows different data blocks be mapped to difference Memory node or rack, the ID value of record object memory node.

3rd step: utilize the crash rate (crash rate can be estimated by relevant physical parameter) of target storage node to cold data Block is ranked up, if after Pai Xu being { D0,D1,...,D4};D0For the data block that the most easily lost efficacy, D4For the data block that the most easily lost efficacy.

4th step: data block is divided into 2 packets.Will 2, foremost data block (i.e. D0~D1) put into first group, By 3 data block (i.e. D subsequently2~D4) put into second group.

5th step: build check code generator matrix G.Build G it is crucial that effectively meet formula (3) at finite field spatial choice Code coefficient a, b.Concrete system of selection is to use a kind of staggered mode: allowing the high position of code coefficient of first group is 0, Low level is 1, and a high position for second group of code coefficient is 1, and low level is 0.Therefore, at finite field gf (25) space selects coefficient: a0 =00011, a1=00010, b0=11000, b1=10000, b2=01000.The most often wheel data block coding with these is all Number.

Then verification data block generator matrix is:

G = 1 1 0 0 0 0 0 1 1 1 a 0 a 1 b 0 b 1 b 2 a 0 2 a 1 2 b 0 2 b 1 2 b 2 2 - - - ( 4 )

The equation group then generating verification data block is:

D 0 + D 1 = p 0 D 2 + D 3 + D 4 = p 1 a 0 D 0 + a 1 D 1 + b 0 D 2 + b 1 D 3 + b 2 D 4 = P 0 a 0 2 D 0 + a 1 2 D 1 + b 0 2 D 2 + b 1 2 D 3 + b 2 2 D 4 = P 1 - - - ( 5 )

Fig. 1 shows the packet schematic diagram after the coding of UFP-LRC (2,2,2).

6th step: according to formula (1), utilizes G to generate the verification data block of redundancy.During specific coding, it is to use endless form Choose the fritter byte to each data block every time and carry out coded treatment (as taken 16 bits from each data block D), with Ensure coding rate, then encoding operation is at GF (216Carry out on);Finally obtain the data block set after codingThey are deployed to corresponding memory node, and all data blocks D are according to The memory node ID value of record is disposed, and verification data block carries out portion by Random Maps method or other conventional mapping method Administration.

7th step: when some memory node lost efficacy, if data block D0,D1,D2All lose, verify data block p0During loss, Verification data block p that will lose efficacy0Corresponding equation is deleted, more still can find out 3 equation structures from remaining linear equation Build a new equation group, D3,D4As constant, the equation group solving down facial (6) can recover D0,D1,D2

D 2 = p 1 - D 3 - D 4 a 0 D 0 + a 1 D 1 + b 0 D 2 = P 0 - b 1 D 3 - b 2 D 4 a 0 2 D 0 + a 1 2 D 1 + b 0 2 D 2 = P 1 - b 1 2 D 3 - b 2 2 D 4 - - - ( 6 )

I.e.

Shown in the coefficient matrix G' such as formula (7) of its correspondence.Owing to effectively have selected code coefficient, it is ensured that coefficient square The vector of battle array is linear independence, i.e. | G'|=a0a1(a1-a0) ≠ 0, therefore formula (7) has inverse matrix, formula (6) has and uniquely determines Solve.

G ′ = 0 0 1 a 0 a 1 b 0 a 0 2 a 1 2 b 0 2 - - - ( 7 )

I.e. as shown in formula (8), can be with extensive restored data D by finding the inverse matrix0,D1,D2.Loss of data in the case of other Similar method is used to recover.

D 0 D 1 D 2 = 0 0 1 a 0 a 1 b 0 a 0 2 a 1 2 b 0 2 - 1 . p 1 - D 3 - D 4 P 0 - b 1 D 3 - b 2 D 4 P 1 - b 1 2 D 3 - b 2 2 D 4 - - - ( 8 )

Relative to existing RS (Reed-Solomon), LRC coding techniques, present invention improves over the packet scheme of data, Realize packet size uneven, and the data block easily lost efficacy is placed on less group, owing to using local verification data block simultaneously With the overall situation verification data block data are protected, will bring both sides advantage:

(1) so that the data block easily lost efficacy is when losing, consume a small amount of expense of recovering and can recover data, thus always The recovery expense recovering fail data is reduced on body.

(2) so that easily the appearance of fail data to delete ability more preferable, can resist more data failure pattern, thus always Improve data on body and revert to power and the reliability of storage system.

UFP-LRC is than classical RS coding, and (existing commercialization is at windows at top-level meeting 2012 USENIX ATC In server) the LRC method delivered above, will have more preferable restorability and availability.Hereinafter use theoretical numerical analysis Advantages of the present invention is proved with experimental data.

First, in terms of theoretical numerical analysis, mainly two performance indications: (1) averagely recovers expense: refer to recover The par of the required download/acquisition data block of the data block lost (refers to network traffics expense, computation delay Relative Network Time delay almost can be ignored);(2) availability: when there being some memory nodes to lose efficacy, individual data block can become work recovery/acquisition Probability.

First UFP-LRC and LRC coded method under identical storage overhead is contrasted by Fig. 2.The crash rate ratio of X-axis Represent the ratio of high failure rate memory node and the mean failure rate of low crash rate memory node.UFP-LRC (2,3,2) and LRC (9, 3,2) there is identical data block, verification data block and storage overhead.UFP-LRC (1,3,2) and LRC (6,3,2) has identical Data block, verification data block and storage overhead.But be as crash rate ratio is gradually increased that (time such as more than 2, Microsoft grinds Study carefully display being commonly stored node maximum failure rate is 6-7 times of minimum crash rate), i.e. inefficacy rate variance between each memory node Time the biggest, UFP-LRC substantially has less recovery expense than LRC.

Fig. 3 shows the contrast under many kinds of parameters combined situation.When crash rate ratio ρ becomes big, UFP-LRC is at each seed ginseng Recovery expense in the case of number is all low than LRC (during the most identical storage overhead, having lower recovery expense).

Fig. 4 shows that availability contrasts situation.X-axis represents the memory node ratio of inefficacy at present.UFP-LRC than RS can Much higher by property, and under the same terms and the availability of LRC is essentially the same, gap only has 4.00E-08, substantially can neglect Slightly disregard.

Then, in terms of experiment, by building the true distributed memory system of 30 memory node compositions, deposit identical The performance of UFP-LRC (2,3,2), LRC (9,3,2) and RS (6,3) in the case of storage expense has carried out analyzing contrast.It is primarily upon Two performance parameters: (1) averagely recovers expense.(2) mortality is recovered: i.e. data are recovered/decoded the frequency of failure and repair divided by total Again count.

Fig. 5 shows and averagely recovers expense, belongs to the node ratio of high failure rate in X-axis table non ageing memory node (i.e. crash rate is higher than 0.2), hence it is evident that UFP-LRC and LRC is better than the restorability of classical RS coding, the recovery expense base of RS This is fixing.And the UFP-LRC low 10-12% of recovery expense than LRC generally.Because more failure conditions occurs easily losing efficacy In packet, the recovery expense of the data of these packets is less, so it is less to recover expense generally.

Fig. 6 shows recovery mortality.Owing to RS (6,3) can only tolerate that 3 data blocks lost efficacy, its mortality is higher than it Its two coded method.UFP-LRC and LRC can tolerate that 3-5 data block lost efficacy.Along with the memory node number lost efficacy in system It is continuously increased, mortality LRC to be less than (9,3,2) of UFP-LRC (2,3,2), this is because more inefficacy can occur In easy fail data place is grouped, but the protection that the data in these packets obtain is more preferable, holds deletion ability higher, so Better reliability generally.

Claims (4)

1. the non-homogeneous erasure code method of a cloud storage system, it is characterised in that comprise the following steps:
1) find out cold data block to be encoded, and described cold data block is grouped, volatile data block is put into little packet, no The data block easily lost efficacy puts into big packet;
2) structure non-uniform encoding generator matrix, and utilize code generator matrix that cold data block is encoded;
3) data block after coding and redundancy check block are disposed.
The non-homogeneous erasure code method of cloud storage system the most according to claim 1, it is characterised in that step 1) tool Body realizes process and includes:
1) use cold data judgment method to find out and need cold data block set C to be processed, cold data block is processed, often in batches One subset of secondary processSC={D0,D1,...,Dz},DzRepresent cold data block;Z states the number of cold data block;
2) data block comprising subset SC carries out Random Maps respectively, i.e. uses random Harsh function, certain cold data is reflected It is mapped on certain memory node, represents that this cold data block to move to this memory node after coded treatment, record memory node ID value;
3) utilize the crash rate of current acquired memory node that cold data are ranked up, high failure rate node will be stored in On easy inefficacy data block come foremost, be stored on relatively low crash rate node be difficult to fail data block row after;
4) by foremost k0Individual data block, i.e.Put into first group, by (k subsequently0+ 1) individual data block puts into second Group, then by (k subsequently0+ 2) individual data block puts into the 3rd group, by that analogy, until last (k0+ l-1) individual data block puts Enter the l packet, and
The non-homogeneous erasure code method of cloud storage system the most according to claim 2, it is characterised in that construct non-homogeneous Code generator matrix, and the process that implements utilizing code generator matrix to encode cold data block includes: build coding Generator matrix G, has local verification data block p by G coding relief (i+1) individual packeti, i=0,1 ..., l-1, this Outward, r overall situation verification data block P is also shared in all packetsj, j=0,1 ..., r-1, piBy the cold data block XOR in each group Generate, PjBeing multiplied by after code coefficient XOR again by all of cold data block to generate, G meets following formula:
G · D 0 D 1 ... D | S C | - 1 = p 0 p 1 ... p l - 1 P 0 P 1 ... P r - 1 ;
It is expressed as with system of linear equations:
D 0 + D 1 + ... + D k 0 - 1 = p 0 D k 0 + D k 0 + 1 + ... + D 2 k 0 = p 1 ...... D | S C | - k 0 - l + 1 + ... + D | S C | - 1 = p l - 1 a 0 D 0 + a 1 D 1 + ... a k 0 - 1 D k 0 - 1 + b 0 D k 0 + ... + b k 0 D 2 k 0 + c 0 ... = P 0 a 0 2 D 0 + a 1 2 D 1 + ... a k 0 - 1 2 D k 0 - 1 + b 0 2 D k 0 + ... + b k 0 2 D 2 k 0 + c 0 2 ... = P 1 ...... a 0 r - 2 D 0 + a 1 r - 2 D 1 + ... a k 0 - 1 r - 2 D k 0 - 1 + b 0 r - 2 D k 0 + ... + b k 0 r - 2 D 2 k 0 + c 0 r - 2 ... = P r - 1 .
The non-homogeneous erasure code method of cloud storage system the most according to claim 3, it is characterised in that after coding The process that implements that data block and redundancy check block carry out disposing includes: utilizes G to generate the verification data block of redundancy, is compiled Cold data block set after Ma They are deployed to phase The memory node answered, all cold data blocks are disposed according to the memory node ID value recorded, and verify data block by reflecting at random Shooting method is disposed.
CN201610743775.7A 2016-08-29 2016-08-29 A kind of non-homogeneous erasure code method of cloud storage system CN106100801B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610743775.7A CN106100801B (en) 2016-08-29 2016-08-29 A kind of non-homogeneous erasure code method of cloud storage system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610743775.7A CN106100801B (en) 2016-08-29 2016-08-29 A kind of non-homogeneous erasure code method of cloud storage system

Publications (2)

Publication Number Publication Date
CN106100801A true CN106100801A (en) 2016-11-09
CN106100801B CN106100801B (en) 2019-04-19

Family

ID=57223812

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610743775.7A CN106100801B (en) 2016-08-29 2016-08-29 A kind of non-homogeneous erasure code method of cloud storage system

Country Status (1)

Country Link
CN (1) CN106100801B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106383664A (en) * 2016-08-31 2017-02-08 北京小米移动软件有限公司 Data storage method and apparatus
CN106648471A (en) * 2016-12-29 2017-05-10 深圳市中博科创信息技术有限公司 Cloud platform data storage method and device
CN106788455A (en) * 2016-11-29 2017-05-31 陕西尚品信息科技有限公司 A kind of building method of the optimal partial repairable system code based on bag
CN106788891A (en) * 2016-12-16 2017-05-31 陕西尚品信息科技有限公司 A kind of optimal partial suitable for distributed storage repairs code constructing method
CN107656832A (en) * 2017-09-18 2018-02-02 华中科技大学 A kind of correcting and eleting codes method of low data reconstruction expense

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070133691A1 (en) * 2005-11-29 2007-06-14 Docomo Communications Laboratories Usa, Inc. Method and apparatus for layered rateless coding
CN103944678A (en) * 2014-04-21 2014-07-23 浙江大学 Fountain-code encoding method with feedback and unequal error protection capacity

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070133691A1 (en) * 2005-11-29 2007-06-14 Docomo Communications Laboratories Usa, Inc. Method and apparatus for layered rateless coding
CN103944678A (en) * 2014-04-21 2014-07-23 浙江大学 Fountain-code encoding method with feedback and unequal error protection capacity

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
XIAODAN SONG,ET AL.,: ""UNEQUAL ERROR PROTECTION FOR SCALABLE VIDEO STORAGE IN THE CLOUD"", 《2015 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO(ICME)》 *
程振东等,: ""云文件系统中纠删码技术的研究与实现"", 《计算机科学与探索》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106383664A (en) * 2016-08-31 2017-02-08 北京小米移动软件有限公司 Data storage method and apparatus
CN106383664B (en) * 2016-08-31 2019-12-03 北京小米移动软件有限公司 Date storage method and device
CN106788455A (en) * 2016-11-29 2017-05-31 陕西尚品信息科技有限公司 A kind of building method of the optimal partial repairable system code based on bag
CN106788455B (en) * 2016-11-29 2019-11-22 陕西尚品信息科技有限公司 A kind of building method of the optimal partial repairable system code based on packet
CN106788891A (en) * 2016-12-16 2017-05-31 陕西尚品信息科技有限公司 A kind of optimal partial suitable for distributed storage repairs code constructing method
CN106648471A (en) * 2016-12-29 2017-05-10 深圳市中博科创信息技术有限公司 Cloud platform data storage method and device
CN107656832A (en) * 2017-09-18 2018-02-02 华中科技大学 A kind of correcting and eleting codes method of low data reconstruction expense

Also Published As

Publication number Publication date
CN106100801B (en) 2019-04-19

Similar Documents

Publication Publication Date Title
EP3014450B1 (en) Erasure coding across multiple zones
US9552258B2 (en) Method and system for storing data in raid memory devices
Sathiamoorthy et al. Xoring elephants: Novel erasure codes for big data
Cidon et al. Copysets: Reducing the frequency of data loss in cloud storage
Chen et al. Enabling data integrity protection in regenerating-coding-based cloud storage: Theory and implementation
US20180074898A1 (en) Data storage method, data recovery method, related apparatus, and system
Rashmi et al. A solution to the network challenges of data recovery in erasure-coded distributed storage systems: A study on the Facebook warehouse cluster
US8630983B2 (en) Backup of data across network of devices
Plank The raid-6 liber8tion code
Huang et al. Erasure coding in windows azure storage
CN104871138B (en) For the method and system that multidimensional RAID reconstruction and defect are avoided
US8010874B2 (en) Triple parity technique for enabling efficient recovery from triple failures in a storage array
Corbett et al. Row-diagonal parity for double disk failure correction
Goldberg et al. Towards an archival intermemory
CN103688515B (en) The coding of a kind of minimum bandwidth regeneration code and memory node restorative procedure
US7970996B1 (en) Concentrated parity technique for handling double failures and enabling storage of more than one parity block per stripe on a storage device of a storage array
US7945729B2 (en) System and method for tolerating multiple storage device failures in a storage system using horizontal and vertical parity layouts
CN106021016A (en) Virtual point in time access between snapshots
TW442751B (en) Method of encoding data signals for storage
US8171102B2 (en) Smart access to a dispersed data storage network
US8364891B2 (en) Storage assignment technique for scalable and fault tolerant storage system
US7240236B2 (en) Fixed content distributed data storage using permutation ring encoding
JP4546387B2 (en) Backup system, method and program
Papailiopoulos et al. Simple regenerating codes: Network coding for cloud storage
US7437652B2 (en) Correcting multiple block data loss in a storage array using a combination of a single diagonal parity group and multiple row parity groups

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant