CN109358987B

CN109358987B - A kind of backup cluster based on two-stage data deduplication

Info

Publication number: CN109358987B
Application number: CN201811261374.3A
Authority: CN
Inventors: 杨天明; 周书臣; 杨志强; 吴海涛; 黄平; 樊宜和; 杨奕
Original assignee: Huanghuai University
Current assignee: Suzhou elok Information Technology Co.,Ltd.
Priority date: 2018-10-26
Filing date: 2018-10-26
Publication date: 2019-09-24
Anticipated expiration: 2038-10-26
Also published as: CN109358987A

Abstract

The invention discloses a kind of backup cluster based on two-stage data deduplication, which includes multiple memory nodes, these memory nodes interconnect by storing network, cooperate the backup and recovery for completing data.The first order of two-stage data deduplication uses splits' positions technology, not only eliminates the repeated data block in entire backup cluster interior nodes between node, and the similar new data block cluster of content to identical memory node.The second level is individually completed by each memory node, is carried out Delta compression to the set of metadata of similar data block in this memory node using the Delta compress technique based on block grade data deduplication, is eliminated byte level repeated data.Present invention employs effective fingerprint queries algorithm and data bufferings, and data backup and resume performance is good, while online data being supported to migrate, and backup cluster scalability is strong.

Description

A kind of backup cluster based on two-stage data deduplication

Technical field

The invention belongs to computer storage backup technology field more particularly to a kind of backup sets based on two-stage data deduplication Group.

Background technique

With the explosive increase of data, data catastrophic failure-tolerant backup is faced with unprecedented challenge.On the one hand, traditional number A large amount of repeated datas are produced according to protection technique such as periodic backups, snapshot, continuous data protection and version file system etc., are added Fast data growth rate, force the memory capacity of standby system constantly to be expanded, enterprise are made to face huge cost pressure sum number According to management problems.On the other hand, the requirement due to application to data protection is more and more harsher, and backup window is gradually shortened, largely Data need online backup and failure instant recovery, and high requirement is proposed to system throughput and network bandwidth.In order to press down Data excessively rapid growth processed, improves resource utilization, and data deduplication technology recently becomes the research topic being concerned.

Data deduplication refers to file, data block or the byte of eliminating redundancy to guarantee only single data instance storage Process on disk, it is also referred to as a kind of capacity optimization protection technique, for reducing the capacity requirement of data protection.Data Duplicate removal mainly uses Delta compression and splits' positions technology.

Delta compression saves memory space by generating the Delta file of difference between record similar documents.It is storing File set with larger similitude especially version file collection when, Delta compression can greatly reduce memory space and occupy Rate, the even up to compression ratio of 100:1 or more.Meanwhile Delta compression algorithm itself has good runnability.But it is right Degree similar to each other is not very high file, and the effect of Delta compression is poor.Since file size distribution is wider in real system, Very likely similitude is smaller and be unsuitable for Delta compression for two files for sharing more repeated data.Therefore, it is actually depositing It is compressed in storage system using single Delta, the effect is not very satisfactory for data de-duplication.

The basic thought of splits' positions is to carry out piecemeal to data flow (or file), then elimination of duplicate data block.Simply Fixed length piecemeal can lead to the problem of bit offset, the common such as CDC(Content Defined of the Method of Partitioning based on content at present ) etc. Chucking the elongated data block that size surrounds the variation of some desired value is obtained determining data block boundary.Use encryption Hash function (such as MD5, SHA-1) calculates fingerprint of the cryptographic Hash of each data block contents as the data block, uses fingerprint Data block is indexed and comes elimination of duplicate data block by comparing fingerprint (the identical data block of fingerprint is repeated data block). It is compressed compared to Delta, splits' positions are realized simple but can not eliminate the redundancy between the similar data block of content.In addition, really A fixed optimal data block desired size is relatively difficult.Lesser data block is conducive to improve data compression rate, but when unit Interior data block to be treated is more, and the readwrite performance of system can reduce, while also increasing the metadata storages such as index and opening Pin.The block grade data deduplication storage of mainstream is typically chosen the expected data block size of 4KB or 8KB at present, this leads to granularity Repeated data less than 4KB or 8KB cannot be deleted.Studies have shown that about 50% file is less than 4KB in file system, and it is more than 80% file size is in 64KB hereinafter, can generate the repeated data that a large amount of granularities are less than 4KB or 8KB to the modification of these files. For this kind of load, it is difficult to reach ideal data compression effects using splits' positions.

Since splits' positions have many advantages, such as to realize that simple, storage management is convenient, the data deduplication of mainstream stores system at present System uses single splits' positions algorithm, and the inquiry of repeated data block is improved using Bloom filter, sparse index technology Efficiency achieves higher readwrite performance.But splits' positions only delete the repeated data of data block granularity, fail to obtain most Good data compression effects.Some systems, as DeepStore and REBL realizes a variety of compression algorithms in data storage Fusion: deleting the identical data block of content using splits' positions first, is then eliminated between set of metadata of similar data block using Delta compression The redundant data of byte level.Test shows that, compared to single splits' positions technology, the data compression effects highest of REBL improves 6.67 times.But REBL takes simple linear sweep method when searching repeated data block and set of metadata of similar data block, data are looked into Pin is changed with system scale linear increase, it is difficult to be applied to extensive high-performance data backup occasion.

In order to cope with the rapid growth of data, other than using data deduplication technology, using high-performance, expansible cluster Backup framework is also necessary.Current company-data duplicate removal technology is typically found on the basis of distributed hashtable, is supported Extensive backup cluster parallel processing has preferable performance and scalability, but all uses single splits' positions technology. Data block is distributed to corresponding memory node by the prefix of data block fingerprint by HYDRAstor and MAD2, is then eliminated each Repeated data block in memory node.Although this data distribution also eliminates the repeated data block between node, but can not carry out Set of metadata of similar data block cluster and Delta compression.Data flow is divided into superblock by DDGDA, extracts mark data from superblock content Superblock is distributed to corresponding memory node according to similar signatures prefix, in memory node by the similar signatures of similar features Elimination of duplicate data block.The technology can cluster set of metadata of similar data to identical node, but can not eliminate the repeat number between node According to block.

Summary of the invention

The object of the present invention is to provide a kind of backup cluster based on two-stage data deduplication, be conducive to improve data processing and Restorability, scalability are strong.

In order to achieve the above objectives, the technical solution adopted by the present invention is that: a kind of backup cluster based on two-stage data deduplication, Including multiple memory nodes interconnected by storing network, each memory node can the number that sends over of subscribing client According to and back up data in cluster, or restore specified data from cluster；It is equipped on the memory node based on two-stage The software systems and disk unit of data deduplication；The software systems based on two-stage data deduplication are divided into responsible first order data The upper layer of duplicate removal, that is, block grade data deduplication storage, the bottom, that is, Delta for being responsible for second level data deduplication compress storage assembly； Data block index, container index and container storage pond are installed on the disk unit；The data block index is by block grade Data deduplication system management memory and use；The container index and container storage pond by Delta compression storage assembly management and It uses；

Described piece of grade data deduplication storage includes data read-write module, fingerprint queries module and container module for reading and writing three It is slow to be additionally provided with Bloom filter, fingerprint routing table, container routing table, input block, filebuf, fingerprint for a module It rushes area and data restores buffer area；The data read-write module includes data back up method and data reconstruction method；The fingerprint Enquiry module includes fingerprint queries order, fingerprint location order, data block index upgrade order and distributed fingerprint inquiry life It enables；The container module for reading and writing includes writing container order, reading container order, read the migration order of container fingerprint order and data；

Delta compression storage assembly is provided with container access module, the container access module using similarity indexing, Similar buffer area and vessel buffers area data structure run container storage algorithm and container recovery algorithms, Delta compression storage Component receives and execute that the block grade data deduplication storage on upper layer sends over writes container order, reads container order or read to hold Device fingerprint order: to container order is write, Delta compresses storage assembly and carries out Delta compression to container, and after Delta is compressed Container write-in disk on container storage pond in；To container order is read, Delta compresses storage assembly and passes through container index from magnetic Specified container is read in container storage pond on disk, the block grade data deduplication that upper layer is returned to after the container of reading is restored is deposited Storage system；To the order of container fingerprint is read, Delta compression storage assembly reads specified containers packet from the container storage pond on disk These fingerprints are sent to the block grade data deduplication storage on upper layer by the fingerprint contained；

The first order of two-stage data deduplication is started by the memory node of execution data backup, using splits' positions technology, The similar new data block cluster of content makes do not have repeated data block in entire cluster to identical memory node；The new data Block refers to the data block being different from all data blocks existing in cluster；

The second level of two-stage data deduplication is individually completed by each memory node, using Delta compress technique, is saved to this storage Set of metadata of similar data block in point carries out Delta compression, eliminates byte level repeated data.

The data back up method in turn includes the following steps:

21) data flow: the data that subscribing client sends over, is received, input block is written into the data received；

22), piecemeal and calculating fingerprint: the dividing the data in input block based on the block algorithm of content of logical knowledge is used At data block, the keyed Hash function of logical knowledge is used to calculate the cryptographic Hash of each data block contents as the fingerprint of the data block；

23) similar signatures, are calculated: calculating the similar signatures of each data block, i.e., since the initial position of data block, It is slided in data block with the window of a fixed size, as soon as before every sliding byte, use the logical sieve guest's fingerprint algorithm known Sieve guest's fingerprint for falling into data patch in window is calculated, takes in all data patch the smallest guest sieve fingerprint as data block Similar signatures；

24) it, creates file index: file index being established to the file for including in the data of input block, by file rope Cause the client computer given and initiate data backup requests；The fingerprint for the data block that file index is included by file forms, fingerprint The sequence consensus that the sequence occurred in file index data block corresponding with its occurs hereof；

25) it, is segmented: the data in input block being segmented using the fragmentation technique based on content, i.e., in order Data, are divided into non-fixed length using these data blocks as boundary by the data block that r are 0 after lookup similar signatures in input block Data segment, each data segment include 2^rA data block, wherein r is pre-selected positive integer；

26) it, extracts hook signature: extracting hook signature for each data segment, i.e., sign and take out according to predetermined hook Sample rate is sampled all data block similar signatures for including in data segment, using the similar signatures of extraction as the hook of data segment Son signature；Select similar label of the smallest similar signatures as data segment in all data block similar signatures for including in data segment Name；

27) for each data segment, all fingerprints for including in the data segment, data segment fingerprint duplicate removal: are sent to this Fingerprint queries module on memory node, and distributed fingerprint querying command is sent to fingerprint queries module；

28), container encapsulation step: according to the return of fingerprint queries module as a result, successively handling each data segment: abandoning number According to the data block corresponding with its of the fingerprint in returning the result is not comprised in section, if still remaining data block in data segment, this A little data blocks are new data block, distribute a container for the data segment to store new data block；Take data segment similar signatures and Hook signature is respectively the similar signatures and hook signature of container；Similar signatures, the hook of container are signed, the finger of new data block Line, new data block similar signatures write-in container meta-data region, new data block write-in container data field；After processing Data segment deleted from input block；The container is by meta-data region and data district's groups at the meta-data region is used to deposit The metadata of storage container, the metadata of the container include container identifier, the similar signatures of container, hook signature, in container The similar signatures and finger print information for the data block for being included, the data field are used to storing data block；

29), data clusters: each container is handled as follows: inquiry container routing table, according to the similar label of the container Name prefix finds corresponding route entry in container routing table, sends container according to the memory node address indicated in the route entry Container order is write to the container module for reading and writing of corresponding memory node, and to the transmission of the container module for reading and writing of the memory node；Institute It states container routing table to be made of route entry, for establishing container identifier prefix or container similar signatures prefix and memory node Mapping between location, the route entry be binary group<container identifier prefix, memory node address>；Before the container identifier Sew the container similar signatures prefix equal to same container；

291), terminate judgement: if not receiving the Backup end request of client computer, going to step (21)；Otherwise terminate This backup job.

The data reconstruction method in turn includes the following steps:

31) it, initializes: generating an empty data in memory and restore buffer area and an empty filebuf, if The fingerprint number that a counter Counter is used to record processing is set, Counter is reset；

32) file index: the file index that subscribing client sends over, is received, a read pointer P is set and is directed toward First fingerprint in file index；

33), buffer area is inquired: being read fingerprint pointed by P, is denoted as fp, the value of Counter is added 1, restores slow in data It rushes in the fingerprint index table in area inquiry fingerprint fp: if found, being found from the container chained list that data restore buffer area and include The container of fingerprint fp, the counter field of chained list node, reads from the container where the value of Counter is assigned to the container Data block corresponding to fingerprint fp, is denoted as D, enters step 38), otherwise, enters step 34)；The data restore buffer area by Fingerprint index table and container chained list composition；The fingerprint index table is memory Hash table；The memory Hash table includes a bucket Group；A barrel corresponding number each of in the bucket group, and the mapping between fingerprint and bucket number is established using hash function, it reflects The fingerprint being mapped in bucket is stored in the index node of index node chained list；The index node includes that fingerprint field, container refer to Needle field and chain table pointer field；The fingerprint field stores a fingerprint, and the container pointer field is stored where the fingerprint Address of the container in container chained list, the chain table pointer field stores next index in the same index node chained list and ties The address of point；The container chained list is the logical memory chained list known, and the container that write-in data restore buffer area is linked at the memory In chained list；The memory chained list is made of a head pointer and multiple chained list nodes linked together, and chained list node includes one A counter field and a container；

34), fingerprint location: inquiry fingerprint routing table finds corresponding road in fingerprint routing table according to the prefix of fingerprint fp By item, fingerprint fp is sent to the fingerprint queries mould of corresponding memory node according to the memory node address indicated in the route entry Block, and fingerprint location order is sent to the fingerprint queries module of the memory node；The fingerprint routing table is made of route entry, is used In the mapping established between fingerprint prefix and memory node address, the route entry is binary group < fingerprint prefix, memory node Address >；

35), fingerprint location result judges: the positioning result for receiving fingerprint fp is gone to step if positioning result is negative 392)；Otherwise, a container identifier is obtained in positioning result, is denoted as cid, is entered step 36)；

36), read container: inquiry container routing table finds corresponding route entry in container routing table according to the prefix of cid, Cid is sent to the container module for reading and writing of corresponding memory node according to the memory node address indicated in the route entry, and to The container module for reading and writing of the memory node, which is sent, reads container order；The container identifier is one M+N+S binary numbers, The first M full prefix for container identifier is preceding M of the container similar signatures, and intermediate N is number, is container place The number of memory node, it is serial number of the container on memory node that last S, which is serial number,；The container identifier prefix is Refer to that the m(m of the full prefix of the container identifier is the integer for being less than or equal to M more than or equal to 1) position prefix；

37), writing buffer: receiving and read the container that container order returns, and container write-in data is restored buffer area, from this Data block corresponding to fingerprint fp is read in container, is denoted as D；

38), restore file data: filebuf is written into data block D；If filebuf is full, from wherein moving A part of data out, and the data of removal are sent to client computer；

39), file index judges: read pointer P moves forward a step, next fingerprint of file index is directed toward, if P is non- 33) sky is then gone to step；Otherwise, the remaining data in filebuf is removed and is sent to client computer, and sent to client computer File data restores end signal, enters step 391)；

391), terminate judgment step: if the data for not receiving client computer restore ending request, entering step 32)； Otherwise, it enters step 393)；

392), error handling processing: sending file index error signal to client computer, and malfunction reason are as follows: can not find out finger in system Line fp；

393), terminate: deleting data and restore the retrogressing of the data structures such as buffer area, filebuf, counter Counter Out.

The fingerprint queries order in turn includes the following steps:

41) it, takes the fingerprint: extracting the fingerprint to be inquired from fingerprint queries order, be denoted as fp；

42), filter is inquired: fp is inquired in Bloom filter, if do not found, to depositing for request fingerprint queries Terminate after the information of storage node return " fp is new fingerprint "；Otherwise, it goes to step 43)；The Bloom filter is the logical data known Query structure, all fingerprints in data block index for indicating this memory node in memory；

43), data block search index: the data block index is the logical disk Hash table known, the disk Hash Table is mapped fingerprints in corresponding bucket using hash function, in the bucket store binary group<fp, cid>；In data block Fp is inquired in index, if found, the container identifier of container, is denoted as cid where obtaining fingerprint fp, to request fingerprint queries Memory node return to the information of " fp is old fingerprint, be included in container cid in " after terminate；Otherwise, to request fingerprint queries Terminate after the information of memory node return " fp is new fingerprint ".

The fingerprint location order in turn includes the following steps:

51) it, takes the fingerprint: extracting the fingerprint to be positioned from fingerprint location order, be denoted as fp；

52), data block search index: inquiring fp in data block index, if found, obtains the place fingerprint fp The container identifier of container, is denoted as cid, terminates after returning to container identifier " cid " to the memory node of request fingerprint location；It is no Then, terminate after returning to negative " -1 " to the memory node of request fingerprint location.

The execution method of the data block index upgrade order are as follows:

61) it, extracts binary group: extracting binary group<fp, cid>wherein from data block index upgrade order, fp is Fingerprint, cid are the container identifiers of container where fp；

62), fp is inserted into Bloom filter；By binary group<fp, in cid>insertion data block subindex.

The distributed fingerprint querying command in turn includes the following steps:

71), receiving data segment fingerprint: receiving the data segment fingerprint that the data read-write module of this memory node sends over, It is denoted as fingerprint collection, a read pointer P is set and is directed toward first fingerprint that fingerprint is concentrated；

72), buffer area is inquired: being read fingerprint pointed by P, is denoted as fp, inquires fp in fingerprint buffer area, if looked into It arrives, then enters step 77)；Otherwise, it enters step 73)；The fingerprint buffer area is the logical memory Hash table known, and the memory is breathed out Uncommon table is mapped fingerprints in corresponding bucket using hash function, stores fingerprint in the bucket；When the fingerprint buffer area is full, adopt Some fingerprints are deleted with the least recently used replacement algorithm of logical knowledge；

73), fingerprint queries: inquiry fingerprint routing table finds corresponding road in fingerprint routing table according to the prefix of fingerprint fp By item, fingerprint fp is sent to the fingerprint queries mould of corresponding memory node according to the memory node address indicated in the route entry Block, and fingerprint queries order is sent to the fingerprint queries module of the memory node；

74), query result judges: receiving the query result of fingerprint fp, if fp is new fingerprint, fp insertion fingerprint is delayed It rushes in area, turns the 78) step, otherwise, fp is old fingerprint, and the container identifier of container where obtaining fingerprint fp, is denoted as cid, turns In next step；

75), read container fingerprint: inquiry container routing table finds corresponding routing in container routing table according to the prefix of cid , cid is sent to the container module for reading and writing of corresponding memory node according to the memory node address indicated in the route entry, and It is sent to the container module for reading and writing of the memory node and reads the order of container fingerprint；

76), buffer area updates: receiving after reading the fingerprint that the order of container fingerprint returns, these fingerprints insertion fingerprint is delayed It rushes in area；

77) it, deletes old fingerprint: fingerprint fp being concentrated from fingerprint and is deleted；

78), terminate judgement: read pointer P moves forward a step, is directed toward next fingerprint that fingerprint is concentrated, if P non-empty, Then turn the 72) step otherwise turn in next step；

79), terminate: if fingerprint concentrates still Yu Zhiwen, the remaining fingerprint that fingerprint is concentrated being returned into this memory node Data read-write module backed off after random, otherwise, the data read-write module backed off after random for returning to this memory node for 0.

8. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: described to write container life The execution method of order in turn includes the following steps:

81) it, receives container: reading container from writing in container order, be denoted as Container, the value of container counter is added 1；The container counter is safeguarded by container module for reading and writing, writes container order write-in originally for recording container module for reading and writing execution Number of containers on memory node；

82) container identifier, is generated:

First: reading the similar signatures of Container, and take its first M full prefix as container identifier；

Secondly: the number of this memory node is read, using the number as the number of container identifier；

Again: the current value of container counter is read, as the serial number of container identifier；

It is last: to generate one M+N+S container identifiers for Container, be denoted as cid, cid is written The meta-data region of Container；

83), write container: Delta Container being sent on this memory node compresses storage assembly, and Xiang Qifa It send and writes container order；

84), data block index upgrade: the data block index is the logical distributed hashtable known, by being distributed in each deposit It stores up the data block on node and indexes composition, the fingerprint for including in these data blocks index is all different, entire backup set There is no duplicate fingerprint in group；For each fingerprint fp for including in Container, a binary group < fp, cid are generated >, fingerprint routing table is inquired, corresponding route entry in fingerprint routing table is found according to the prefix of fingerprint fp, according in the route entry The fingerprint queries module of the corresponding memory node of memory node the address general<fp, cid indicated>be sent to, and to the storage The fingerprint queries module of node sends data block index upgrade order.

The execution method for reading container order are as follows:

First: extracting the container identifier read in container order, be denoted as cid；

Secondly: the cid Delta for being sent to this memory node being compressed into storage assembly, and is sent to it and reads container order；

It is last: to receive the container that the Delta compression storage assembly of this memory node returns, and the container is returned into request Read the memory node of container.

The execution method for reading the order of container fingerprint are as follows:

First: extracting the container identifier read in the order of container fingerprint, be denoted as cid；

Secondly: the cid Delta for being sent to this memory node being compressed into storage assembly, and is sent to it and reads container fingerprint life It enables；

It is last: to receive the container fingerprint that the Delta compression storage assembly of this memory node returns, and these fingerprints are returned The memory node of container fingerprint is read to request.

The execution of the Data Migration order in turn includes the following steps:

91), subindex migrates: all binary groups in the data block index of this memory node is read, for reading Each binary group<fp, cid>, if the kth of fingerprint fp+1 is 0, general<fp, cid>address is sent to as the new of addr2 The fingerprint queries module of memory node, and it is sent to it data block index upgrade order；If the kth of fingerprint fp+1 is 1, Then general<fp, cid>it is sent to the fingerprint queries module for the new memory node that address is addr3, and it is sent to it data block rope Draw more newer command；

92) it, redirects: fingerprint queries order, the fingerprint location order, data block index that this memory node is received More newer command is redirected to new memory node, that is, kth+1 for detecting fingerprint, and if it is 0, corresponding order is transmitted to ground The new memory node that location is addr2 executes；If it is 1, corresponding order is transmitted to the new memory node that address is addr3 It executes；The container order of writing that this memory node is received is redirected to new memory node, the i.e. w of detection container similar signatures + 1, if it is 0, the new memory node execution that address is addr2 is transmitted to container order is write；If it is 1, hold writing Device order is transmitted to the new memory node that address is addr3 and executes；Reading container, the reading container fingerprint that this memory node is received Order redirects, i.e. the number of detection container identifier, if number is num1, executes order by this memory node, If number is num2, corresponding order is transmitted to the new memory node that address is addr2 and is executed；If number is Corresponding order is then transmitted to the new memory node that address is addr3 and executed by num3；

93), container migrates: sending to the Delta of this memory node compression storage assembly and reads container order, reads its storage All containers each container of reading, if it is 0, is sent out the container by w+1 of detection container similar signatures It gives address and is the new memory node of addr2, and be sent to it and write container order；If it is 1, which is sent to address For the new memory node of addr3, and it is sent to it and writes container order；

94), routing update:

First: the fingerprint routing table of this memory node and container routing table being sent to new memory node, as new storage The fingerprint routing table and container routing table of node；

Then: all memory nodes for including into backup cluster include that new memory node broadcast updates, by fingerprint Route entry < a in routing table₁a₂…a_k, addr1>deletion, and increase new route entry<a₁a₂…a_k0, addr2>and<a₁a₂… a_k1, addr3>；By route entry <b in container routing table₁b₂…b_w, addr1>deletion, and increase new route entry<b₁b₂… b_w0, addr2>with<b₁b₂…b_w1, addr3>；

Terminate: this memory node stops the data backup and resume request of subscribing client, the existing reading of this memory node The order of container fingerprint is read in container order, and data backup and data resume operation the backed off after random that is finished.

The container storage algorithm in turn includes the following steps:

(201), it initializes:

First: parameter S, R and L are read from configuration file；The configuration file is stationed on disk unit, for recording The configuration information of system；

The parameter S is preset positive integer, indicates read from the container storage pond of disk when Delta compression Enter the maximum number of the similar vessels of memory, the similar vessels refer to the similar container of content；

The parameter R is preset positive number, indicates that the minimum Delta compression ratio allowed, the Delta compression ratio are Refer to after generating Delta block to data block progress Delta compression using reference block, the ratio of data block size and Delta block size；

The parameter L is preset positive integer, indicates maximum Delta chain length；

Then: judging whether to be system configuration initial stage, then generate an empty similarity indexing in memory in this way；If not, The similarity indexing of backup is read in into memory from configuration file；

It is last: to generate an empty similar buffer area in memory；An empty vessel buffers area is generated in memory, is used To keep in from the container read in the container storage pond on disk in memory；An empty storehouse is generated in memory, is denoted as Stack；

(202), it receives container: receiving the container of writing that one sends over from the block grade data deduplication storage on upper layer and order It enables, extracts container to be written in container order from writing, be denoted as upper layer container；An empty format is generated in memory to hold Device is denoted as Work container, and vessel buffers area is written in Work container；It is spare to empty similar buffer area；The upper layer container refers to Container used in the block grade data deduplication storage on upper layer；The format container refers to Delta compression storage assembly Container used in container access module；

(203), copies of metadata:

First: reading the similar signatures of container identifier and container from the meta-data region of upper layer container, be respectively written into work The container identification field and container signature field of the vessel head of container；

Secondly: from the meta-data region read block fingerprint of upper layer container, these data block fingerprints are original according to it Sequence is sequentially written in the fingerprint region of Work container；

Again: being successively read similar signatures from the meta-data region of upper layer container, generated for each similar signatures read One similar signatures block, and by the similar signatures field of similar signatures write-in similar signatures block, according to corresponding similar signatures original Similar signatures block is sequentially written in the similar signatures area of Work container by the sequence come；

It is last: to read hook signature from the meta-data region of upper layer container, these hooks are signed according to its original sequence It is sequentially written in the hook signature area of Work container；

(204), similarity indexing updates:

First: the container identifier of Work container is assigned to variable cid；

Secondly: signing, be handled as follows: by the hook for each hook for including in Work container hook signature area Son signature is assigned to variable hook, generates a mapping<hook, and cid>, general<hook, cid>be inserted into similarity indexing；

(205), similar vessels are searched: inquiry similarity indexing, are found out and are shared those of hook signature container with Work container, It is if it is not found, then go to step (226)；Otherwise, according to sharing the quantity of hook signature with Work container from big to small from looking for To container in choose most S containers, confirm that these containers for being selected are the similar vessels of Work container；In vessel buffers These similar vessels are searched in area, and the similar vessels not in vessel buffers area are read into vessel buffers from container storage pool Area；

(206), similar buffer area is write:

The similar signatures area for successively scanning each similar vessels of Work container in vessel buffers area, reads similar signatures All similar signatures blocks in area are handled as follows each similar signatures block of reading: generation one is similar in memory Buffer area index node, is denoted as Node；The type field value of the similar signatures block and offset word segment value are individually copied to Node Type field and offset field；By the container identifier word of the container identifier write-in Node of the similar signatures block said container Section；The similar signatures field for reading the similar signatures block remembers that the similar signatures of reading are sign, general<sign, Node>insertion To similar buffer area；

(207), prepare processing data block: one read pointer P1 of setting is directed toward first phase in Work container similar signatures area Like signaling block, first data block that a read pointer P2 is directed toward upper layer container data area is set；

(208), read block: data block pointed by a P2 is read from the container data area of upper layer, is denoted as D_r, from Similar signatures block pointed by a P1 is read in Work container similar signatures area, is denoted as Block, is read the similar label of Block The value of file-name field, is denoted as sign1；

(209), similar buffer area is searched: the similar label that similar signatures field value is sign1 are searched in similar buffer area Name node, goes to step (222) if it is not found,；Otherwise, a read pointer P3 is set and is directed toward the similar signatures node just found Index node chained list first index node, go to step (210)；

(210), judge data block: if the type field value for the index node that P3 is directed toward for Delta block mark, turn the (212) step；Otherwise, it is data block mark, goes to step (211)；

(211), short chain Delta operation: by the value and offset word of the container identification field of index node pointed by P3 The value of section is assigned to variable cid0 and offset0 respectively, and from address<cid0 in container buffer area, offset0>place is read One data block, is denoted as D₀；With D₀For reference block and D_rDelta operation is carried out, Delta block △ is generated_0,r；If Delta is pressed Shrinkage is greater than or equal to R, then compresses success, go to step (223)；Otherwise, compression failure, goes to step (212)；

(212), skip Delta block: P3 moves forward a step, is directed toward next index node, if P3 non-empty, turns the (210) otherwise step shows the tail portion for having arrived at index node chained list, P3 is directed toward to the first of the index node chained list again A index node goes to step (213)；

(213), judge Delta block: if the type field value for the index node that P3 is directed toward for data block mark, turn the (221) otherwise step is Delta block mark, goes to step (214)；

(214), Delta block: the value and offset field of the container identification field of index node pointed by P3 is read Value designate address of the Delta block in vessel buffers area, from the address read a Delta block；

(215), it is pressed into storehouse: the Delta block being pressed into Stack, reference is read from the Delta build of the Delta block The reference block address of reading is stored in variable<cid1 by block address, and offset1>in, wherein cid1 is container identifier, Offset1 is the position of the reference block in the data field of container cid1；

(216), reference block is read: if container cid1 in vessel buffers area, reads reference from container buffer area Block<cid1, offset1>, otherwise, container cid1 is read in into vessel buffers area from container storage pool first, is then read again Reference block<cid1, offset1>；

(217), judgement reference block: if reference block<cid1, offset1>it is Delta block, turn (215) step, it is no Then, it is data block, the content of the data block is stored in variables D₀In, general<cid1, offset1>it is assigned to variable<cid0, Offset0 >, it is assigned to variable length by 1, is turned in next step；

(218), long-chain Delta calculation step: with D₀For reference block and D_rIt carries out Delta operation and generates Delta block △_0,r；If Delta compression ratio is greater than or equal to R and length is less than or equal to L, success is compressed, (223) are gone to step, Otherwise, compression failure turns in next step；

(219), judge storehouse: if Stack goes to step (221) for sky, otherwise, turning in next step；

(220), pop-up a stack: popping up a Delta block from Stack, be denoted as △, by the address of △ deposit variable < Cid0, offset0 >, to D₀Delta inverse operation is carried out with △, the result of Delta inverse operation is stored in variables D₀In, by variable The value of length increases by 1, goes to step (218)；

(221), skip data block: P3 moves forward a step, is directed toward next index node, if P3 non-empty, turns the (213) otherwise step shows the tail portion for having arrived at index node chained list, turn in next step；

(222), storing data block: in data block D_rBefore add a data block head, be written in the data block head Data block mark and D_rSize information；If the data field non-empty of Work container, by the data block D after additional data build_r It adds in the data field of Work container behind data with existing, otherwise, by the data block D after additional data build_rWork is written Make the initial position of the data field of container；By the type field of data block mark write-in Block, by data block D_rIn Work container The offset field of the location information write-in Block of data field, goes to step (224)；

(223), Delta block is stored: in Delta block △_0,rBefore add a Delta build, in the Delta build In be written Delta block mark, △_0,rSize and △_0,rReference block address<cid0, offset0>；If work is held The data field non-empty of device, then by the Delta block △ after additional Delta build_0,rAddition has number in the data field of Work container Behind, otherwise, by the Delta block △ after additional Delta build_0,rThe initial position of the data field of Work container is written；It will The type field of Block is written in Delta block mark, by Delta block △_0,rLocation information in Work container data field is written The offset field of Block；Storehouse Stack is emptied；

(224), similar buffer area updates: generating a similar buffer area index node in memory, is denoted as Node1；It will The type field value and offset word segment value of Block are individually copied to the type field and offset field of Node1；By Work container The container identification field of container identifier write-in Node1；General<sign1, Node1>it is inserted into similar buffer area；

(225), data block is disposed judgement: P1 being moved forward a step, is directed toward in the similar signatures area of Work container Next similar signatures block, by P2 move forward a step, be directed toward upper layer container data area in next data block；If P2 For sky, then shows that the data block in the container of upper layer is all disposed, go to step (227)；Otherwise, (208) are gone to step；

(226), former container is stored: first: from first data BOB(beginning of block), successively handling every in upper layer container data area One data block adds data block head before the data block, writing data blocks mark and the data block in data block head Size information；It is if the data field non-empty of Work container, the data block after additional data build is additional in Work container In data field behind data with existing, otherwise, by rising for the data field of the data block write-in Work container after additional data build Beginning position；Similar signatures block corresponding with the data block in the similar signatures area of Work container is handled, by data block mark The type field of the similar signatures block is written in will, and the similar label are written in the location information by the data block in Work container data field The offset field of name block (NAM)；

Secondly: abandoning upper layer container；It calculates the size of Work container and size information is written to the vessel head of Work container； If container storage pond non-empty, Work container is added in container storage pond behind data with existing, otherwise, work is held The initial position in device write-in container storage pond；

It is last: by the container identifier of the Work container in rigid write-in container storage pond and the Work container in container storage Container index is written in location information in pond, goes to step (228)；

(227), new container is stored:

First: abandoning upper layer container；It calculates the size of Work container and size information is written to the vessel head of Work container； If container storage pond non-empty, Work container is added in container storage pond behind data with existing, otherwise, work is held The initial position in device write-in container storage pond；

Secondly: by the container identifier of the Work container in rigid write-in container storage pond and the Work container in container storage Container index is written in location information in pond；

(228), end of run judges: judging whether to receive end of run instruction, such as otherwise goes to step (202)；If so, Then step (229)；

(229), terminate:

First: stopping receiving the container that the block grade data deduplication storage on upper layer sends over；

Then: configuration file is written into the similarity indexing in memory；

It is last: to destroy similarity indexing, vessel buffers area, similar buffer area and the storehouse Stack backed off after random in memory.

The container recovery algorithms in turn include the following steps:

(301), it initializes: generating an empty vessel buffers area in memory, deposited for temporary from the container on disk The container in memory is read in reservoir；An empty storehouse is generated in memory, is denoted as Stack；

(302), it receives read command: receiving a reading container sended over from the block grade data deduplication storage on upper layer Order is denoted as cid from extraction vessel identifier in container order is read；An empty upper layer format container, note are generated in memory For upper layer container；

(303), it reads container: reading the container that container identifier is cid in the container storage pond on disk, be denoted as work Make container, vessel buffers area is written into Work container；

(304), metadata recovering step: according to the type and call format of upper layer container metadata from the member of Work container Corresponding metadata is read in data field, by the meta-data region of these metadata write-in upper layer container of reading；

(305), prepare processing data field: one read pointer P1 of setting is directed toward first object of Work container data field；

(306), judge object: if object pointed by P1 is a data block, which being denoted as D_r, turn Step (312)；Otherwise, it is Delta block, turns in next step；

(307), it is pressed into storehouse: the Delta block being pressed into Stack, reference is read from the Delta build of the Delta block Block address, remember reading reference block address be<cid1, offset1>, wherein cid1 is container identifier, and offset1 is this Quote position of the block in the data field of container cid1；

(308), reference block is read: if container cid1 in vessel buffers area, reads reference from container buffer area Block<cid1, offset1>, otherwise, container cid1 is read in into vessel buffers area from container storage pool first, is then read again Reference block<cid1, offset1>；

(309), judgement reference block step: if reference block<cid1, offset1>it is Delta block, go to step (307)； Otherwise, it is data block, which is stored in variables D, turns in next step；

(310), pop-up a stack: popping up a Delta block from Stack, be denoted as △, carries out the inverse fortune of Delta to D and △ It calculates, it will be in the result deposit variables D of Delta inverse operation；

(311), judge storehouse: if Stack non-empty, turning (310) step；Otherwise, the content of variables D is denoted as data block D_r, turn in next step；

(312), data block is copied: if upper layer container data area non-empty, by data block D_rIt adds in upper layer container In data field behind data with existing；Otherwise, by data block D_rThe initial position in upper layer container data area is written；

(313), judge data field: read pointer P1 moves forward a step, and it is next right in Work container data field to be directed toward As going to step (306) if P1 non-empty；Otherwise, data field is disposed, and turns in next step；

(314), end of run judges: the upper layer container handled well is sent to the block grade data deduplication storage system on upper layer System goes to step (302) if being not received by end operation order；Otherwise, turn in next step；

(315), terminate: destroying vessel buffers area and the operation of storehouse Stack backed off after random.

The invention proposes a kind of backup cluster based on two-stage data deduplication, which saves comprising multiple storages Point interconnects by storing network, cooperates the backup and recovery for completing data, and data backup and resume performance is good, props up simultaneously Online data migration is held, backup cluster scalability is strong.

1, using piecemeal and fragmentation technique based on content, reduce the boundary shifts generated by the partial modification of data Phenomenon protects the redundancy locality of data, is conducive to improve data de-duplication ratio；It is stored using container by logical order The new data block also effective protection redundancy locality of data, is conducive to improve data processing and restorability；

2, both elimination of duplicate data block in global scope eliminates set of metadata of similar data further through data clusters and Delta compression Repeated data between block, to support fine-grained data duplicate removal；Only new data block saves storage in storage transmission over networks Network bandwidth；

3, distributed fingerprint querying command indexes three-level fingerprint queries using fingerprint buffer area, Bloom filter and data block Mechanism had not only reduced the magnetic disc i/o expense of fingerprint queries, but also supported distributed parallel inquiry, looked into so as to effectively improve fingerprint Ask efficiency and data deduplication performance；

4, the design that data restore buffer area can effectively reduce the magnetic disc i/o expense in data recovery procedure, improve data Restorability；

5, being handled in set of metadata of similar data cluster to identical memory node and as unit of container, be conducive to bottom Delta compresses the seeking scope that storage assembly reduces set of metadata of similar data, improves the search efficiency and Delta compressibility of set of metadata of similar data Can, because the set of metadata of similar data block of data block is most probably also in same container in same container；

6, online data migration is supported, so that system be allowed to increase more storage sections as needed in the process of running Point, so that the performance of system and capacity are with good expansibility.

Detailed description of the invention

Fig. 1 is the present invention and client system architecture schematic diagram；

Fig. 2 is storage node structure schematic diagram；

Fig. 3 is data back up method flow chart；

Fig. 4 is data reconstruction method flow chart；

Fig. 5 is distributed fingerprint querying command flow chart；

Fig. 6 is Data Migration order flow chart；

Fig. 7 is fingerprint index table structure schematic diagram；

Fig. 8 is fingerprint index table index node structure figure；

Fig. 9 is container storage algorithm flow chart；

Figure 10 is container recovery algorithms flow chart；

Figure 11 is structure of container figure；

Figure 12 is similar signatures block structural diagram；

Figure 13 is data block head structure chart；

Figure 14 is Delta build structure chart；

Figure 15 is similarity indexing structural schematic diagram；

Figure 16 is hook signature node structure figure；

Figure 17 is similar buffer area structural schematic diagram；

Figure 18 is similar signatures node structure figure；

Figure 19 is similar buffer area index node structure chart.

Specific embodiment

The invention discloses a kind of backup clusters based on two-stage data deduplication, as shown in Figure 1, passing through storage including multiple The memory node of network interconnection, each memory node the data that send over of subscribing client and can back up data to In cluster, or restore specified data from cluster.

As shown in Fig. 2, being equipped with the software systems and disk unit based on two-stage data deduplication on the memory node； The upper layer i.e. block grade data deduplication that the software systems based on two-stage data deduplication are divided into responsible first order data deduplication stores System, the bottom, that is, Delta for being responsible for second level data deduplication compress storage assembly；Data block is installed on the disk unit Index, container index and container storage pond；The data block index is by the data deduplication storage management of block grade and use；Institute It states container index and container storage pond and storage assembly management and use is compressed by Delta.Described piece of grade data deduplication storage Including three data read-write module, fingerprint queries module and container module for reading and writing modules, it is additionally provided with Bloom filter, fingerprint road Restore buffer area by table, container routing table, input block, filebuf, fingerprint buffer area and data.The reading and writing data Module includes data back up method and data reconstruction method；The fingerprint queries module includes fingerprint queries order, fingerprint location Order, data block index upgrade order and distributed fingerprint querying command；The container module for reading and writing include write container order, It reads container order, read the migration order of container fingerprint order and data；Data read-write module is monitored client computer on network and is sended over Data backup or recovery request, it is standby to execute the data that data back up method or data reconstruction method customer in response machine send over Part or recovery request.

The Delta compression storage assembly is provided with container access module, similarity indexing, similar buffer area and vessel buffers Area, Delta compression storage assembly receive and execute that the block grade data deduplication storage on upper layer sends over write container order, Read container order or read the order of container fingerprint: to container order is write, Delta compresses storage assembly and carries out Delta compression to container, And it will be in the container storage pond on the compressed container write-in disk of Delta；To container order is read, Delta compresses storage assembly Specified container is read from the container storage pond on disk by container index, returns to upper layer after the container of reading is restored Block grade data deduplication storage；To the order of container fingerprint is read, Delta compresses storage assembly from the container storage pond on disk These fingerprints, are sent to the block grade data deduplication storage on upper layer by the middle fingerprint for reading specified containers and including；

The first order of two-stage data deduplication is started by the memory node of execution data backup, using splits' positions technology, The similar new data block cluster of content makes do not have repeated data block in entire cluster to identical memory node；The new data Block refers to the data block being different from all data blocks existing in cluster.It is saved by each storage the second level of two-stage data deduplication Point is individually completed, and using Delta compress technique, is carried out Delta compression to the set of metadata of similar data block in this memory node, is eliminated byte Grade repeated data.

As shown in figure 3, the data back up method in turn includes the following steps:

21) data flow: the data that subscribing client sends over, is received, input block is written into the data received； The input block uses queue structure, and the queue structure is the mature prior art.

22), piecemeal and calculating fingerprint: the dividing the data in input block based on the method for partition of content of logical knowledge is used At data block, the keyed Hash function of logical knowledge is used to calculate the cryptographic Hash of each data block contents as the fingerprint of the data block；

In the present embodiment, data can be divided by the elongated data block that desired size is 8KB using the logical CDC algorithm known, Data block fingerprint is calculated using SHA-1 hash function, fingerprint length is 20 bytes.

23) similar signatures, are calculated: calculating the similar signatures of each data block, i.e., since the initial position of data block, It is slided in data block with the window of a fixed size, as soon as before every sliding byte, use the logical sieve guest's fingerprint algorithm known Sieve guest's fingerprint for falling into data patch in window is calculated, takes in all data patch the smallest guest sieve fingerprint as data block Similar signatures；In the present embodiment, the size of the window is predetermined a constant, can use 512 bytes, and guest sieve refers to The length of line can use 4 bytes.

In the present embodiment, r is an important parameter, and r setting is too small and is mostly unfavorable for data deduplication efficiency and processing excessively Performance, in an implementation, r, which takes, 12 or 13 to be advisable, and such a data segment averagely includes 2¹²Or 2¹³A data block.In the present embodiment, Data are segmented with the fragmentation technique based on content, such application program is just difficult to influence to the modification of data segment Data outside data segment are conducive to the redundancy locality for protecting data.

In the present embodiment, the hook signature sampling rate is a critically important parameter, if its value is too small, the hook generated Son signature is very little, will affect the lookup precision of set of metadata of similar data block, if its value is excessive, the hook signature generated is too many, and meeting So that the similarity indexing of the Delta compression storage assembly of bottom is excessive, memory overhead is high.In an implementation, hook signature sampling rate It can value 1/64 or 1/32.

28), container encapsulation step: according to the return of fingerprint queries module as a result, successively handling each data segment: abandoning number According to the data block corresponding with its of the fingerprint in returning the result is not comprised in section, if still remaining data block in data segment, this A little data blocks are new data block, distribute a container for the data segment to store new data block；Take data segment similar signatures and Hook signature is respectively the similar signatures and hook signature of container；Similar signatures, the hook of container are signed, the finger of new data block Line, new data block similar signatures write-in container meta-data region, new data block write-in container data field；After processing Data segment deleted from input block；The container is by meta-data region and data district's groups at the meta-data region is used to deposit The metadata of storage container, the metadata of the container include container identifier, the similar signatures of container, hook signature, in container The information such as the similar signatures and fingerprint of the data block for being included, the data field are used to storing data block；

It is that unit is handled according to data segment, in addition to using the new data in container encapsulation of data section in the present embodiment Outside block, container also is arrived using the similar signatures of data segment and hook signature as the similar signatures of container and hook signature storage In, and it includes the phase of old data block that the similar signatures of data segment and hook signature, which are all data blocks for including in data segment, It is handled and is obtained like signature, this makes container not have to storing data section while protecting the redundancy locality of data segment In old data block, clustered to not only avoid the storage of repeated data block and be conducive to set of metadata of similar data block, and support bottom Delta compresses storage assembly and is indexed to set of metadata of similar data block and quickly searches；The old data block refers to and backs up in cluster The identical data block of data with existing block.

29), data clusters: each container is handled as follows: inquiry container routing table, according to the similar label of the container Name prefix finds corresponding route entry in container routing table, sends container according to the memory node address indicated in the route entry Container request is write to the container module for reading and writing of corresponding memory node, and to the transmission of the container module for reading and writing of the memory node；Institute It states container routing table to be made of route entry, for establishing container identifier prefix or container similar signatures prefix and memory node Mapping between location, the route entry be binary group<container identifier prefix, memory node address>；Before the container identifier Sew the container similar signatures prefix equal to same container；

In the present embodiment, the container with same and similar signature is clustered on identical memory node, this is conducive to phase Likelihood data block cluster, because very high with its corresponding data segment contents of the container of same and similar signature probability similar to each other, Conversely, content its corresponding container of two data segments similar to each other has the probability of same and similar signature also very high.Due to number The redundancy locality of data is protected according to section, so that the set of metadata of similar data block of data block is very likely also in same appearance in same container In device.

As shown in figure 4, the data reconstruction method in turn includes the following steps:

31) it, initializes: generating an empty data in memory and restore buffer area and an empty filebuf, if The fingerprint number that a counter Counter is used to record processing is set, Counter is reset；The filebuf uses queue Structure, the queue structure are the mature prior art.

33), buffer area is inquired: being read fingerprint pointed by P, is denoted as fp, the value of Counter is added 1, restores slow in data It rushes in the fingerprint index table in area inquiry fingerprint fp: if found, being found from the container chained list that data restore buffer area and include The container of fingerprint fp, the counter field of chained list node, reads from the container where the value of Counter is assigned to the container Data block corresponding to fingerprint fp, is denoted as D, enters step 38), otherwise, enters step 34)；The data restore buffer area by Fingerprint index table and container chained list composition；Such as Fig. 7, the fingerprint index table is memory Hash table；The memory Hash table includes One bucket group；A barrel corresponding number each of in the bucket group, and established between fingerprint and bucket number using hash function Mapping, the fingerprint being mapped in bucket are stored in the index node of index node chained list；As shown in figure 8, the index node packet Include fingerprint field, container pointer field and chain table pointer field；The fingerprint field stores a fingerprint, the container pointer word Address of the container in container chained list where the Duan Cunfang fingerprint, the chain table pointer field store the same index node chain The address of next index node in table；The container chained list is the logical memory chained list known, and write-in data restore the appearance of buffer area Device is linked in the memory chained list；The memory chained list is by a head pointer and multiple chained list node groups linked together At chained list node includes a counter field and a container.

34), fingerprint location: inquiry fingerprint routing table finds corresponding road in fingerprint routing table according to the prefix of fingerprint fp By item, fingerprint fp is sent to the fingerprint queries mould of corresponding memory node according to the memory node address indicated in the route entry Block, and fingerprint location request is sent to the fingerprint queries module of the memory node；The fingerprint routing table is made of route entry, is used In the mapping established between fingerprint prefix and memory node address, the route entry is binary group < fingerprint prefix, memory node Address >；

In the present embodiment, the M determines the storage section for allowing to include in the maximum-norm of system, that is, backup cluster Point number is no more than 2^M；The N is the digit of memory node number in backup cluster, each memory node in the backup cluster There is a unique number, which is a N bit；In an implementation, it should ensure that M is greater than N, M can use 12, N can 10 are taken, in this way, backup cluster can at most have 2¹⁰A memory node is able to satisfy the needs of large-scale cluster backup；The S is determined The maximum storage capacity of single memory node, that is, the container number for allowing to store on single memory node are no more than 2^S, In implementation, relatively large S value may be selected, there are enough leeway to System Expansion；Such as S value 26, single memory node 2 can at most be stored²⁶A container, the logical data of each one data segment of container storage, by each data segment average 2¹³A data Block is averaged each data block 8KB to calculate, and the largest logical memory capacity for backing up cluster can reach 2¹⁰×2²⁶×2¹³×8KB= 4EB, if it is considered that many data segments may be without new data block thus the case where no need to consume containers, actual logic storage Capacity is also greater than 4EB, and still, the logical data of 4EB amount of physical memory actually required after duplicate removal can be far smaller than 4EB。

The detailed process of " container write-in data are restored into buffer area " are as follows:

First: judge that data restore whether buffer area has expired, it, will be in container chained list if data, which restore buffer area, to have expired The smallest chained list node deletion of the value of counter field, and by all fingerprints for including in the container of the chained list node from fingerprint rope Draw in table and deletes；It is described to judge whether full method is the mature prior art for data recovery buffer area；

Secondly: the container being linked in container chained list, and the value of counter Counter is assigned to container place chain The counter field of table node；

Last: all fingerprints for including by the container are inserted into fingerprint index table, and by the container in container chained list In address the container pointer field of index node where these fingerprints is written.

In an implementation, the data, which restore buffer area, can effectively improve data recovery performance, the reason for this is that: read a finger When the corresponding data block of line, restore to inquire this fingerprint in buffer area first in data, if hit, directly can restore slow in data It rushes in area when reading the corresponding data block, only miss of the fingerprint, just needs the data block inquired on disk index, finds phase The container identifier answered is restored in buffer area according to container identifier from container being read into corresponding memory node data, Magnetic disc i/o can read in whole container into memory, and the data block in the same container is very likely accessed again, Restore buffer area hit rate to maintain higher data, reduces the magnetic disc i/o expense needed for data are restored.

The fingerprint queries module is monitored and executes other memory nodes or this memory node in backup cluster and sends over Fingerprint queries order, the more newer command of fingerprint location order or data block subindex；The fingerprint queries module is monitored and is executed The distributed fingerprint querying command that the data read-write module of this memory node sends over.

The fingerprint queries order in turn includes the following steps:

In the present embodiment, the fingerprint queries order has used Bloom filter and data block subindex second level fingerprint queries Mechanism, the Bloom filter are stationed in memory, and the data block index is stationed on disk；When inquiring a fingerprint, It is inquired in Bloom filter first, if do not found, can affirm that the fingerprint is new fingerprint, if found, because Bloom filter cannot affirm that the fingerprint is old fingerprint there are false alarm rate, need to continue to inquire in data block index；It is described New fingerprint refers to and backs up that the fingerprint that existing all fingerprints are different from cluster, the old fingerprint refer in backup cluster Some fingerprints；Appropriately sized Bloom filter is set according to system average size, the false alarm rate foot of Bloom filter can be made It is enough small, it is identified to make most new fingerprint that can inquire by Bloom filter, reduces the magnetic disc i/o of fingerprint queries Expense.

In an implementation, Bloom filter size can be according to average each in system average size, that is, backup cluster The physical storage capacity of memory node is set, it is assumed that system average size is vKB, and x is the digit of Bloom filter, and y is cloth The fingerprint number stored in grand filter, b are data block size, and r is the average Delta compression ratio of bottom, then have y=vr/b, Guarantee that the false alarm rate of Bloom filter is less than or equal to 2%, need only guarantee that x/y is greater than or equal to 8, under typical case, b is general It may be set to 4 for 8KB, r, then can set x=8y=8vr/b=4v, the size of Bloom filter is v*4*2^-3*2^-30GB=v*2^- ³¹GB, that is to say, that the physical storage capacity that the Bloom filter of every 1GB is supported is 2³¹KB=2TB；If guaranteeing the grand filtering of cloth The false alarm rate of device is less than or equal to 2%, then is more than that 98% new fingerprint can inquire identification by Bloom filter.

The execution method of the fingerprint location order in turn includes the following steps:

The execution method of the data block index upgrade order are as follows:

As shown in figure 5, the distributed fingerprint querying command in turn includes the following steps:

In the present embodiment, the distributed fingerprint querying command has used three-level fingerprint queries mechanism: fingerprint buffer area, Bloom filter and data block subindex, wherein fingerprint buffer area and Bloom filter are stationed in memory, data block index It stations on disk；When inquiring a fingerprint, inquired in the fingerprint buffer area of this memory node first, it, can if hit Determine that the fingerprint is old fingerprint, if in recklessly, further inquired on corresponding memory node by fingerprint queries order； The fingerprint queries order has used Bloom filter and data block subindex second level to inquire mechanism and has further identified fingerprint；If Confirm that inquired fingerprint is old fingerprint eventually by inquiry data block index, then passes through the step 75) and step 76) All fingerprints in container comprising the fingerprint are read in into fingerprint buffer area, because container protects the redundancy locality of data, Fingerprint in same container is very likely accessed again, in this way, a magnetic disc i/o can create hundreds of buffer area Chance is hit, so that fingerprint buffer area is able to maintain that higher hit rate；In three-level inquiry mechanism, the grand mistake of cloth Filter can identify the new fingerprint more than 98%, and the fingerprint buffer area hit rate with higher is most so as to identify Old fingerprint significantly reduces the magnetic disc i/o expense of fingerprint queries.

The container module for reading and writing is monitored and executes other memory nodes or this memory node in backup cluster and sends over Write container order, read container order or read the order of container fingerprint；When adding new memory node in storage cluster, the appearance Device module for reading and writing can also carry out Data Migration order on the Data Migration on this memory node to two new memory nodes, institute Stating Data Migration order can execute online, not influence the normal work of backup cluster；

The container module for reading and writing safeguards a container counter, writes container life for recording container module for reading and writing execution The number of containers being written on this memory node is enabled, the container counter is one S binary counters, and wherein S is to hold The digit of device Identifier Sequence Number.

The execution method for writing container order in turn includes the following steps:

81) it, receives container: reading container from writing in container order, be denoted as Container, the value of container counter is added 1；

82) container identifier, is generated:

The execution method for reading container order are as follows:

The Data Migration order can will be on the Data Migration on this memory node to two new memory nodes；This storage Address of node is denoted as addr1, and the address of two new memory nodes is denoted as addr2 and addr3 respectively；The number of this memory node It is denoted as num1, the number of two new memory nodes is denoted as num2 and num3 respectively；The routing of this memory node in fingerprint routing table Item is denoted as < a₁a₂…a_k, addr1 >, wherein a_i(i=1,2 ..., k) is 0 or 1, the road of this memory node in container routing table By Xiang Jiwei <b₁b₂…b_w, addr1 >, wherein b_i(i=1,2 ..., w) is 0 or 1, and k and w are greater than or equal to 1 integer； Before carrying out Data Migration, the data block index of the new memory node is sky, and container counter is sky；As shown in fig. 6, described The execution of Data Migration order in turn includes the following steps:

92) it, redirects: fingerprint queries order, the fingerprint location order, data block index that this memory node is received More newer command is redirected to new memory node, that is, kth+1 for detecting fingerprint, and if it is 0, corresponding order is transmitted to ground The new memory node that location is addr2 executes；If it is 1, corresponding order is transmitted to the new memory node that address is addr3 It executes；The container order of writing that this memory node is received is redirected to new memory node, the i.e. w of detection container similar signatures + 1, if it is 0, the new memory node execution that address is addr2 is transmitted to container order is write；If it is 1, hold writing Device order is transmitted to the new memory node that address is addr3 and executes；Container is read in the reading container order that receives to this memory node Fingerprint order redirects, i.e. the number of detection container identifier, if number is num1, executes life by this memory node It enables, if number is num2, corresponding order is transmitted to the new memory node that address is addr2 and is executed；If number is Corresponding order is then transmitted to the new memory node that address is addr3 and executed by num3；

94), routing update:

After the Data Migration order is finished, this memory node has just exited backup cluster, while two new storages Node joined backup cluster, and the memory capacity and parallel performance for backing up cluster are all expanded；The data migration process It is transparent to other memory nodes of backup cluster, backup cluster normal work is not influenced.

In the present embodiment, increase memory node using Data Migration algorithm, so that backup cluster can according to need not Disconnected expansion, during backup set group extension, fingerprint routing table and container routing table can also be automatically updated；In configuration, both Can by fingerprint routing table with container configuration as, memory node each so only needs a routing table, can also With by fingerprint routing table and container configuration at different, in this way can be flexible by the load of fingerprint queries and container storage Distribute to different memory nodes in ground；

Assuming that by fingerprint routing table with container configuration as, and there are two deposit backup cluster configuration in the early stage Store up node n1 and n2, then routing table can be set as {<0, n1>,<1, n2>}, if by n1 expand into two memory node n3 and N4, then routing table be automatically updated into<00, n3>,<01, n4>,<1, n2>}, n2 is further expanded into two storages Node n5 and n6, then routing table be automatically updated into again<00, n3>,<01, n4>,<10, n5>,<11, n6>}, pass through Data Migration algorithm, backup cluster can flexibly be expanded, and guarantee that system has stronger scalability.

As shown in figure 9, the container storage algorithm of the container access module in turn includes the following steps:

(201), it initializes:

First: parameter S, R and L are read from configuration file；

The configuration file is stationed on disk unit, for recording the configuration information of system.

The parameter S is preset positive integer, indicates read from the container storage pond of disk when Delta compression Enter the maximum number of the similar vessels of memory, the similar vessels refer to the similar container of content；Parameter S setting is excessive, can drop The Delta compression performance of low data block, setting is too small, then can reduce the Delta compression ratio of data block, in an implementation, S can be set It is set to 2,3 or 4.

The parameter R is preset positive number, indicates that the minimum Delta compression ratio allowed, the Delta compression ratio are Refer to after generating Delta block to data block progress Delta compression using reference block, the ratio of data block size and Delta block size； In an implementation, R may be configured as 2,2.5 or 3.

The parameter L is preset positive integer, indicates maximum Delta chain length；Parameter L setting is too small, can reduce Delta compression effectiveness, is arranged excessive, then can reduce reading and writing data performance, while data compression income obtained and little；? In implementation, parameter L may be configured as 5,6 or 7.

The method for judging system configuration initial stage is the mature prior art.

The similarity indexing is memory Hash table；As shown in figure 15: the memory Hash table includes a bucket group；The bucket A barrel corresponding number each of in group, and the mapping between hook signature and bucket number is established using hash function, it is mapped to Hook signature in bucket is stored in hook signature node；Each hook signature node stores a unique hook and signs and close Join a container identifier queue, the identifier of container of the storage comprising hook signature in container identifier queue；Such as figure Shown in 16, the hook signature node is by hook signature field, spilling chain table pointer field and container identifier queue pointer word Duan Zucheng；The hook signature field is for storing a unique hook signature；The spilling chain table pointer field is for locating The address for another hook signature node that storage is mapped in the same bucket when managing hash-collision；The container identifier queue Pointer field is used to store the first address of the hook signature associated container identifier queue of node；

The similarity indexing is used to establish the mapping between hook signature and the container signed comprising the hook, comprising same The possible more than one of container of a hook signature, in this way, shared hook signature can be quickly found out by inquiring similarity indexing Container；The present invention confirms that the container of shared hook signature is similar vessels, and the similar vessels refer to the similar container of content.

In the present embodiment, the similarity indexing is stationed in memory, is convenient for quick search.

The similar buffer area is memory Hash table；As shown in figure 17, the memory Hash table includes a bucket group；It is described A barrel corresponding number each of in bucket group, and the mapping between similar signatures and bucket number, mapping are established using hash function Similar signatures in bucket are stored in similar signatures node；Each similar signatures node stores a unique similar signatures simultaneously It is associated with an index node chained list, stores index node in index node chained list, wherein each index node stores one The information of data block or Delta block with the similar signatures；The similar signatures of the Delta block refer to that the Delta block is corresponding Data block similar signatures；As shown in figure 18: the similar signatures node is by similar signatures field, spilling chain table pointer field It is formed with index node chain table pointer field；The similar signatures field is for storing a unique similar signatures；It is described to overflow The ground for another similar signatures node that storage is mapped in the same bucket when chain table pointer field is used to handle hash-collision out Location；The index node chain table pointer field is used to store the first address of the associated index node chained list of the similar signatures node； As shown in figure 19: the index node is made of type field, container identification field, offset field and chain table pointer field； The type field is for storing data block mark or Delta mark；The container identification field and offset field give The address information of data block or Delta block；The chain table pointer field is for storing next index in the index node chained list The address of node.

Each similar signatures node is associated with an index node chained list, the index knot in the similar buffer area Each index node stores the index information of a data block or Delta block in point chained list, in the same index node chained list Data block or Delta block similar signatures having the same.

In the present embodiment, the similar buffer area is stationed in memory, convenient for being quickly found out when carrying out Delta compression The reference block of data block to be compressed.

The vessel buffers area is the logical memory chained list known, and the container being read into vessel buffers area is linked at the memory In chained list, when vessel buffers area is full, deleted from the memory chained list using the least recently used replacement algorithm of logical knowledge Container；Work container and its similar vessels are stationed always in vessel buffers area until new Work container and its similar vessels are read When entering vessel buffers area, old Work container and its similar vessels are likely to be set to scaling method to choose and delete from container buffer area It removes.

Being provided with for the vessel buffers area is conducive to improve reading and writing data performance, because container protects the redundancy of data flow Locality so that in the same container set of metadata of similar data block of data block very likely also in a same vessel, in this way, from disk Last time reads whole container, not only can be to avoid the random small letter I/O of disk, but also buffer area hit rate can be improved, and reduces disk Read and write number.

(202), it receives container: receiving the container of writing that one sends over from the block grade data deduplication storage on upper layer and order It enables, extracts container to be written in container order from writing, be denoted as upper layer container；An empty format is generated in memory to hold Device is denoted as Work container, and vessel buffers area is written in Work container；It is spare to empty similar buffer area；The upper layer container refers to Container used in the block grade data deduplication storage on upper layer；The format container refers to Delta compression storage assembly Container used in container access module.As shown in figure 11, container is by meta-data region and data district's groups at the read-write of meta-data region From top to bottom, the read-write sequence of data field is packaged into appearance from bottom to top, by the meta-data region after finishing writing and data field docking to sequence Device；The meta-data region is made of vessel head, fingerprint region, similar signatures area and hook signature area.The vessel head is by container mark Know symbol field, size field and container signature field composition, be respectively used to store the container identifier of the container, container size and Similar signatures；Fingerprint region block fingerprint for storing data；The similar signatures area is for storing similar signatures block；Such as Figure 12 Shown, the similar signatures block is made of similar signatures field, type field and offset field；The similar signatures field is used for Store the similar signatures of corresponding data block；The type field block mark or Delta block mark for storing data；The offset Field is for storing the address of corresponding data block or Delta block within a data area.The hook signature area is for storing hook label Name；The data field block or Delta block for storing data；The data block attached one when storage is to data field in front A data block head, as shown in figure 13, the data block head are made of data block mark and data block size field.The Delta Block attached a Delta build when storage is to data field in front, and as shown in figure 14, the Delta build is by Delta block Mark, Delta block size field and reference block address field composition, the reference block address field by container identification field with Offset field composition；

(203), copies of metadata:

(204), similarity indexing updates:

In the present embodiment, the method for general<hook, cid>be inserted into similarity indexing is equal into memory Hash table Insertion<key, value>, for the mature prior art.

(206), similar buffer area is write:

In the present embodiment, the method for general<sign, Node>be inserted into similar buffer area is equal to memory Hash table Middle insertion<key, value>, for the mature prior art.

To data block D_rBefore carrying out Delta compression, need to find and D_rThe similar data block of content is as reference block.At this In embodiment, with data block D_rSimilar signatures sign1 be keyword corresponding similar signatures knot is searched in similar buffer area Point then shows not finding data block D if it is not found,_rReference block, go to data block D in (222) step_rIt deposits as former state Storage, if it is found, the then index node storage of linked list of similar signatures node data block D_rAll potential reference blocks Information, then, further find data block D by traversing the index node chained list_rReference block.

Following (210), (21), (212) step operation in, preferential detection data block index node, only when all Data block index node pointed by data block be unsuitable for be used as data block D_rReference block when just further detect Delta Block index node.This processing method can effectively improve Delta compression performance, because data block can be directly used as reference block, And Delta block then needs to be first converted into and just can serve as quoting block after data block, this is related to traversing Delta chain, Delta inverse operation Deng operation, time overhead is larger.

The data block pointed by all data block index nodes is unsuitable for being used as data block D_rReference block when Further detect Delta block index node.Index node is successively detected and handles in the operation of following (213) ~ (221) step Each of chained list Delta block index node, until finding a reference block appropriate to data block D_rCarry out Delta compression Succeed and turns storing data block D in (223) step_rDelta block, or can not find reference block appropriate and turn (222) step It is middle by data block D_rIt stores as former state.

For any Delta block index node in index node chained list, detects and handle and be divided to two processes, first Process is traversal Delta chain, and second process is detection reference block.

(214) ~ (217) step is traversal Delta chain process below, and the process is pointed by Delta block index node Delta block is starting point, data of each Delta block until Delta chain end on the direction of Delta chain reading Delta chain Block.

(218) ~ (220) step is detection reference block process below, which uses the data block of Delta chain end first It is reference block to data block D_rDelta compression is carried out, turns storing data block D in (223) step if compressing successfully_rDelta Otherwise block carries out Delta inverse operation against the direction of Delta chain, be reference block pair with the data block that Delta inverse operation generates Data block D_rDelta compression is carried out, successfully turns storing data block D in (223) step until Delta compresses_rDelta block, or All Delta blocks in Delta chain, which all detect to finish, does not find reference block appropriate yet, at this moment, turn (221) step detection and Handle next Delta block index node in index node chained list.

In the present embodiment, the method for general<sign1, Node1>be inserted into similar buffer area is equal to memory Hash Insertion<key in table, value>, for the mature prior art.

(227), new container is stored:

The container storage pond is stationed on disk unit, and storage container is used for.

The container index is stationed on disk unit, for establishing container identifier and with the appearance of the container identifier Mapping of the device between the position in container storage pond.

(229), terminate:

Then: configuration file is written into the similarity indexing in memory；

As shown in Figure 10, the container recovery algorithms of the container access module in turn include the following steps:

(302), it receives read command: receiving a reading container sended over from the block grade data deduplication storage on upper layer Order is denoted as cid from extraction vessel identifier in container request is read；An empty upper layer format container, note are generated in memory For upper layer container；

Delta block pointed by P1 is reduced into data block by (307) ~ (310) step below, and operation is divided into two mistakes Journey, first process are traversal Delta chain, and second process is Delta chain inverse operation.

(307), (308), (309) step are traversal Delta chain process below, and the process is with Delta block pointed by P1 For starting point, data block of each Delta block until Delta chain end on the direction of Delta chain reading Delta chain.

(310), (311) step carry out the inverse operation of Delta chain below, i.e., carry out the inverse fortune of Delta against the direction of Delta chain It calculates, Delta block pointed by P1 is finally reduced into data block.

In the implementation that said vesse stores algorithm and container recovery algorithms, the Delta operation and Delta inverse operation can To select the Delta tool of compression such as vdelta, xdelta and zdelta, the Delta such as described vdelta, xdelta and zdelta pressure Contracting tool is the mature prior art.

Claims

1. a kind of backup cluster based on two-stage data deduplication, it is characterised in that: including it is multiple interconnect by storing network deposit Storing up node, each memory node the data that send over of subscribing client and can back up data in cluster, or from Restore specified data in cluster；Software systems and disk based on two-stage data deduplication are installed to set on the memory node It is standby；The upper layer i.e. block grade data deduplication that the software systems based on two-stage data deduplication are divided into responsible first order data deduplication is deposited Storage system, the bottom, that is, Delta for being responsible for second level data deduplication compress storage assembly；Data block is installed on the disk unit Subindex, container index and container storage pond；The data block index is by the data deduplication storage management of block grade and use； The container index and container storage pond are by Delta compression storage assembly management and use；

Described piece of grade data deduplication storage includes three data read-write module, fingerprint queries module and container module for reading and writing moulds Block is additionally provided with Bloom filter, fingerprint routing table, container routing table, input block, filebuf, fingerprint buffer area Restore buffer area with data；The data read-write module includes data back up method and data reconstruction method；The fingerprint queries Module includes fingerprint queries order, fingerprint location order, data block index upgrade order and distributed fingerprint querying command；Institute Stating container module for reading and writing includes writing container order, reading container order, read the migration order of container fingerprint order and data；

Delta compression storage assembly is provided with container access module, and the container access module uses similarity indexing, similar Buffer area and vessel buffers area data structure run container storage algorithm and container recovery algorithms, and Delta compresses storage assembly It receives and execute that the block grade data deduplication storage on upper layer sends over writes container order, reads container order or read container and refer to Line order: to writing container order, Delta compresses storage assembly and carries out Delta compression to container, and by the compressed appearance of Delta Device is written in the container storage pond on disk；To container order is read, Delta compresses storage assembly and passes through container index from disk Container storage pond in read specified container, will reading container restore after return to upper layer block grade data deduplication storage system System；To the order of container fingerprint is read, Delta compression storage assembly reads specified containers from the container storage pond on disk and includes These fingerprints are sent to the block grade data deduplication storage on upper layer by fingerprint；

The first order of two-stage data deduplication is started by the memory node of execution data backup, using splits' positions technology, content Similar new data block cluster makes do not have repeated data block in entire cluster to identical memory node；The new data block is Refer to the data block being different from existing all data blocks in cluster；

The second level of two-stage data deduplication is individually completed by each memory node, using Delta compress technique, in this memory node Set of metadata of similar data block carry out Delta compression, eliminate byte level repeated data.

2. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the data back up method It in turn includes the following steps:

22) data in input block, piecemeal and calculating fingerprint: are divided by number based on the block algorithm of content using logical knowledge According to block, the keyed Hash function of logical knowledge is used to calculate the cryptographic Hash of each data block contents as the fingerprint of the data block；

23) similar signatures, are calculated: calculating the similar signatures of each data block, i.e., since the initial position of data block, with one The window of a fixed size slides in data block, as soon as before every sliding byte, calculated using the logical sieve guest's fingerprint algorithm known Sieve guest's fingerprint for falling into the data patch in window takes in all data patch the smallest guest sieve fingerprint as the similar of data block Signature；

24) it, creates file index: file index being established to the file for including in the data of input block, file index is sent out Give the client computer for initiating data backup requests；The fingerprint for the data block that file index is included by file forms, and fingerprint is in text The sequence consensus that the sequence data block corresponding with its occurred in part index occurs hereof；

25) it, is segmented: the data in input block being segmented using the fragmentation technique based on content, i.e., in order defeated Enter the data block that r are 0 after lookup similar signatures in buffer area, data are divided into non-fixed-length data using these data blocks as boundary Section, each data segment include 2^rA data block, wherein r is pre-selected positive integer；

26) it, extracts hook signature: extracting hook signature for each data segment, i.e., according to predetermined hook signature sampling rate All data block similar signatures for including in data segment are sampled, using the similar signatures of extraction as the hook label of data segment Name；Select similar signatures of the smallest similar signatures as data segment in all data block similar signatures for including in data segment；

27) for each data segment, all fingerprints for including in the data segment, data segment fingerprint duplicate removal: are sent to this storage Fingerprint queries module on node, and distributed fingerprint querying command is sent to fingerprint queries module；

28), container encapsulation step: according to the return of fingerprint queries module as a result, successively handling each data segment: abandoning data segment In be not comprised in the data block corresponding with its of the fingerprint in returning the result, if still remaining data block in data segment, these numbers It is new data block according to block, distributes a container for the data segment to store new data block；Take the similar signatures and hook of data segment Signature is respectively the similar signatures and hook signature of container；The similar signatures of container, hook are signed, the fingerprint of new data block, The meta-data region of the similar signatures write-in container of new data block, the data field of new data block write-in container；After processed Data segment is deleted from input block；The container is by meta-data region and data district's groups at the meta-data region is used to store The metadata of container, the metadata of the container include container identifier, the similar signatures of container, hook signature, institute in container The similar signatures and finger print information for the data block for including, the data field are used to storing data block；

29), data clusters: each container is handled as follows: inquiry container routing table, before the similar signatures of the container Sew and find corresponding route entry in container routing table, container is sent to phase according to the memory node address indicated in the route entry The container module for reading and writing for the memory node answered, and container order is write to the transmission of the container module for reading and writing of the memory node；The appearance Device routing table is made of route entry, for establish container identifier prefix or container similar signatures prefix and memory node address it Between mapping, the route entry be binary group<container identifier prefix, memory node address>；Described container identifier prefix etc. In the container similar signatures prefix of same container；

3. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the data reconstruction method It in turn includes the following steps:

31) it, initializes: generating an empty data in memory and restore buffer area and an empty filebuf, setting one A counter Counter is used to record the fingerprint number of processing, and Counter is reset；

32) file index: the file index that subscribing client sends over, is received, a read pointer P is set and is directed toward file First fingerprint in index；

33), buffer area is inquired: being read fingerprint pointed by P, is denoted as fp, the value of Counter is added 1, restores buffer area in data Fingerprint index table in inquire fingerprint fp: if found, from data restore buffer area container chained list in find comprising fingerprint The container of fp, the counter field of chained list node, reads fingerprint from the container where the value of Counter is assigned to the container Data block corresponding to fp, is denoted as D, enters step 38), otherwise, enters step 34)；The data restore buffer area by fingerprint Concordance list and container chained list composition；The fingerprint index table is memory Hash table；The memory Hash table includes a bucket group；Institute A barrel corresponding number each of is stated in barrel group, and establishes the mapping between fingerprint and bucket number using hash function, is mapped to Fingerprint in bucket is stored in the index node of index node chained list；The index node includes fingerprint field, container pointer word Section and chain table pointer field；The fingerprint field stores a fingerprint, and the container pointer field stores the appearance where the fingerprint Address of the device in container chained list, the chain table pointer field store next index node in the same index node chained list Address；The container chained list is the logical memory chained list known, and the container that write-in data restore buffer area is linked at the memory chained list In；The memory chained list is made of a head pointer and multiple chained list nodes linked together, and chained list node includes a meter Number device field and a container；

34), fingerprint location: inquiry fingerprint routing table finds corresponding route entry in fingerprint routing table according to the prefix of fingerprint fp, Fingerprint fp is sent to the fingerprint queries module of corresponding memory node according to the memory node address indicated in the route entry, and Fingerprint location order is sent to the fingerprint queries module of the memory node；The fingerprint routing table is made of route entry, for building Mapping between vertical fingerprint prefix and memory node address, the route entry are binary group < fingerprint prefix, memory node address >；

36), read container: inquiry container routing table finds corresponding route entry in container routing table according to the prefix of cid, according to Cid is sent to the container module for reading and writing of corresponding memory node by the memory node address indicated in the route entry, and is deposited to this The container module for reading and writing for storing up node, which is sent, reads container order；The container identifier is one M+N+S binary numbers, preceding M Position is the full prefix of container identifier, is preceding M of the container similar signatures, and intermediate N is number, is storage where the container The number of node, it is serial number of the container on memory node that last S, which is serial number,；The container identifier prefix refers to this The m(m of the full prefix of container identifier is the integer for being less than or equal to M more than or equal to 1) position prefix；

37), writing buffer: receiving and read the container that container order returns, and container write-in data is restored buffer area, from the container Data block corresponding to middle reading fingerprint fp, is denoted as D；

38), restore file data: filebuf is written into data block D；If filebuf is full, from wherein removing one Partial data, and the data of removal are sent to client computer；

39), file index judges: read pointer P moves forward a step, is directed toward next fingerprint of file index, if P non-empty, It then goes to step 33)；Otherwise, the remaining data in filebuf is removed and is sent to client computer, and send text to client computer 391) number of packages is entered step according to end signal is restored；

391), terminate judgment step: if the data for not receiving client computer restore ending request, entering step 32)；It is no Then, it enters step 393)；

392), error handling processing: sending file index error signal to client computer, and malfunction reason are as follows: can not find out fingerprint fp in system；

393), terminate: deleting data and restore buffer area, filebuf, counter Counter data structure backed off after random.

4. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the fingerprint queries order It in turn includes the following steps:

42), filter is inquired: fp is inquired in Bloom filter, if do not found, to the storage section of request fingerprint queries Terminate after the information of point return " fp is new fingerprint "；Otherwise, it goes to step 43)；The Bloom filter is the logical data query known Structure, all fingerprints in data block index for indicating this memory node in memory；

43), data block search index: the data block index is the logical disk Hash table known, and the disk Hash table makes Mapped fingerprints in corresponding bucket with hash function, in the bucket store binary group<fp, cid>；It is indexed in data block Middle inquiry fp, if found, the container identifier of container, is denoted as cid where obtaining fingerprint fp, to depositing for request fingerprint queries Terminate after storing up the information of node return " fp is old fingerprint, is included in container cid "；Otherwise, to the storage of request fingerprint queries Terminate after the information of node return " fp is new fingerprint ".

5. as claimed in claim 3 based on the backup cluster of two-stage data deduplication, it is characterised in that: the fingerprint location order It in turn includes the following steps:

52), data block search index: inquiring fp in data block index, if found, obtains container where fingerprint fp Container identifier, be denoted as cid, to request fingerprint location memory node return container identifier " cid " after terminate；Otherwise, Terminate after returning to negative " -1 " to the memory node of request fingerprint location.

6. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the data block index The execution method of more newer command are as follows:

61) it, extracts binary group: extracting binary group<fp, cid>wherein from data block index upgrade order, fp refers to Line, cid are the container identifiers of container where fp；

7. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the distributed fingerprint is looked into Order is ask in turn include the following steps:

71), receiving data segment fingerprint: the data segment fingerprint that the data read-write module of this memory node sends over is received, is denoted as Fingerprint collection is arranged a read pointer P and is directed toward first fingerprint that fingerprint is concentrated；

72), buffer area is inquired: fingerprint pointed by P is read, fp is denoted as, inquires fp in fingerprint buffer area, if found, It enters step 77)；Otherwise, it enters step 73)；The fingerprint buffer area is the logical memory Hash table known, the memory Hash table It is mapped fingerprints in corresponding bucket using hash function, stores fingerprint in the bucket；When the fingerprint buffer area is full, using logical The least recently used replacement algorithm known deletes some fingerprints；

73), fingerprint queries: inquiry fingerprint routing table finds corresponding route entry in fingerprint routing table according to the prefix of fingerprint fp, Fingerprint fp is sent to the fingerprint queries module of corresponding memory node according to the memory node address indicated in the route entry, and Fingerprint queries order is sent to the fingerprint queries module of the memory node；

74), query result judges: receiving the query result of fingerprint fp, if fp is new fingerprint, fp is inserted into fingerprint buffer area In, turn the 78) step, otherwise, fp is old fingerprint, and the container identifier of container where obtaining fingerprint fp, is denoted as cid, turns next Step；

75), read container fingerprint: inquiry container routing table finds corresponding route entry in container routing table according to the prefix of cid, Cid is sent to the container module for reading and writing of corresponding memory node according to the memory node address indicated in the route entry, and to The container module for reading and writing of the memory node, which is sent, reads the order of container fingerprint；

76), buffer area updates: receiving after reading the fingerprint that the order of container fingerprint returns, these fingerprints are inserted into fingerprint buffer area In；

78), terminate judgement: read pointer P moves forward a step, next fingerprint that fingerprint is concentrated is directed toward, if P non-empty, turns The 72) step otherwise turn in next step；

79), terminate: if fingerprint concentrates still Yu Zhiwen, the remaining fingerprint that fingerprint is concentrated being returned to the number of this memory node According to module for reading and writing backed off after random, otherwise, the data read-write module backed off after random for returning to this memory node for 0.

8. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: described to write container order Execution method in turn includes the following steps:

81) it, receives container: reading container from writing in container order, be denoted as Container, the value of container counter is added 1；Institute It states container counter to be safeguarded by container module for reading and writing, writes container order for recording container module for reading and writing execution this storage is written Number of containers on node；

82) container identifier, is generated:

It is last: to generate one M+N+S container identifiers for Container, be denoted as cid, Container is written into cid Meta-data region；

83), write container: Delta Container being sent on this memory node compresses storage assembly, and is sent to it and writes Container order；

84), data block index upgrade: the data block index is the logical distributed hashtable known, and is saved by being distributed in each storage Data block on point indexes composition, and the fingerprint for including in these data blocks index is all different, in entire backup cluster There is no duplicate fingerprint；For each fingerprint fp for including in Container, a binary group<fp is generated, cid>, it looks into Fingerprint routing table is ask, corresponding route entry in fingerprint routing table is found according to the prefix of fingerprint fp, according to being indicated in the route entry Memory node address general<fp, cid>be sent to the fingerprint queries module of corresponding memory node, and to the memory node Fingerprint queries module send data block index upgrade order.

9. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the reading container order Execution method are as follows:

It is last: to receive the container that the Delta compression storage assembly of this memory node returns, and the container is returned into request and reads to hold The memory node of device.

10. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the reading container fingerprint The execution method of order are as follows:

Secondly: the cid Delta for being sent to this memory node being compressed into storage assembly, and is sent to it and reads the order of container fingerprint；

It is last: to receive the container fingerprint that the Delta compression storage assembly of this memory node returns, and these fingerprints are returned to and are asked Seek the memory node for reading container fingerprint.

11. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the Data Migration life The execution of order in turn includes the following steps:

91), subindex migrates: all binary groups in the data block index of this memory node is read, for each of reading Binary group<fp, cid>, if the kth of fingerprint fp+1 is 0, general<fp, cid>it is sent to the new storage that address is addr2 The fingerprint queries module of node, and it is sent to it data block index upgrade order；If the kth of fingerprint fp+1 is 1, will <fp, cid>it is sent to the fingerprint queries module for the new memory node that address is addr3, and it is sent to it data block index more Newer command；

92) it, redirects: fingerprint queries order, fingerprint location order, the data block index upgrade that this memory node is received Order is redirected to new memory node, that is, kth+1 for detecting fingerprint, and if it is 0, corresponding order, which is transmitted to address, is The new memory node of addr2 executes；If it is 1, corresponding order is transmitted to the new memory node that address is addr3 and is held Row；The container order of writing that this memory node is received is redirected to new memory node, the i.e. w+1 of detection container similar signatures Position is transmitted to the new memory node execution that address is addr2 container order is write if it is 0；If it is 1, writing container Order is transmitted to the new memory node that address is addr3 and executes；Reading container, the reading container fingerprint received to this memory node is ordered Order redirects, i.e. the number of detection container identifier, if number is num1, executes order by this memory node, such as Fruit number is num2, then corresponding order is transmitted to the new memory node that address is addr2 and executed；If number is num3, Corresponding order is then transmitted to the new memory node that address is addr3 to execute；

93), container migrates: sending to the Delta of this memory node compression storage assembly and reads container order, reads the institute of its storage There is container, for each container of reading, w+1 of detection container similar signatures, if it is 0, which is sent to Address is the new memory node of addr2, and is sent to it and writes container order；If it is 1, which, which is sent to address, is The new memory node of addr3, and be sent to it and write container order；

94), routing update:

First: the fingerprint routing table of this memory node and container routing table being sent to new memory node, as new memory node Fingerprint routing table and container routing table；

Then: all memory nodes for including into backup cluster include that new memory node broadcast updates, and fingerprint is routed Route entry < a in table₁a₂…a_k, addr1>deletion, and increase new route entry<a₁a₂…a_k0, addr2>and<a₁a₂…a_k1, addr3>；By route entry <b in container routing table₁b₂…b_w, addr1>deletion, and increase new route entry<b₁b₂…b_w0, Addr2>with<b₁b₂…b_w1, addr3>；

Terminate: this memory node stops the data backup and resume request of subscribing client, the existing reading container of this memory node The order of container fingerprint is read in order, and data backup and data resume operation the backed off after random that is finished.

12. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the container storage is calculated Method in turn includes the following steps:

(201), it initializes:

First: parameter S, R and L are read from configuration file；The configuration file is stationed on disk unit, for recording system Configuration information；

The parameter S is preset positive integer, when indicating to carry out Delta compression out of in the container storage pond of disk read in The maximum number for the similar vessels deposited, the similar vessels refer to the similar container of content；

The parameter R is preset positive number, indicates the minimum Delta compression ratio allowed, the Delta compression ratio is to instigate After carrying out Delta compression generation Delta block to data block with reference block, the ratio of data block size and Delta block size；

Then: judging whether to be system configuration initial stage, then generate an empty similarity indexing in memory in this way；If not, from matching It sets and the similarity indexing of backup is read in into memory on file；

It is last: to generate an empty similar buffer area in memory；An empty vessel buffers area is generated in memory, is used to temporary It deposits from the container read in the container storage pond on disk in memory；An empty storehouse is generated in memory, is denoted as Stack；

(202), it receives container: receiving one and write container order from what the block grade data deduplication storage on upper layer sended over, Container to be written is extracted in container order from writing, and is denoted as upper layer container；An empty format container, note are generated in memory For Work container, vessel buffers area is written into Work container；It is spare to empty similar buffer area；The upper layer container refers to upper layer Container used in block grade data deduplication storage；The format container refers to that the container of Delta compression storage assembly is deposited Container used in modulus block；

(203), copies of metadata:

First: reading the similar signatures of container identifier and container from the meta-data region of upper layer container, be respectively written into Work container Vessel head container identification field and container signature field；

Secondly: from the meta-data region read block fingerprint of upper layer container, by these data block fingerprints according to its original sequence It is sequentially written in the fingerprint region of Work container；

Again: being successively read similar signatures from the meta-data region of upper layer container, generate one for each similar signatures read Similar signatures block, and by the similar signatures field of similar signatures write-in similar signatures block, originally according to corresponding similar signatures Similar signatures block is sequentially written in the similar signatures area of Work container by sequence；

It is last: to read hook signature from the meta-data region of upper layer container, these hooks are signed according to its original sequence successively The hook signature area of Work container is written；

(204), similarity indexing updates:

Secondly: signing, be handled as follows: by the hook label for each hook for including in Work container hook signature area Name is assigned to variable hook, generates a mapping<hook, and cid>, general<hook, cid>be inserted into similarity indexing；

(205), similar vessels are searched: inquiry similarity indexing, are found out and are shared those of hook signature container with Work container, if It does not find, then goes to step (226)；Otherwise, according to sharing the quantity of hook signature with Work container from big to small from finding Most S containers are chosen in container, confirm that these containers being selected are the similar vessels of Work container；In vessel buffers area These similar vessels are searched, the similar vessels not in vessel buffers area are read into vessel buffers area from container storage pool；

(206), similar buffer area is write:

The similar signatures area for successively scanning each similar vessels of Work container in vessel buffers area, reads in similar signatures area All similar signatures blocks, each similar signatures block of reading is handled as follows: in memory generate a similar buffering Area's index node, is denoted as Node；The type field value of the similar signatures block and offset word segment value are individually copied to the class of Node Type-word section and offset field；By the container identification field of the container identifier write-in Node of the similar signatures block said container； The similar signatures field for reading the similar signatures block remembers that the similar signatures of reading are sign, general<sign, Node>be inserted into phase Like buffer area；

(207), prepare processing data block: one read pointer P1 of setting is directed toward first similar label in Work container similar signatures area First data block that a read pointer P2 is directed toward upper layer container data area is arranged in name block (NAM)；

(208), read block: data block pointed by a P2 is read from the container data area of upper layer, is denoted as D_r, from work Similar signatures block pointed by a P1 is read in container similar signatures area, is denoted as Block, is read the similar signatures word of Block The value of section, is denoted as sign1；

(209), similar buffer area is searched: the similar signatures knot that similar signatures field value is sign1 is searched in similar buffer area Point goes to step (222) if it is not found,；Otherwise, the rope that a read pointer P3 is directed toward the similar signatures node just found is set First index node for drawing node chained list, goes to step (210)；

(210), judge data block: if the type field value for the index node that P3 is directed toward for Delta block mark, turns (212) Step；Otherwise, it is data block mark, goes to step (211)；

(211), short chain Delta operation: by the value of the container identification field of index node pointed by P3 and offset field Value is assigned to variable cid0 and offset0 respectively, and from address<cid0 in container buffer area, offset0>place reads one Data block is denoted as D₀；With D₀For reference block and D_rDelta operation is carried out, Delta block △ is generated_0,r；If Delta compression ratio More than or equal to R, then success is compressed, goes to step (223)；Otherwise, compression failure, goes to step (212)；

(212), skip Delta block: P3 moves forward a step, is directed toward next index node, if P3 non-empty, turns (210) Step, otherwise, shows the tail portion for having arrived at index node chained list, and P3 is directed toward to first index of the index node chained list again Node goes to step (213)；

(213), judge Delta block: if the type field value for the index node that P3 is directed toward for data block mark, turns (221) Step, otherwise, is Delta block mark, goes to step (214)；

(214), Delta block: the value of the container identification field of index node pointed by P3 and the value of offset field is read Address of the Delta block in vessel buffers area is designated, a Delta block is read from the address；

(215), it is pressed into storehouse: the Delta block is pressed into Stack, reference block is read from the Delta build of the Delta block The reference block address of reading is stored in variable<cid1 by location, and offset1>in, wherein cid1 is container identifier, offset1 It is the position of the reference block in the data field of container cid1；

(216), read reference block: if container cid1 in vessel buffers area, from container buffer area read reference block < Cid1, offset1 >, otherwise, container cid1 is read in into vessel buffers area from container storage pool first, then reads reference block again < cid1, offset1>；

(217), judgement reference block: if reference block<cid1, offset1>it is Delta block, otherwise turning (215) step is The content of the data block is stored in variables D by data block₀In, general<cid1, offset1>it is assigned to variable<cid0, Offset0 >, it is assigned to variable length by 1, is turned in next step；

(218), long-chain Delta calculation step: with D₀For reference block and D_rIt carries out Delta operation and generates Delta block △_0,r；Such as Fruit Delta compression ratio is greater than or equal to R and length is less than or equal to L, then compresses success, go to step (223), otherwise, pressure Contracting failure, turns in next step；

(220), pop-up a stack: popping up a Delta block from Stack, be denoted as △, and the address of △ is stored in variable < cid0, Offset0 >, to D₀Delta inverse operation is carried out with △, the result of Delta inverse operation is stored in variables D₀In, by variable length Value increase by 1, go to step (218)；

(221), skip data block: P3 moves forward a step, is directed toward next index node, if P3 non-empty, turns (213) Step, otherwise, shows the tail portion for having arrived at index node chained list, turns in next step；

(222), storing data block: in data block D_rBefore add a data block head, the writing data blocks in the data block head Mark and D_rSize information；If the data field non-empty of Work container, by the data block D after additional data build_rIt adds In the data field of Work container behind data with existing, otherwise, by the data block D after additional data build_rWork container is written Data field initial position；By the type field of data block mark write-in Block, by data block D_rIn Work container data field Location information write-in Block offset field, go to step (224)；

(223), Delta block is stored: in Delta block △_0,rBefore add a Delta build, write in the Delta build Enter Delta block mark, △_0,rSize and △_0,rReference block address<cid0, offset0>；If Work container Data field non-empty, then by the Delta block △ after additional Delta build_0,rThe additional data with existing in the data field of Work container Below, otherwise, by the Delta block △ after additional Delta build_0,rThe initial position of the data field of Work container is written；It will The type field of Block is written in Delta block mark, by Delta block △_0,rLocation information in Work container data field is written The offset field of Block；Storehouse Stack is emptied；

(224), similar buffer area updates: generating a similar buffer area index node in memory, is denoted as Node1；By Block Type field value and offset word segment value be individually copied to the type field and offset field of Node1；By the container of Work container The container identification field of identifier write-in Node1；General<sign1, Node1>it is inserted into similar buffer area；

(225), data block is disposed judgement: P1 being moved forward a step, under being directed toward in the similar signatures area of Work container P2 is moved forward a step, the next data block being directed toward in upper layer container data area by one similar signatures block；If P2 is Sky then shows that the data block in the container of upper layer is all disposed, goes to step (227)；Otherwise, (208) are gone to step；

(226), former container is stored: first: from first data BOB(beginning of block), successively handling each in upper layer container data area Data block adds data block head before the data block, the size of writing data blocks mark and the data block in data block head Information；If the data field non-empty of Work container, by the additional data in Work container of data block after additional data build In area behind data with existing, otherwise, by the start bit of the data field of the data block write-in Work container after additional data build It sets；Similar signatures block corresponding with the data block in the similar signatures area of Work container is handled, data block mark is write The similar signatures block is written in the type field for entering the similar signatures block, the location information by the data block in Work container data field Offset field；

Secondly: abandoning upper layer container；It calculates the size of Work container and size information is written to the vessel head of Work container；If Work container is then added and behind data with existing, otherwise, Work container is write in container storage pond by container storage pond non-empty Enter the initial position in container storage pond；

It is last: by the container identifier of the Work container in rigid write-in container storage pond and the Work container in container storage pond Location information be written container index, go to step (228)；

(227), new container is stored:

First: abandoning upper layer container；It calculates the size of Work container and size information is written to the vessel head of Work container；If Work container is then added and behind data with existing, otherwise, Work container is write in container storage pond by container storage pond non-empty Enter the initial position in container storage pond；

Secondly: by the container identifier of the Work container in rigid write-in container storage pond and the Work container in container storage pond Location information be written container index；

(228), end of run judges: judging whether to receive end of run instruction, such as otherwise goes to step (202)；If so, then walking Suddenly (229)；

(229), terminate:

Then: configuration file is written into the similarity indexing in memory；

13. as described in claim 1 based on the backup cluster of two-stage data deduplication, it is characterised in that: the container restores to calculate Method in turn includes the following steps:

(301), it initializes: generating an empty vessel buffers area in memory, for temporary from the container storage pond on disk In read in memory in container；An empty storehouse is generated in memory, is denoted as Stack；

(302), it receives read command: receiving the reading container sended over from the block grade data deduplication storage on upper layer a life It enables, from extraction vessel identifier in container order is read, is denoted as cid；An empty upper layer format container is generated in memory, is denoted as Upper layer container；

(303), it reads container: reading the container that container identifier is cid in the container storage pond on disk, be denoted as work appearance Vessel buffers area is written in Work container by device；

(304), metadata recovering step: according to the type and call format of upper layer container metadata from the metadata of Work container Area reads corresponding metadata, by the meta-data region of these metadata write-in upper layer container of reading；

(306), judge object: if object pointed by P1 is a data block, which being denoted as D_r, go to step (312)；Otherwise, it is Delta block, turns in next step；

(307), it is pressed into storehouse: the Delta block is pressed into Stack, reference block is read from the Delta build of the Delta block Location, remember reading reference block address be<cid1, offset1>, wherein cid1 is container identifier, and offset1 is the reference Position of the block in the data field of container cid1；

(308), read reference block: if container cid1 in vessel buffers area, from container buffer area read reference block < Cid1, offset1 >, otherwise, container cid1 is read in into vessel buffers area from container storage pool first, then reads reference block again < cid1, offset1>；

(309), judgement reference block step: if reference block<cid1, offset1>it is Delta block, go to step (307)；It is no Then, it is data block, which is stored in variables D, turns in next step；

(310), pop-up a stack: popping up a Delta block from Stack, be denoted as △, carries out Delta inverse operation to D and △, will In the result deposit variables D of Delta inverse operation；

(312), data block is copied: if upper layer container data area non-empty, by data block D_rIt adds in upper layer container data area In behind data with existing；Otherwise, by data block D_rThe initial position in upper layer container data area is written；

(313), judge data field: read pointer P1 moves forward a step, the next object being directed toward in Work container data field, such as Fruit P1 non-empty, goes to step (306)；Otherwise, data field is disposed, and turns in next step；

(314), end of run judges: the upper layer container handled well being sent to the block grade data deduplication storage on upper layer, such as Fruit is not received by end operation order, then goes to step (302)；Otherwise, turn in next step；