TWI441034B - Processing method for duplicate data - Google Patents

Description

Duplicate data processing method

The present invention relates to a method for processing repeated data, and more particularly to a method for processing fast repeated data.

Deduplication is a data reduction technique commonly used in disk-based backup systems with the primary goal of reducing the storage capacity used in storage systems. It works by finding duplicate variable-size blocks of data in different locations in a different time period. Duplicate data blocks are replaced with indicators. Because the storage system is always full of redundant data. In order to solve this problem and save more space, the technology of deduplication has become the focus of attention. The deduplication technology can reduce the stored data to 1/20 of the original, which allows more backup space, not only can save the backup data on the storage system for a longer period of time, but also save the offline storage. A large amount of bandwidth required.

Since the data to be stored is stored in the server, the client needs to transfer the data to the server from time to time. Then, the server performs deduplication processing on the data data. However, in order to judge whether the data to be backed up by the client has been stored in the disk of the server, the relevant entire data block needs to be loaded into the memory, and then the part of the metadata (data) in the entire data block is read. However, regardless of whether the data block is duplicate data, the raw data in the data block loaded into the memory is not used.

The present invention provides a method of processing repeated data. The processing method of the repeated data first divides a storage file into a plurality of raw data blocks and a plurality of meta tanks, wherein the original data blocks and the metadata blocks have a one-to-one correspondence, and each element A stored fingerprint value of the corresponding original data block is stored in the data block. A decision repeat data request is then received, wherein the determine duplicate data request includes a request fingerprint value. The processing method of the duplicate data reads at least one metadata block and compares the requested fingerprint value with the stored fingerprint value of the read metadata block. When the requested fingerprint value is the same as the stored fingerprint value of the read metadata block, a referenced counter of the read metadata block is modified, and the modified metadata block is restored.

Each of the original data blocks may include a plurality of original data chunks, each of the metadata chunks may include a plurality of metadata chunks, and the original data unit and the metadata unit have a one-to-one correspondence.

According to an embodiment, the stored file may be stored on a disk of a server, and the modified metadata block may be restored to the disk.

In the step of "reading at least one metadata block and comparing the requested fingerprint value with the stored fingerprint value of the read metadata block", it may first determine whether a metadata block corresponding to the requested fingerprint value exists in a memory. When the metadata block corresponding to the requested fingerprint value exists in the memory, the metadata block corresponding to the requested fingerprint value in the memory is read, and the stored fingerprint value of the requested fingerprint value and the read metadata block is compared.

In the step of "reading at least one metadata block and comparing the requested fingerprint value with the stored fingerprint value of the read metadata block", when the metadata block corresponding to the requested fingerprint value does not exist in the memory, the magnetic The disc reads the metadata block corresponding to the requested fingerprint value into the memory, and compares the requested fingerprint value with the stored fingerprint value of the read metadata block.

According to another embodiment, a plurality of metadata blocks can be used as an allocation group. "Reading at least one metadata block and comparing the requested fingerprint value with the stored fingerprint value of the read metadata block" may first read at least one allocation group, and then compare the requested fingerprint value with the read The stored fingerprint value of the metadata block in the allocation group. The data size of the allocation group may be a positive integer multiple of 16 kilobytes (KB).

In addition, in the step of "changing the reference count value of the read metadata block and returning the modified metadata block", the reference count value of the read metadata block can be modified, and the modified metadata is modified. The allocation group corresponding to the block is saved.

The detailed features and advantages of the present invention are set forth in the Detailed Description of the Detailed Description of the <RTIgt; </ RTI> <RTIgt; </ RTI> </ RTI> </ RTI> <RTIgt; The objects and advantages associated with the present invention can be readily understood by those skilled in the art.

The present invention relates to a method of processing repeated data, which is applicable to a server. A server that implements a method of processing duplicate data can be used to back up data of at least one client, and the server can have a function of deduplication.

Please refer to FIG. 1 , which is a schematic diagram of a server according to an embodiment of the present invention. The server 20 can connect to a plurality of clients 10 through various networks such as the Internet or the Internet, and back up the data transmitted by the client 10. The server 20 can have a memory 30, a disk 40, and a hardware such as a processor. The disk 40 stores a plurality of storage files 50 that have been completely received from the client 10; and when the file 50 is processed, the data to be processed can be loaded from the disk 40 into the memory 30 for processing.

The storage file 50 can be, for example, a plain text file, various multimedia files, or a snapshot generated when the client 10 performs a system backup. The method of processing the duplicate data can quickly determine whether the file to be backed up is one of the storage files 50 of the server 20 when the client 10 wants to perform backup.

Please refer to FIG. 2, which is a flowchart of a method for processing repeated data according to an embodiment of the present invention.

First, the server 20 may divide the storage file 50 into a plurality of meta tanks 60A, 60B, and 60C in advance, and a plurality of raw tanks 70A, 70B, and 70C (step S100). In addition, the following is collectively referred to as a metadata block 60 and an original data block 70. The original data block 70 and the metadata block 60 have a one-to-one correspondence. A stored fingerprint value of the corresponding original data block 70 is stored in each of the metadata blocks 60. For example, metadata blocks 60A, 60B, and 60C may be individually corresponding to original data blocks 70A, 70B, and 70C; and stored fingerprint values of original data block 70A are stored in metadata block 60A.

The server 20 may first divide the entire storage file 50 into a plurality of sets of data blocks by using a fixed-size partition or a content-defined chunking (CDC) method, wherein each set of data blocks A corresponding one of the metadata block 60 and the original data block 70 are included. The paired metadata block 60 and the original data block 70 are then separately stored for subsequent reading.

However, the server 20 may first divide the content portion of the storage file 50 into the original data block 70, and then obtain the corresponding metadata block 60 according to the divided original data block 70. The server 20 can calculate the stored fingerprint value of each original data block 70 through an algorithm such as MD5, SHA-1, SHA-256, SHA-512 or One-way Hash, and store the fingerprints. The value is stored in the corresponding metadata block 60.

In an embodiment, each of the original data blocks 70 may include a plurality of raw data chunks 72, as shown in FIG. In contrast, each metadata block 60 can include a plurality of meta chunks 62, and these raw data units 72 and metadata units 62 also have a one-to-one correspondence. For example, the six metadata units 62 in the metadata block 60A correspond one-to-one to the six original data units 72 in the original data block 70A.

Taking a fixed length partitioning method as an example, assuming that each original data unit 72 has a length of 64 kilobytes (KB), the corresponding metadata unit 62 will have a length of 28 bytes, and each fixed The original data block 70 has a length of 2 million bytes (MB). Then, it can be calculated that 32 blocks are included in each data block, and the length of the metadata block 60 is 896 KB.

The server 20 can then receive a duplicate data determination request from any of the clients 10, wherein the duplicate data determination request includes a request fingerprint value (step S120). In order to minimize the data transmission between the client 10 and the server 20, the client 10 may only transmit a request fingerprint value representing the requested data block to the server 20 when pre-backing up a request data block. However, the algorithm for calculating the requested fingerprint value based on the request data block must be the same as the algorithm for calculating the stored fingerprint value based on the original data block 70.

After receiving the duplicate data determination request, the server 20 reads at least one metadata block 60 of the storage file 50 corresponding to the duplicate data determination request, and compares the requested fingerprint value with the stored fingerprint value of the read metadata block 60 (step S130).

Please refer to FIG. 4 at the same time, which is a flowchart of step S130 according to an embodiment of the present invention.

The server 20 first judges whether or not the metadata block 60 corresponding to the requested fingerprint value exists in the memory 30 (step S132). When the metadata block 60 corresponding to the requested fingerprint value is already present in the memory 30, the server 20 can directly read the metadata block 60 corresponding to the requested fingerprint value in the memory 30 (step S134), and compare the requested fingerprint. The value and the stored fingerprint value of the read metadata block 60 (step S136). Conversely, if the metadata block 60 corresponding to the requested fingerprint value is not found after searching in the memory 30, the metadata block 60 corresponding to the requested fingerprint value is first read into the memory 30 by the disk 40 (step S138). ), step S134 and step S136 are performed.

For example, assume that metadata block 60C corresponds to a request fingerprint value. Since the metadata block 60C is not found in the memory 30, the metadata block 60C is first loaded into the memory 30. It should be noted that it is only necessary to load the metadata block 60C having a very small data length in step S138 without loading the corresponding original data block 70C. In the period in which the metadata block 60C has not been swapped out of the memory 30 due to insufficient space of the memory 30, if the requested fingerprint value of the corresponding metadata block 60C is received again, the server 20 can directly execute steps S134 and S136. .

The server 20 then determines whether the requested fingerprint value is the same as the stored fingerprint value of the read metadata block 60 (step S140). When the requested fingerprint value is the same as the stored fingerprint value of the read metadata block 60, it indicates that the requested fingerprint value corresponds to an existing original data block 70, and thus the requested data block represented by the requested fingerprint value is duplicate data. For duplicate data, the server 20 only has to modify a referred counter of the read metadata block 60 and restore the modified metadata block 60 (step S150). The reference count value represents the number of times the corresponding original data block 70 is referenced; the same original data block 70 may be referenced multiple times by the same client 10, or may be simultaneously referenced by different clients 10. The modified metadata block 60 is stored back to the disk 40 by the memory 30.

If the storage file 50 or the metadata block 60 corresponding to the requested fingerprint value is not found in step S130, it indicates that the requested data block corresponding to the requested fingerprint value is a new data block instead of the duplicate data. Then, the server 20 can execute a new program, requesting the client 10 to transmit the request data block, and add the received request data block to the disk 40.

When the requested fingerprint value is different from the stored fingerprint value of the read metadata block 60, it indicates that a hash collision occurs. The server 20 can re-search the memory 30 or the disk 40 to search for the stored file 50 or the metadata block 60 corresponding to the requested fingerprint value by searching for a hash conflict table or the like, and determine whether to execute the storage file 50 or the metadata block 60 corresponding to the requested fingerprint value. Steps S130 to S150 are counted.

It should be noted that although "FIG. 2" is an example of finding a request fingerprint value in the metadata block 60 of a single storage file 50, the method of processing the duplicate data may also search among the plurality of storage files 50. A metadata block 60 corresponding to the fingerprint value is requested.

In addition, the server 20 can further use a plurality of consecutive metadata blocks 60 as an allocation group, and use the allocation group as a unit of the access metadata block 60. Please refer to FIG. 5, which is a schematic diagram of an allocation group according to an embodiment of the present invention.

The server 20 can treat the plurality of consecutive metadata blocks 60 as an allocation group 64, and load the entire allocation group 64 into the memory 30 at a time, and then read at least one of the metadata blocks 60. Then, in step S130, the server 20 may first read at least one allocation group 64; and then compare the requested fingerprint value with the stored fingerprint value of the metadata block 60 in the read allocation group 64. When the memory is restored in step S150, the allocation group 64 corresponding to the entire modified metadata block 60 may be used at one time.

The data size of the allocation group 64 may be a positive integer multiple of 16 KB, such as 16 KB, 64 KB, or 128 KB. The data size of the allocation group 64 can be selected in conjunction with the disk sector. Taking the server 20 with a disk sector of 64 KB as an example, the unit of single input/output (IO) is 64 KB. Therefore, the plurality of metadata blocks 60 are mashed into an allocation group 64 in accordance with the setting of the disk sector, and the maximum number of metadata blocks 60 can be read and written in the primary disk IO. In this way, it is avoided that 64 KB of data is read and written each time for a single metadata block 60 of only 3 k in size.

In summary, the method for processing the repeated data only needs to load the corresponding metadata block into the memory when determining whether the requested data block corresponding to the requested fingerprint value already exists. In this way, the server does not need to load the entire associated metadata block and the original data block into the memory, thereby saving a large amount of disk IO time. In the numerical value of the foregoing embodiment, the processing method of the repeated data only needs to load a metadata block of about 3 KB to perform discrimination, and it is not necessary to load the unused 2 MB original data block into the disk.

Further, the method of processing the repeated data can allocate a group in accordance with the setting of the disk sector, and access a plurality of consecutive metadata blocks at a time. Therefore, for a service such as a full backup with a large number of files or data blocks, since the continuity of the request data block is high, the practice of loading multiple consecutive metadata blocks at one time can improve the hit rate of the search and further reduce The number of disk IOs required. Moreover, since the data length of the metadata block is small, the processing method of the duplicate data may also be provided by the server to store the fingerprint value to be compared to the client, and the steps of comparing and removing the duplicate data by each client.

The above detailed description of the preferred embodiments of the present invention is intended to provide a further understanding of the scope of the invention. On the contrary, the intention is to cover various modifications and equivalent arrangements within the scope of the invention as claimed.

10. . . Client

20. . . server

30. . . Memory

40. . . Disk

50. . . Save file

60, 60A, 60B, 60C. . . Metablock

62. . . Metadata unit

64. . . Assignment group

70, 70A, 70B, 70C. . . Raw data block

72. . . Raw data unit

Figure 1 is a schematic diagram of a server in accordance with an embodiment of the present invention.

2 is a flow chart of a method of processing repeated data according to an embodiment of the present invention.

Figure 3 is a schematic diagram of a metadata unit and an original data unit in accordance with an embodiment of the present invention.

Figure 4 is a flow chart of step S130 in accordance with an embodiment of the present invention.

Figure 5 is a schematic diagram of an allocation group in accordance with an embodiment of the present invention.

Claims (8)

A method for processing repeated data, comprising: dividing a storage file into a plurality of original data blocks and a plurality of metadata blocks, wherein the original data blocks and the metadata blocks are one-to-one correspondence, and each of the elements Storing a stored fingerprint value of the original data block; receiving a duplicate data determination request, wherein the duplicate data determination request includes a request fingerprint value; reading at least one of the metadata blocks, and comparing the request a fingerprint value and the stored fingerprint value of the read metadata block; and modifying a reference to the read metadata block when the requested fingerprint value is the same as the stored fingerprint value of the read metadata block The value is returned to the modified metadata block. The processing method of the duplicate data according to Item 1, wherein each of the original data blocks includes a plurality of original data units, each of the metadata blocks includes a plurality of metadata units, and the original data units and the Some metadata units have a one-to-one correspondence. The method for processing duplicate data according to claim 1, wherein the stored file is stored in a disk of a server, and the modified metadata block is stored in the disk. The method for processing duplicate data according to claim 3, wherein the step of reading at least one of the metadata blocks and comparing the stored fingerprint value of the requested fingerprint value with the read metadata block comprises: Determining whether the metadata block corresponding to the requested fingerprint value exists in a memory; and when the metadata block corresponding to the requested fingerprint value exists in the memory, reading the memory corresponding to the request The metadata block of the fingerprint value and compared to the stored fingerprint value of the requested fingerprint value and the read metadata block. The method for processing duplicate data according to claim 4, wherein the step of reading at least one of the metadata blocks and comparing the stored fingerprint value to the requested fingerprint value and the read metadata block comprises: When the metadata block corresponding to the requested fingerprint value does not exist in the memory, the metadata block corresponding to the requested fingerprint value is read into the memory by the disk, and the requested fingerprint value is compared. The stored fingerprint value of the metadata block with the read. The method for processing duplicate data according to claim 1, wherein a plurality of the metadata blocks are used as an allocation group, and the at least one metadata block is read, and the fingerprint value and the read fingerprint value are compared. The step of storing the fingerprint value of the metadata block includes: reading at least one of the allocation groups; and comparing the stored fingerprint value to the requested fingerprint value and the read metadata blocks in the allocation group. The method for processing duplicate data as described in claim 6, wherein the data size of the allocation groups is a positive integer multiple of 16 kilobytes. The method for processing the repeated data according to Item 6, wherein the step of modifying the reference count value of the read metadata block and returning the modified metadata block comprises: modifying the read The reference count value of the metadata block, and the modified allocation group corresponding to the metadata block is restored.