TW201308115A - A distributed de-duplication system and the method therefore - Google Patents

Abstract

A distributed de-duplication system and the method therefore. The client performs a de-duplication process for the input file and generates a polarity of data blocks and the corresponded fingerprint value. The client sends the query request to the dispatch server and the query request has the fingerprint value. The dispatch server stores the location of the fingerprint value. The dispatch server redirects the query request to the de-duplication device according to the fingerprint value. If the de-duplication device has not the fingerprint value, the de-duplication device stores the new data block into the storage server according to the new fingerprint value.

Description

Decentralized deduplication system and processing method thereof

A deduplication system and method thereof, in particular, a decentralized deduplication system and a processing method thereof.

With the rise of the Internet, many network providers provide a lot of storage space on the network in order to effectively save users' files. In the past, it was a storage service that provided network space by a single server. However, a single server has limited computing power, so it evolved into a multi-server to provide storage services in a parallel processing manner. This type of storage is called a decentralized storage system.

Please refer to "Figure 1", which is a schematic diagram of stored data of the prior art. In general, a decentralized storage system is a file data that can completely back up users. Therefore, the same data is stored in different server terminals 121. For example, a distributed storage system has three storage servers 121. When the client 111 wants to store 100 Mbytes of data into the network space, the distributed storage system stores the 100 Mbytes into the three storage servers 121, respectively. As a result, all storage servers 121 will occupy 300 Mbytes of space. If the files of each client 111 are backed up on each storage server 121, this is a heavy burden for the network provider.

In view of the above problems, the present invention provides a distributed deduplication system for storing at least all sub-blocks generated by a client.

The decentralized deduplication system disclosed in the present invention comprises: a client, a dispatch server, a duplicate data processing device and a storage server. The client deduplicates the input file and generates a segmentation data block and a corresponding fingerprint feature value (Fingerprint).

The Dispatch Server records the storage location of the segmentation data block of the input file; the dispatch server forwards the query request to the corresponding duplicate data processing device according to the fingerprint feature value; and the duplicate data processing device (Dedup. Engine) from the fingerprint feature If the fingerprint feature value is already stored in the lookup table, if the fingerprint feature value is not stored in the fingerprint feature lookup table, the duplicate data processing device assigns the corresponding segmentation data block to the storage server according to the fingerprint feature value, and sends the fingerprint data block to the client. Contains the storage node information of the assigned storage server.

Fingerprint feature values are generated by SHA-1, Hash, or one-way algorithms, so that each sub-block can only correspond to a unique fingerprint feature value. And after the storage server stores the new segmentation data block, the duplicate data processing device runs the synchronization process of the fingerprint feature lookup table to update the fingerprint feature lookup table of the other duplicate data processing device.

The invention further provides a decentralized processing method for deduplication comprising the steps of: the client receives the input file to generate a segmentation data block, and sends a query request with the fingerprint feature value to the dispatch server; the dispatch server according to the fingerprint feature value Forwarding the query request to the corresponding duplicate data processing device; the repeated data processing device determines whether the fingerprint feature value already exists in the fingerprint feature lookup table; if the fingerprint feature value is not stored in the fingerprint feature lookup table, the duplicate data processing device according to the fingerprint feature The value assigns the corresponding segmentation data block to the storage server, and sends a storage node message containing the assigned storage server to the client; the client transmits the segmentation data block to the storage server according to the storage node message.

The decentralized deduplication system and method thereof proposed by the invention are processed by hierarchical assignment and repeated data comparison, so that the data volume of each data storage server can be effectively reduced, thereby improving the storage of the overall data amount. space.

The features and implementations of the present invention are described in detail below with reference to the drawings.

Please refer to "Figure 2" for a schematic diagram of the architecture of the present invention. The decentralized deduplication system of the present invention can be applied to a local area network or an internet network, and the distributed deduplication system of the present invention includes: a client 211, a dispatch server 212 (Dispatch Server), and deduplication processing. Device 213 (De-dup Engine) and storage server 214. The client 211 is configured to receive an input file and perform a segmentation process on the input file for determining the deduplication.

Deduplication is a data reduction technique commonly used in disk-based backup systems with the primary goal of reducing the storage capacity used in storage systems. It works by looking for duplicate variable-sized data blocks at different locations in different files over a certain period of time (defined in the text as a split data block). Duplicate data blocks are replaced with tokens. The use of "deduplication" technology can make more backup space, not only can save the backup data on the storage server 214 for a longer period of time, but also save a lot of bandwidth required for offline storage.

In the process of performing deduplication, the client 211 performs a process of segmenting the input file. The input file will generate multiple sliced data blocks after being segmented. Subsequently, the client 211 hashes the data block and generates a hash value of each data block. The client 211 compares the obtained hash value with the hash value stored in the storage server 21, and judges whether or not there is the same hash value. If the same hash value exists, it means that the data block has been stored in the storage server 21.

After the client 211 of the present invention completes the process of data segmentation, a plurality of segmentation data blocks corresponding to the input file and its fingerprint feature value (Fingerprint) are generated. The fingerprint feature value is generated by the SHA-1 program, the hash program, or the One way function, so that each of the sub-blocks can only correspond to a unique fingerprint feature value. The client 211 sends a query request with a fingerprint feature value to the dispatch server 212.

The dispatch server 212 forwards the query request to the corresponding deduplication processing device based on the fingerprint feature value, and the dispatch server 212 is further operable to record the storage location of the sliced data block of the input file. The number of deduplication processing devices is determined by the number of clients 211. Each of the repeated data processing devices 213 further includes a fingerprint feature lookup table for recording the fingerprint feature values corresponding to each of the segmented data blocks. After receiving the fingerprint feature value, the repeated data processing device 213 determines whether the fingerprint feature value already exists. When there is no fingerprint feature value to be queried in the fingerprint feature lookup table, the deduplication processing device selects any storage server 214 for storing the corresponding segmentation data block.

In order to clearly explain the operation process of the present case, please refer to the "Fig. 3", which is a schematic diagram of the operational flow of the present invention, and the present invention includes the following steps:

Step S310: After receiving the input file, the client generates a segmentation data block, and sends a query request with a fingerprint feature value to the dispatch server;

Step S320: the dispatching server forwards the query request to the corresponding duplicate data processing device according to the fingerprint feature value;

Step S330: The repeated data processing device determines whether the fingerprint feature value already exists in the fingerprint feature value lookup table;

Step S340: If the fingerprint feature value is stored in the fingerprint feature value lookup table, the duplicate data processing device returns the pen segmentation data block to the client through the dispatch server;

Step S350: If the fingerprint feature value is not stored in the fingerprint feature value lookup table, the repeated data processing device assigns the corresponding segmentation data block to the storage server according to the fingerprint feature value, and sends the client with the assigned storage server. Storage node information;

Step S360: The client transmits the segmentation data block to the storage server according to the storage node message.

The client 211 receives the input file and performs a segmentation process to generate a sliced data block. The client 211 transmits a query request with a fingerprint feature value to the dispatch server 212 for transmission. The dispatch server 212 forwards the query request to the corresponding duplicate data processing device 213 based on the fingerprint feature value. The repeated data processing device 213 can perform the remainder processing according to the fingerprint feature value, and forward the query request to the dispatch server 212 according to the result of the remainder processing.

For example, the client 211 divides the input file into 1024-segment data blocks, and generates corresponding fingerprint feature values (also 1024 pens) for the segmented data blocks through SHA-1. In addition, if the number of dispatch servers 212 is three, the remainder of the 1024 fingerprint feature values are respectively taken (that is, the remainder of 3 is taken). In actual operation, the parameters of the remainder may be determined according to the number of dispatch servers 212. Next, the query request is forwarded to the corresponding duplicate data processing device 213 based on the result of the remainder. For example, the query request for the fingerprint feature value whose remainder is "0" is required to be forwarded to the first duplicate data processing device 213, and the query request for the fingerprint feature value having the remainder "1" is forwarded to the second duplicate data processing device 213, and the remainder is The query request of the fingerprint feature value of "2" is forwarded to the third iterative data processing device 213.

Next, after the repeated data processing device 213 receives the query request, the duplicate data processing device 213 searches for a fingerprint feature value in the fingerprint feature value lookup table. If the fingerprint feature value is already stored in the fingerprint feature value lookup table, the duplicate data processing device 213 returns to the client 211 via the dispatch server 212 that the pen segmentation data block already exists. Otherwise, the repeated data processing means 213 assigns the corresponding sliced data block to the storage server 214 based on the fingerprint feature value, and transmits a storage node message containing the assigned storage server 214 to the client 211. The manner of notifying the client 211 is as follows: the dispatch server 212 forwards the query request to the corresponding duplicate data processing device 213, and sends the storage node message to the client 211. Alternatively, the dispatch server 212 forwards the query request to the corresponding duplicate data processing device 213 and transmits the storage node message to the client 211 via the duplicate data processing device 213.

Further, the duplicate data processing device 213 additionally records the metadata information (Metadata) of the segmentation data block. The metadata information is used to maintain the storage location and length of the server stored in the segmentation data block and on the corresponding storage server. When the client 211 needs to read the segmentation data block, the duplicate data processing device 213 can further find and read the position of the corresponding segmentation data block through the metadata information, and can also confirm the segmentation data block by using the fingerprint feature value. The correctness.

Finally, when the client 211 receives the storage node message specifying the storage location, the client 211 transmits the segmentation data block to the storage server 214 according to the storage node message. At the same time, the repeated data processing device 213 performs a synchronization process of the fingerprint feature lookup table for updating the fingerprint feature values recorded by the fingerprint feature lookup table in the other duplicate data processing devices 213 and the corresponding segmentation data. The location where the block is stored. When the other duplicate data processing device 213 receives the query request of the stored segment data block, the duplicate data processing device 213 can immediately determine whether the pen slice data block already exists.

The decentralized deduplication system and method thereof proposed by the invention are processed by hierarchical assignment and repeated data comparison, so that the data volume of each data storage server can be effectively reduced, thereby improving the storage of the overall data amount. space.

While the present invention has been described above in terms of the preferred embodiments thereof, it is not intended to limit the invention, and the invention may be modified and modified without departing from the spirit and scope of the invention. The patent protection scope of the invention is subject to the definition of the scope of the patent application attached to the specification.

111. . . Client

121. . . Servo end

211. . . Client

212. . . Dispatch server

213. . . Repeated data processing device

214. . . Storage server

Figure 1 is a schematic diagram of stored data of the prior art.

Figure 2 is a schematic diagram of the architecture of the present invention.

Figure 3 is a schematic diagram of the operational flow of the present invention.

Claims (9)

A decentralized deduplication system for storing at least all the sub-blocks generated by the client, the deduplication system comprising: at least one storage server for storing the segmented data blocks; and a client An input file runs a deduplication program, and generates the segmentation data block and a corresponding fingerprint feature value (Fingerprint), the client sends a query request having the fingerprint feature value, and according to a storage node message The split data block is sent to the storage server; a de-dup engine is used to determine whether the fingerprint feature value already exists, and the new slice data block is added according to the new fingerprint feature value. Assigned to the storage server; and a dispatch server (Dispatch Server), which records the storage location of the segmented data blocks of the input file, and the dispatch server forwards the query request to the corresponding identifier according to the fingerprint feature value. The duplicate data processing device. The deduplication data deletion system of claim 1, wherein the data processing device performs the remainder processing on the fingerprint feature value, and forwards the query request to the dispatch server according to the result of the remainder processing. The distributed deduplication system of claim 1, wherein the dispatching server forwards the query request to the corresponding data processing device and sends the storage node message to the client. The deduplication system of claim 1, wherein the dispatching server forwards the query request to the corresponding data processing device, and sends the storage node message to the client through the data processing device. end. The distributed deduplication system of claim 1, wherein the duplicate data processing device further records the metadata information of the segmentation data block. The deduplication data deletion system of claim 1, wherein after the storage server stores the segmentation data blocks, the data processing devices run a synchronization process of a fingerprint feature lookup table. The fingerprint feature lookup table is used to update other such data processing devices. A deduplication processing method for deduplicating data is used to store at least all the data blocks generated by a client. The processing method includes: the client receiving the input file, generating the segmentation data blocks, and sending a servo to the server. Transmitting a query request having a fingerprint feature value; the dispatching server forwards the query request to a corresponding one of the repeated data processing devices according to the fingerprint feature value; and the repeated data processing device determines whether the fingerprint feature value already exists in the In the fingerprint feature lookup table, if the fingerprint feature value is not stored in the fingerprint feature lookup table, the duplicate data processing device assigns the corresponding segmentation data block to the storage server according to the fingerprint feature value, and sends the sliced data block to the client The terminal sends a storage node message including the assigned storage server; and the client transmits the segmentation data block to the storage server according to the storage node message. The deduplication processing method of the data deduplication method of claim 7, wherein the deduplication data processing device performs the remainder processing on the fingerprint feature value, and forwards the query request to the dispatch server according to the result of the remainder processing. . A decentralized processing method for deduplication as described in claim 7, wherein the duplicate data processing device additionally records the metadata information of the segmentation data block.