KR101382273B1 - A Hybrid Data Deduplication Method for Reducing Wear-Level of SSD-based Server Storage - Google Patents

Abstract

The present invention relates to a complex data deduplication method of an SSD-based server-class storage device which stores data received from a server in a solid state disk (SSD) storage, and removes and stores duplicated portions. Obtaining the number of writes of the first data); (b) selecting one of an immediate deduplication scheme and a reserved deduplication scheme using a write count; (c) if the deduplication method is selected immediately, removing the duplicated portion from the first data and storing the duplicated portion in the SSD storage device; (d) storing the first data in the backup storage unit when the reserved deduplication method is selected; And (e) reading data (hereinafter referred to as second data) from the backup storage unit, and removing the duplicated portion from the second data and storing the data in the SSD storage device.
By using the above method, by using both the instant deduplication method and the reservation deduplication method, the number of writes is reduced compared to the reservation deduplication method and the processing performance is improved compared to the instant deduplication method, and as a result, SSD wear can be extremely reduced. .

Description

A Hybrid Data Deduplication Method for Reducing Wear-Level of SSD-based Server Storage}

According to the present invention, when SSD is used as server storage, only the advantages of the existing data deduplication method are combined, and the SSD-based method uses an instant deduplication method when there is a small amount of data moving to the server and a reservation deduplication method when there is a lot of data. The present invention relates to a complex data deduplication method for server class storage devices.

Generally, solid state disks (SSDs) are new storage devices using NAND flash memory, and have low power consumption, fast data access speed, and strong external shock.

Unlike a conventional hard disk (HDD), NAND flash memory must perform a delete operation in order to write data back to a location where a write operation is performed. Another characteristic is that the erase operation is much slower than the read or write operation, and has the disadvantage of limiting the number of write / delete operations. The SLC (Single Level Cell) type stores one data in one cell. 100,000 times, MLC (Multi Level Cell) type that stores two data in one cell has 10,000 durability. In addition, storing more than three data in a cell has lower durability, and in the case of storage using several SSDs as a server, in the case of a write-intensive environment, the loss of disk increases, resulting in financial damages. .

Therefore, in order to make up for the shortcomings of SSDs with these characteristics, data deduplication can be used to free up disk space by removing redundant data in the target storage. Data deduplication is a technology that reduces data costs by eliminating duplicate data files and storing only unique contents, which reduces the cost of building large servers and is used for efficient backup.

This technology is classified into source-based deduplication and target-based deduplication depending on where deduplication occurs. Source-based deduplication is a method in which data is deduplicated in advance before data is sent to the backup target.

Target-based deduplication uses two methods. First, in-line deduplication is a method of deduplicating data before it is stored on the target storage. Second, post-processing deduplication is a method of deduplicating data after storing it on a temporary disk.

If you do not manage redundant data in storage effectively, you will run out of available space, garbage collection will occur frequently, and unnecessary write / delete operations will occur.

Meanwhile, technologies are being studied by companies such as IBM, Data Domain, Dell, and university research labs to effectively use redundant data. University of Michigan's Landon P. Cox is conducting research on Pastiche, which uses only duplicated data on the network to back up only the data that is not duplicated. Data Domain, Inc. and Princeton University Using techniques to avoid disk bottlenecks, we are researching techniques to improve disk access rates of data servers while minimizing and deduplicating disk access. In addition, IBM and Texas State University are researching a technique to remove redundant hash information and hard-linked data by hashing the redundant data in the image with SHA-1 because the images generated by the virtual machine generate a lot of redundant data.

 There is a DRED system that uses delta encoding to remove redundant data such as e-mails or web pages on the Internet, and uses MD5 and SHA-1 hash functions to prevent duplicate storage of files in P2P overlay networks such as Pastiche and Chord. When saving, create a hash of the file and compare it with each other.

LBFS (Low-bandwidth Network File System) is a network file system designed to design when network conditions are not good. The protocol is designed to allow data transmission even when the network connection is interrupted and reconnected. In order to use the network bandwidth efficiently, the scheme is designed to prevent the retransmission of redundant data. LBFS uses CDC (Content-defined Chunks). Insert an anchor block into the file, hashing with Rabin Fingerprint, and returning a random value. Then, create a hash using hash functions such as MD5 and SHA1 and transfer the hashes to the server before sending data. Compare duplicates to hash lookup tables.

1 is a schematic flowchart divided into classifications according to where and when deduplication occurs. The study is divided into source-based and target-based deduplication methods, and studies on more efficient removal methods are being conducted by each company and research institute.

Target-based data deduplication includes immediate deduplication and scheduled deduplication.

Instant deduplication uses a dedicated appliance to analyze the data moving to the server. The appliance divides the data into 4KB chunk blocks in real time while generating data, generates hashes, and sends only non-redundant data to the server. It is efficient when the amount of data being moved to the server is small, but when there is a large amount of data, it is limited to IO operations per second (IOPS) of the SSD, which causes a data bottleneck.

The reserved deduplication method deduplicates data outside the data path. It requires a separate storage to store data in the intermediate stage before deduplication, and has the advantage of creating a complete native backup image for fast recovery directly from the disk target. However, it is important to consider that additional storage capacity is needed to store native backup images before applying deduplication.

Recent research suggests HP's Sparse Indexing and Data Domain's Stream-Informed Segment Layout Summary Vector, and focuses on efficiently processing hash data that is problematic when processing large amounts of data.

An object of the present invention is to solve the problems described above, when using the SSD as server storage only when combined with the advantages of the existing data deduplication technique to use a deduplication immediately if there is less data moving to the server and the data In many cases, it provides a complex data deduplication method for SSD-based server-class storage devices using reserved deduplication.

In order to achieve the above object, the present invention relates to a complex data deduplication method of an SSD-based server-class storage device for storing the data received from the server in a solid state disk (SSD) storage device, removing the duplicated portion, (a) obtaining a number of writes of data (hereinafter, referred to as first data) received by the server; (b) selecting one of an immediate deduplication scheme and a reserved deduplication scheme using the number of writes; (c) removing the duplicated portion from the first data and storing the duplicated portion in the SSD storage device when the immediate deduplication method is selected; (d) storing the first data in a backup storage unit when the reserved deduplication method is selected; And (e) reading data (hereinafter referred to as second data) from the backup storage unit, and removing the duplicated portion from the second data and storing the data in the SSD storage device.

In addition, the present invention provides a complex data deduplication method of the SSD-based server-class storage device, in the step (b), if the number of times of writing the first data is more than a predetermined threshold value is determined by the reservation deduplication method, If the number of writes is less than the threshold, it is determined by the deduplication method immediately.

In addition, the present invention, in the complex data deduplication method of the SSD-based server-class storage device, in the step (c) or (e), the first or second data is divided into chunk blocks, and the chunk The method may determine whether the blocks are duplicated on a block basis, and in case of overlapping, remove the chunk blocks from the first or second data.

In addition, in the complex data deduplication method of the SSD-based server-class storage device, in the step (c) or (e), the unique hash value is extracted by applying a fingerprinting technique to the chunk block, If the unique hash value of the chunk block has the same unique hash value as compared with the previously stored unique hash values, the chunk block is determined to be a duplicate.

In addition, the present invention is a complex data deduplication method of the SSD-based server-class storage device, in the step (a), by hooking the metadata of the first data, characterized in that the number of writes from the hooked metadata is obtained; do.

As described above, according to the complex data deduplication method of the SSD-based server-class storage device according to the present invention, by using a deduplication method according to the amount of data transmitted to the server by using the server storage device as the SSD An effect that can be expected to reduce performance and wear is obtained.

1 is a diagram showing a classification of a conventional deduplication system.
2 shows a configuration of an entire system for implementing the present invention.
3 is a block diagram of a structure of an SSD storage device used in an embodiment of the present invention.
4 is a block diagram of a file system structure of the deduplication system 30 according to an embodiment of the present invention.
5 is a configuration diagram of a complex deduplication system 30 according to an embodiment of the present invention.
6 is a flowchart illustrating a detailed method by a complex deduplication system according to an embodiment of the present invention.
7 is a graph showing the results of data throughput according to the number of consecutive writes according to an experiment of the present invention.
8 is a graph illustrating a comparison when transmitting 3GB by applying a complex deduplication method according to an experiment of the present invention.
9 is a table showing the results of benchmarking for each data according to an experiment of the present invention.
10 is a graph comparing the operation time of the existing data deduplication method and the complex deduplication method of the present invention according to an experiment of the present invention.
11 is a graph measuring the wear of the reservation deduplication technique and the complex deduplication method of the present invention according to an experiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the drawings.

In the description of the present invention, the same parts are denoted by the same reference numerals, and repetitive description thereof will be omitted.

First, the structure of the whole system for implementing this invention is demonstrated with reference to FIG.

As shown in FIG. 2, the entire system for implementing the present invention consists of a server 20, a storage device 40, and a deduplication system 30.

The server 20 is a conventional server having a function of receiving and storing data 10, and receives data from a user terminal, another server, a network, and the like. The server 20 stores the received data in the storage device 40.

The storage device 40 includes an SSD storage device 41 and a backup storage unit 42. That is, the storage device 40 includes a storage device 41 composed of SSDs, and the received data is mainly stored in the SSD storage device 41.

Solid state disks (SSDs) are large-capacity devices that attach a number of NAND flash memories and receive commands from the host. Unlike conventional hard disks, the block-based structure used in flash memory has a characteristic of limiting life according to wear. 3 shows a block diagram of the structure of the SSD.

As shown in FIG. 3, data stored in the SSD NAND flash memory is applied to the SRAM controller by applying FIFO & Control (First In, First Out) through a flash memory controller. This data is determined to access the RAM according to a command issued by the processor in the SRAM controller.

The deduplication system 30 analyzes the data received from the server 20 to find a duplicate part of the data. And after removing the duplicated portion is a system for controlling to store the data in the storage device (40). The deduplication system 30 may be implemented by being included in the operating system of the server 20 or may be implemented separately from the server 20.

The deduplication system 30 is implemented to access the data received by the server 20 for analysis of the data. In particular, when not changing the conventional server system, it may be implemented to access the data received by the server 20 using a method such as hooking.

Next, the configuration of the file system structure of the deduplication system 30 according to an embodiment of the present invention will be described in more detail with reference to FIG. 4.

As shown in FIG. 4, a file system for deduplication includes a local file system, a deduplication storage engine, a deduplication file system, and a control layer.

By adding a control layer on the deduplication file system, it determines whether to use immediate deduplication or reserved deduplication based on a threshold set according to the amount of data transmission.

On the other hand, some of the local file system may be used as the backup storage 42. That is, in case of using the reservation deduplication method, it may be used as a storage space for backing up and storing the data received from the server.

 An advantage of the present invention is that by applying such a complex data deduplication technique can effectively reduce the wear of the SSD. The complex structure of a data deduplication system consists of three components: volume, file system, and deduplication storage engine (DSE). A volume is a name that contains the size and information of each disk for data deduplication.

The deduplication file system is a file system specialized for deduplication. It is used to connect duplicated parts. In other words, the deduplicated files can be stored, used (loaded), and deleted through the DSE (deduplication engine, software implemented engine) in accordance with the application of the deduplication file system. do. On the other hand, these chunks are stored in hashtables and indexed into memory.

Next, the configuration of the complex deduplication system 30 according to an embodiment of the present invention will be described in more detail with reference to FIG. 5.

When an input request is received from the kernel to the deduplication file system, a control layer is added to hook the metadata of the transmitted data. If the write count operation information of the hooked metadata is more than a predetermined value, the threshold value is used to determine whether to perform deduplication or reserved deduplication immediately. It then passes through the appliance, storing only non-redundant data in storage.

The appliance actually applies the fingerprinting technique by dividing the data into chunks, generating hash values, and determining whether they are duplicated chunks. That is, it is a component corresponding to "dedup" of FIG.

Next, a detailed method by the complex deduplication system according to an embodiment of the present invention will be described in more detail with reference to FIG. 6.

As shown in FIG. 6, the complex data deduplication method according to an embodiment of the present invention includes: (a) acquiring a number of writes of data (hereinafter, referred to as first data) received by a server (S10); (b) selecting any one of an immediate deduplication scheme and a reserved deduplication scheme using a write count (S20); (c) if the deduplication method is selected immediately, removing the duplicated portion from the first data and storing it in the SSD storage device (S30); (d) if the reserved deduplication scheme is selected, storing the first data in the backup storage unit (S40); And (e) reading data (hereinafter referred to as second data) from the backup storage unit, and removing the duplicated portion from the second data and storing the data in the SSD storage device (S50).

First, the deduplication system 30 obtains a write count of data (hereinafter, referred to as first data) received by the server (S10). In particular, the metadata of the first data is hooked to obtain the number of writes from the hooked metadata.

That is, when an input request from the kernel stage of the server enters the deduplication file system, the metadata of the transmitted data (or the first data) is hooked by the added control layer. hooking). The deduplication system 30 obtains a write count from the hooked metadata.

Earlier, the data received by the server will be referred to as "first data". The data stored in the backup storage unit 42 will be referred to as "second data". This is to distinguish the data to be backed up and the data received immediately.

Next, the deduplication system 30 selects any one of an immediate deduplication method and a reserved deduplication method using the number of writes (S20).

In the first embodiment, if the number of times of writing of the first data is greater than or equal to a predetermined threshold value, the deduplication method is determined. If the number of times of writing is less than the threshold value, the deduplication method is immediately determined.

As a second embodiment, if the write count operation information of the hooked metadata is continuous for a predetermined value or more, it is given as a threshold value to determine whether to perform deduplication or reserved deduplication immediately. In other words, the deduplication system 30 determines that the write count operation information of the hooked metadata is more than a predetermined value (or a threshold value) in the case of continuous deduplication, and immediately if the number of consecutive writes is smaller than the threshold value. Judging by the deduplication method.

For example, if you set the threshold to 300, and the requested data operation consists of 900 operations such as 1 write, 2 reads, 1 write, 2 reads, ... (1 write and 2 reads) If the time is repeated). At this time, according to the first embodiment, since the total number of writes is 300 and the threshold value is 300 or more, it is determined as the reserved deduplication method. On the other hand, according to the second embodiment, since the number of consecutive writes does not exceed two, it is immediately determined as a deduplication method.

The number of writes is obtained by hooking metadata when an input request is made to the deduplication file system from the kernel. Metadata refers to information that follows together after the data to analyze and classify the structured information in the data and add additional information. It contains file information such as the address number of where the data will be stored and whether the data is a read or write operation. In this way, the write count is obtained using metadata.

Next, when the deduplication method is selected immediately, the deduplication system 30 controls to remove the duplicated portion from the first data and store it in the SSD storage device 41 (S30). That is, the first data passes through the appliance and stores only non-redundant data in the storage (or SSD storage) 41.

In this case, the data (or the first data) is divided into chunk blocks, and it is determined whether the chunk blocks are duplicated to remove the chunk blocks from the data. In particular, by applying a fingerprinting technique to the chunk block, a unique hash value is extracted, and if the unique hash value of the chunk block has the same unique hash value as compared to previously stored unique hash values, the chunk block is duplicated. I think that.

Data deduplication works by analyzing a set of data. One block of data, a series of bits, or the entire file may be analyzed.

When the algorithm is executed on a 4KB chunk block for deduplication, a fingerprint printing technique is applied to store a unique hash value. By applying SHA-1 algorithm, incoming data is converted into 160bit output data and fingerprinted for each block. Each time new chunk block data is hashed, it is compared with existing hash values in the fingerprint of existing blocks.

If the hash value to be compared is in the fingerprint, it means that the chunk block is duplicated and is not stored. If no hash value matches the fingerprint, the hash value is added to the fingerprint and these operations are repeated over and over.

The larger the amount of data, the larger the amount of hash fingerprint will be. This can lead to hash collisions if the hashing algorithm is not sophisticated. This means that different chunk blocks may have the same hash value. In this case, the analytical engine must use different criteria because there are no hash values and there are no criteria for processing different chunk blocks. If there is no way to resolve this conflict, both hash values that are compared must be deleted, and the corresponding chunk blocks cannot be fingerprinted for deduplication.

Preferably, there are three levels of deduplication: block, file and bit.

Block and bit-level deduplication allows the analysis engine to divide the file into chunks or segments and generate hash values for each. File-level deduplication analyzes an entire file, regardless of file size, which is very inefficient because different hash values are generated by only one bit. On the other hand, block and bit-level overlapping methods are efficient because they generate hash values by dividing the file into fixed or irregular sizes.

Next, when the scheduled deduplication method is selected, the deduplication system 30 controls to store the first data in the backup storage 42 (S40).

Then, the stored data (or second data) is read from the backup storage unit 42, and the duplicated part is removed from the read second data and stored in the SSD storage device 41 (S50).

In this case, the method of removing the duplicated portion from the second data is the same as the method of removing the duplicated portion from the first data.

That is, in the present invention, only the advantages of the two deduplication methods are combined, and when there is a small amount of data moving to the server, an immediate deduplication method is used, and when there is a lot of data, a reservation deduplication method is used. It is not necessary to store all the data in a backup image unnecessarily when the amount of data movement is small. SSDs require writes after deletion to perform write operations, which reduces wear unless performing unnecessary backups.

Next, the effects of the present invention will be described in more detail with reference to FIGS. 7 to 11.

Experimental environment for examining the effects of the present invention is as follows.

The CPU used Intel Core2 Duo 2.13GHz, 2GB RAM. As a storage device used by users and servers, Hana Micron's SSD disk (AHAS900090H) of 32GB MLC type was used in a server and four of them were bundled in RAID5 and a single SSD was used. In the Linux 2.6.17 environment, the open source OPENDEDUP was used and I / O METER was benchmarked to measure the wear efficiency of SSDs using the combined deduplication technology.

For the performance evaluation, a benchmark workload was constructed by randomly extracting document data, image data, image data, and various program data stored in storage of a server shared by 10 people with a time difference of 30 days.

7 is a data throughput rate according to the number of writes when an instant deduplication method is used. When the number of consecutive writes is about 25,000, disk bottleneck occurs. The experiment was benchmarked with this as the threshold. When the deduplication method and the reservation deduplication method are used, and the experimental results of applying the complex data deduplication method using the control layer are shown in FIG. 8. Experimental results show that the deduplication method writes significantly less than the scheduled deduplication method that requires a native backup image.

In other words, the instant deduplication method shows that SSD wear is excellent. In terms of data throughput, however, deduplication is less effective as soon as a disk bottleneck occurs.

In this experiment, we can see that the write count is reduced by 32% than when using the existing reserved deduplication method. It can be concluded that as the number of writes decreased, so did the delete operations.

Meanwhile, FIG. 9 shows deduplication rate for each data in the case where the deduplication technique is not applied and when it is applied. Multimedia data showed a relatively low data deduplication rate compared to document data, and document data showed a high data deduplication rate. Experiments show that the deduplication rate is the same for both the instant deduplication method, the scheduled deduplication method, and the proposed complex data deduplication method.

FIG. 10 is a result of measuring time when 3 GB of data is transmitted to a server when deduplication is not applied and by using the existing data deduplication techniques and the proposed complex data deduplication scheme.

 The experimental results show that the proposed combined deduplication scheme is 16.7% better than the immediate deduplication scheme, while 34.65% overhead is generated compared to the reserved deduplication scheme. However, as shown in FIG. 8, the wear rate of the SSD has a significant effect of reducing the number of writes by 32%. Therefore, the technique can be used in consideration of overhead.

11 is a result of measuring the wear rate of the conventional deduplication scheme and the complex deduplication scheme proposed in the present invention. The experimental results show that the proposed deduplication method has a 29.24% reduction in wear compared with the conventional deduplication method.

As a result, the present invention shows the reduction of wear of SSDs used as server storage using a complex data deduplication technique combining the advantages of immediate deduplication and scheduled deduplication. As wear decreases, storage life can be extended.

Although the present invention has been described in detail with reference to the above embodiments, it is needless to say that the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the spirit of the present invention.

10: data 20: server
30: deduplication system 40: storage device
41: SSD storage 42: backup storage

Claims (5)

In the complex data deduplication method of the SSD-based server-class storage device to store the data received from the server in a solid state disk (SSD) storage, but removes the duplicated portion,
(a) obtaining a number of writes of data (hereinafter, referred to as first data) received by the server;
(b) selecting one of an immediate deduplication scheme and a reserved deduplication scheme using the number of writes;
(c) removing the duplicated portion from the first data and storing the duplicated portion in the SSD storage device when the immediate deduplication method is selected;
(d) storing the first data in a backup storage unit when the reserved deduplication method is selected; And
(e) reading data (hereinafter referred to as second data) from the backup storage unit, removing duplicated portions from the second data, and storing the data in the SSD storage device;
In the step (b), if the number of writes of the first data is greater than or equal to a predetermined threshold value, it is determined by the reserved deduplication method, and if the number of writes is less than the threshold value, it is determined by the deduplication method immediately,
The number of writes is a number of times that a write operation must be performed to store the first data in the SSD storage device, and the write count is proportional to the transmission amount of the first data to perform the write operation. Data Deduplication Method.
delete The method of claim 1,
In step (c) or step (e), the first or second data is divided into chunk blocks, and it is determined whether or not to overlap each of the chunk blocks. Complex data deduplication method of the SSD-based server-class storage device, characterized in that for removing the chunk block.
The method of claim 3,
In step (c) or (e), a unique hash value is extracted by applying a fingerprinting method to the chunk block, and the unique hash value of the chunk block is the same as compared with the previously stored unique hash values. If there is a hash value, the chunk block is determined to be a duplicate data de-duplication method of the SSD-based server-class storage device characterized in that the duplicate.
The method of claim 1,
In the step (a), by hooking the metadata of the first data, complex data deduplication method of the SSD-based server-class storage device, characterized in that to obtain the number of writes from the hooked metadata.

Images