CN103617177A - Stackable repeating data deletion file system - Google Patents

Abstract

A stacked data deduplication file system is proposed, including the file system service module. For normal data, the data of the underlying file system is imported into the file system by direct interface conversion; for the deduplicated data, read Take the corresponding data attribute identifier, redirect the IO process, and realize transparent and seamless access to the data after deduplication; the deduplication service module reads the file system log data exported by the file system service module, and performs data signature after parsing the log content The calculation, the detection and deletion of duplicate data, and the identification of the data after deduplication is completed. The system can make full use of the storage capacity of the existing storage system without upgrading the hardware to save investment to the greatest extent. Through the stacked software design, it provides the deduplication function on the existing file system, optimizes the data storage structure, and reduces the storage cost. The space occupied by the system.

Description

A Stacked Deduplication File System

technical field

The invention relates to the field of computer storage, in particular to a data deduplication file system based on stacked file system technology.

Background technique

In a large-scale storage system, the contradiction between the rapid growth of data and the relatively slow upgrade of storage devices is relatively acute. In order to alleviate the problem of space growth in the storage system, reduce the space occupied by data, reduce costs, and maximize the use of existing resources, deduplication technology has been adopted. Become an indispensable key technology in large-scale systems.

By using the data deduplication technology, users can obtain obvious data reduction effects, which can greatly reduce the bandwidth requirements of the storage system, and reduce operating and maintenance costs. Through data reduction, the actual storage capacity of the backend is greatly reduced, which brings simpler storage management and effectively reduces management costs.

However, currently popular data deduplication solutions are mostly for nearline storage and backup storage, and are often closely integrated with backup systems, so they cannot provide general file system services. There are few products that can directly provide the deduplication function in the online system, and all of them need to use a proprietary file system format. These proprietary file systems often have many limitations in terms of performance, function, reliability, and scalability , making it difficult to apply directly in large-scale online storage systems.

Existing large-scale storage systems are often built based on mature file systems, such as ext3, ext4, xfs, lustre, etc. These file systems do not have the function of deduplication, and if you want to use the function of deduplication, you will face It is necessary to use a proprietary file system, endure obvious perceptible performance degradation, and perform large-scale data migration, which brings extremely high time and space costs, and is basically not feasible in a storage system that already has a large amount of data , the cost is too high.

In view of this situation, the present invention designs a stacked data deduplication file system, which can provide deduplication function based on the existing mature file system, fully maintain the performance of the original storage system, and hardly need any data migrate.

Contents of the invention

The present invention designs and implements a stacked data deduplication file system, which can make full use of the storage capacity of the existing storage system and save investment to the greatest extent without upgrading the hardware. Through the stacked software design, on the existing file system Provides the deduplication function, optimizes the data storage structure, and reduces the space occupied by the storage system.

The system includes:

The file system service module, for normal data, imports the data of the underlying file system into this file system by means of direct interface conversion; for the data that has been deduplicated, reads the corresponding data attribute identification, and performs IO process re- Orientation, to achieve transparent and seamless access to data after deduplication;

The deduplication service module reads the file system log data exported by the file system service module, calculates the data signature, detects and deletes duplicate data after parsing the log content, and identifies the data after deduplication is completed.

The beneficial effects of the present invention are: the design based on the stacked file system can make full use of the existing storage system, and only by installing the software system described in this patent, the existing file system can support the deduplication function to save storage space, There is no need to migrate data, while maintaining the IO performance of the original storage system, achieving full equipment recycling and investment protection.

Description of drawings

Accompanying drawing 1 is a schematic diagram of the architecture of the stacked data deduplication file system proposed in this patent.

Detailed ways

Referring to accompanying drawing 1 below, the content of the present invention is described the process of realizing this system structure with a specific example.

As described in the summary of the invention, the architecture of the present invention mainly includes: a file system service module and a deduplication service module.

The file system service module implements a file system that fully supports the POSIX protocol. It adopts the design strategy of a stacked file system, and fully realizes the services of the underlying file system through mapping and rewriting at the file system interface layer. For normal data, this module imports the data of the underlying file system into this file system by means of direct interface conversion, realizing the seamless access of normal data. For the deduplicated data, this module reads the corresponding data attribute identification according to the agreement of the file system described in the present invention, redirects the IO process, and realizes transparent and seamless access to the deduplicated data.

The deduplication service module runs independently out-of-band. It adopts multi-thread design, fully utilizes the parallel computing capability of the multi-core system, and provides ultra-high-speed deduplication function. This module reads the file system log data exported by the file system service module, analyzes the log content, calculates the data signature, detects and deletes duplicate data, and identifies the data after deduplication. This module can run simultaneously with the file system service module, and through the fine-grained lock designed in the file system service module, the atomicity of data processing is guaranteed and reliable parallel data processing capabilities are provided.

In a typical configuration environment, the file system service module and deduplication service module can be installed in the host system as general application software. After the relevant software configuration is carried out, the file system service module and the deduplication service module can be started. At this time, the file system described in the present invention can be mounted on the host computer and data access can be performed. After a period of file system IO is completed, the deduplication service module can automatically calculate the data signature, detect and delete duplicate data according to the configuration parameters, and complete the marking of the deduplicated data.

So far, the entire stacked deduplication file system has been fully realized, and the function of providing high-performance deduplication service on the existing file system has been realized, which greatly improves the space utilization rate of the storage system and effectively protects customer investment. .

Of course, the present invention can also have other various embodiments, and those skilled in the art can make various corresponding changes and deformations according to the present invention without departing from the spirit and essence of the present invention, but these corresponding Changes and deformations should all belong to the protection scope of the claims of the present invention.

Claims (1)

1. a stack data de-duplication file system, is characterized in that comprising:

File system service module, for normal data, adopts the mode of direct interface conversion by the data importing presents system of bottom document system; For the data of having carried out data de-duplication, read corresponding data attribute sign, carry out being redirected of IO flow process, realize the transparent seamless access of heavily deleting rear data;

Heavily delete service module, the file system journal data that file reading system service module derives, resolve the calculating of the laggard row data signature of log content, detection and the deletion of repeating data, complete heavily to delete rear data to be identified.

Images