KR101652436B1 - Apparatus for data de-duplication in a distributed file system and method thereof - Google Patents

Abstract

The present invention relates to a deduplication technique in a distributed file system, and in a distributed file system in which replicas are distributed and stored in various storage spaces for each file in order to improve data stability and performance, Are grouped into a similar file group, and are stored in a file server, i.e., a storage medium, respectively, assigned to the group, and are then deduplicated for each storage medium. According to the present invention, the files analyzed by the similar file are analyzed into similar groups by analyzing the extension of the files and the content type in the injection file system, and the files are stored in the storage media allocated for each similar group, Removal of duplicate files can be performed for files having different names even if the same file is used, thereby efficiently saving the storage space of the storage media.

Description

[0001] APPARATUS FOR DATA DE-DUPLICATION IN A DISTRIBUTED FILE SYSTEM AND METHOD THEREOF [0002]

The present invention relates to an apparatus and method for deduplication in a distributed file system, and more particularly, to a system and method for deduplication in a distributed file system in which replicas are generated and managed in a plurality of storage spaces for each file in order to improve data stability and performance, And more particularly, to an apparatus and method for deduplication in a distributed file system suitable for duplicating copies of files to be created in a similar file group and storing them in a disk allocated to a group, respectively, and performing deduplication on a per disk basis.

Generally, in a general-purpose hardware-based distributed file system, replicas are generated and managed on a per-file basis in order to improve data stability and performance.

Such a distributed file system includes a symmetric structure in which replicas are distributed and stored in a storage device having the same storage capacity, a storage device having various storage capacities, and an asymmetric structure for storing replicas in various storage media media.

Particularly, in order to mitigate data instability and performance degradation caused by the use of relatively inexpensive disk media, a general-purpose hardware-based distributed file system requires a copy of the original file to be copied to a disk / computer other than the disk on which the original file exists One or more. Duplication technology is used to prevent the increase of disk space caused by this.

Deduplication technology is applied to blocks which are constituent units of a disk or a file. If the same block is a block belonging to a different file, it does not store duplicate blocks, but keeps only one block and maintains link information Thereby reducing the data storage space.

For example, an attachment sent to a number of users in a mail program may be stored on each user's computer, but one file may be stored on the user's disk / computer via redundant control techniques, By keeping the link information 100, the storage capacity can be effectively saved.

In the conventional deduplication technique, the unnecessary files are removed from the files in the plurality of disk apparatuses having the same file name and extension by searching the entire disk, The duplication elimination method for files having the same file name and extension is performed in the case where the file name or extension of each file is not the same, As a result, it is impossible to perform overloading and quick duplication removal on the system as the number of stored files increases.

Accordingly, in order to improve data stability and performance, a distributed file system in which replicas are distributed and stored in a plurality of storage spaces on a file-by-file basis constitutes a replica of files generated by different users as similar file groups, The present invention provides a deduplication apparatus and method in a distributed file system capable of performing deduplication on a per file server basis after being stored in an allocated file server.

Also, according to the present invention, it is possible to search for similar files by checking the content type of each file, organize the retrieved similar files into groups, store the retrieved similar files in the storage mediums allocated for each group, The present invention provides a deduplication apparatus and method in a distributed file system.

An apparatus of an embodiment of the present invention includes a duplicate file checking unit for determining similarity based on type information between files stored in a plurality of file servers, And a duplicate elimination unit for eliminating redundant files in each similar file group based on the similarity determined through the duplicate file checking unit.

The type information is at least one of a file name, a file extension name, a file size, a file type, and metadata information.

If the file name and the file size are the same but the file size or the file name is different or at least one block is the same, And judges as a file.

The pseudo-file group constructing unit may form a group according to at least one of a file extension, a file type, a file importance, and a file usage frequency.

The pseudo file group constructing unit may allocate and store at least one pseudo file group to each file server.

And the deduplication unit removes a file exceeding a predetermined number of duplicated files in the similar file group.

The duplicate elimination unit may remove identical blocks exceeding a predetermined number when at least one block is identical in each similar file in each similar file group.

The method of the present invention includes the steps of: determining similarity based on type information of each of files distributed and stored in a plurality of file servers in a duplicate file checking unit; determining similarity in the similar file grouping unit through the duplicate file checking unit; A step of creating duplicated files in a group and assigning the generated similar file group to a file server, a process of removing redundant files in each similar file group based on the similarity determined through the duplicate file checking unit in the duplicate removal unit .

The type information is at least one of a file name, a file extension name, a file size, a file type, and metadata information.

If the file name and the file size are the same, the file size or file name is different, or at least one block is the same, , It is determined as a similar file.

In addition, the process of assigning the generated similar file group to the file server may include assigning at least one similar file group to each file server and storing the same.

In addition, the step of allocating the generated similar file group to the file server may include sequentially allocating the similar file groups or allocating the similar file groups to the file server based on at least one of the file extension, the file type, the file importance, .

The removing of the duplicated files is performed by removing a file exceeding a predetermined number of duplicated files in each similar file group.

The removing of the duplicated files is performed by removing identical blocks exceeding a predetermined number when at least one block is identical in each similar file in each similar file group.

In the present invention, effects obtained by representative ones of the disclosed inventions will be briefly described as follows.

The present invention differs from the method of performing deduplication for files having the same name by analyzing all the files stored in the distributed file system and analyzing the file extensions and the content type, Files are stored in a similar group and are stored in a file server allocated to each similar group, that is, in a storage medium, and duplication is performed for each storage medium, thereby enabling duplicate removal of files having different names even if they are the same file Accordingly, the storage space of the storage medium can be efficiently saved.

1 is a block diagram showing a structure of a distributed file system according to an embodiment of the present invention;
2 is a block diagram illustrating the structure of a deduplication apparatus in a distributed file system according to an embodiment of the present invention;
3 is a flowchart illustrating an operation procedure of a deduplication apparatus in a distributed file system according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

Each block of the accompanying block diagrams and combinations of steps of the flowchart may be performed by computer program instructions. These computer program instructions may be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus so that the instructions, which may be executed by a processor of a computer or other programmable data processing apparatus, And means for performing the functions described in each step are created. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

In an embodiment of the present invention, in order to improve data stability and performance, in a distributed file system in which replicas are distributed and stored in various storage spaces on a file-by-file basis, replicas of files generated by different users are formed into similar file groups, A file server allocated to a group, that is, a storage medium, and then performs deduplication for each storage medium.

1 is a block diagram showing a structure of a distributed file system according to an embodiment of the present invention.

1, the distributed file system 100 includes a file list management unit 110, a file list database 120, a redundancy processing unit 130, a content type database 140, a control unit 150, a first file server 160 and a second file server 170 to an nth file server 180. The file server 170 may include a plurality of file servers,

The file list management unit 110 generates a list of files stored in the first file server 160 to the nth file server 180 and the file list database 120 stores a list of files generated by the file list management unit 110 . For example, the location of a file uploaded from the user terminal and stored in the first file server 160 may be displayed in the form of a directory. The file list management unit 110 and the file list database 120 may be respectively installed in the first file server 160 to the nth file server 180 for storing files and may be managed and operated by the controller 160 .

The file list management unit 110 periodically updates the list of files stored in the first to n'th file servers 180 to 180 and updates the list of files stored in the first to n'th file servers 160 to n Synchronization with the file server 180 can be performed.

The duplication processing unit 103 may analyze files analyzed in the similar file through the analysis of the content type of each file stored in the first file server 160 to the nth file server 180 into similar groups. As the type information for each file, basic information and metadata information of each file such as a file name, a file extension name, a file size, and a file type can be stored in the content type database 140. The files set as the similar group can be allocated to the storage media, that is, the first file server 160 to the nth file server 180. And it is possible to perform deduplication for each file server allocated to each similar group. The duplication processing unit 103 will be described in detail with reference to FIG.

The control unit 150 controls various components included in the distributed file system 100 and operates in conjunction with the file list management unit 110 and the redundancy processing unit 130 to perform a file list, The first file server 160 to the nth file server 180 may store corresponding files for each similar file group constructed from the redundant processing unit 130, It is possible to perform control such that duplicate elimination or the like is performed.

 The first file server 160 may include a first file manager 162 for storing a file to be stored and a first file database 164 for actually storing the file. The second file server 170 may include a file And a second file database 174 for actually storing the files. Similarly, the n-th file server 180 may include an n-th file management unit 182 for recording a file to be stored, And a control file database 184 that actually stores the file.

The plurality of file servers that can be represented by the first file server 160, the second file server 170 and the nth file server 180 may be various storage media (e.g., SSD (Solid State Drive), HDD Disk drives, optical disks, etc.), and the number of file servers may be variously changed according to the storage capacity, the number of users, the state of the system, and the like.

2 is a block diagram illustrating the structure of a deduplication apparatus in a distributed file system according to an embodiment of the present invention.

Referring to FIG. 2, the redundancy processing unit 130 may include a redundancy file inspection unit 200, a pseudo file group construction unit 202, a redundancy removal unit 204, and the like. The duplicate file checker 200 includes file basic information such as a file name, an extension name, a file size and a file type of each file stored in the first file server 160 to the nth file server 180, And it can be determined whether the file is a similar file based on metadata information that can include attribute information of the file such as the location and content of the file, the creator information, the rights and use conditions, and the usage history. At this time, when basic information and metadata information of each file are stored in the content type database 140, type information about each file obtained by searching type information about each file from the content type database 140 Based on this, you can check for similar files for each file.

In this case, if the file names are the same, the file name is different, the extension name, the file size, and the file type are the same, the file name is the same, but the file name and the file size are different And at least one block may be the same, and the checking method for the similar file may be performed by the user in accordance with the implementation method, or may be preconfigured with at least one reference condition on the distributed file system 100 .

The pseudo file grouping unit 202 may configure at least two files determined as duplicate files into a similar file group through the duplicate file checking unit 200 and allocate them to the file server for each similar pseudo file group.

At this time, the group structure is assigned to at least one of the file formats, file types, file importance, file usage frequency, and the like. It is also possible to arrange such a group structure for each layer. For example, a similar AVI file group, a similar MKV file group, and the like are arranged as a subordinate of a similar movie file group, and an AVI file group of a similar "movie name 1" and an AVI file group of a similar "movie name 2" Group and so on.

That is, since the files included in the similar file group are the same files, at least one of the blocks is the same file, so that they can be configured as a similar file group, thereby saving the deduplication time later.

The similar file group-specific file server allocation method is a method for allocating the similar file groups in sequence or allocating the similar file groups to the file servers in a standard manner set by at least one of file extension, file type, file importance, .

For example, when three pseudo file groups are configured, the pseudo file groups may be distributedly stored in the first file server 160 to the nth file server 180, and then duplicate removal may be performed.

The duplicate removal unit 204 may perform deduplication of similar files for each file server assigned to each similar file group. The duplicate removal method is a method in which the file name and the file size are the same when the file extensions are the same, the file size is the same, the file size or file name is different, or at least one block is the same, The same file or block will be removed. In this case, it is possible to check the number of pre-set replicas through the control unit 150. In addition to the number of replicas that have been checked, unnecessary files or files may be stored in the distributed file system 100, Blocks can be removed.

3 is a flowchart illustrating an operation procedure of a deduplication apparatus in a distributed file system according to an embodiment of the present invention.

Referring to FIG. 3, in step 300, the duplicate file checking unit 200 of the duplication processing unit 130 determines whether duplicate files are stored in the first to nth file servers 160 to 180 At this time, based on the type information of each file stored in the content type database 140, it is determined whether the files are similar or not.

For example, a method of judging whether there is a similarity between files is basically the same when file names and file sizes are the same, the file names are different, but the file sizes are the same, A case where one block is the same or the like is judged.

In step 302, the similar file group construction unit 202 forms a group of two or more files determined as similar files through the duplicate file inspection unit 200. [ In step 304, the similar file group configuration unit 202 allocates and stores a file server for each similar file group. At this time, the allocation method of the file server allocates the file servers in at least one of the standard method of allocating the generated similar file groups sequentially, or by the file extension, the type of the file, the importance of the file, do.

In step 306, the duplicate removal unit 204 deletes duplicate files for each similar file group in units of file servers, and removes the corresponding number of files or blocks based on the predetermined criteria.

As described above, in the embodiment of the present invention, in order to improve the data stability and performance, in the distributed file system in which replicas are distributed and stored in a plurality of storage spaces for each file, a content type for files generated by different users is identified And stores them in a file server, that is, a storage medium, assigned to each group, and then performs deduplication for each storage medium.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but is capable of various modifications within the scope of the invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the scope of the appended claims, and equivalents thereof.

As described above, the apparatus and method for deduplication in the distributed file system according to the present invention can reduce the data instability and performance degradation caused by using a relatively inexpensive disk media. By analyzing all the files stored in a general-purpose hardware-based distributed file system that has at least one disk / computer other than the disk, the system removes redundant removal of files composed of the same name

By analyzing the contents type of files in the distributed file system, the files analyzed by similar files are organized into similar groups and stored in a storage medium such as a file server allocated for each similar group, Thus, it is possible to reduce duplication of files having different names even if the same file is used, which is suitable for reducing the storage space of the storage medium.

100: Distributed file system 110: File list management unit
120: file list database 130: redundant processing section
140: content type database 160:
160: first file server 162: first file manager
164: first file database 170: second file server
172: second file management unit 174: second file database
180: nth file server 182: control file manager
184: Control File Database

Claims (14)

A duplicate file checking unit for determining similarity based on type information between the files stored in the plurality of file servers,
A similar file group constituent unit for generating judged similar files in a group and assigning similar files in the same similar file among the generated similar file groups to the same file server,
A redundancy elimination unit for judging existence of redundant files or duplicated file blocks in each similar file group for each file server based on the similarity determined by the redundancy file checking unit,
And a deduplication unit for deduplicating the distributed file system. The method according to claim 1,
The type information includes:
A file name, a file extension name, a file size, a file type, and metadata information. The method according to claim 1,
The duplicate file checking unit checks,
When the file name and the file size are the same, the file name is different but the file size is the same, the file size or the file name is different, or at least one block is the same. A deduplication device in a distributed file system. A duplicate file checking unit for determining similarity based on type information between the files stored in the plurality of file servers,
A similar file group construction unit for generating the similar files judged as a group and assigning the generated similar file group to the file server,
And a duplicate removal unit for removing duplicated files in each similar file group based on the similarity determined through the duplication file checking unit
Lt; / RTI >
Wherein the similar file group constituent unit comprises:
Wherein the group is constituted by at least one reference method among file extensions, file types, file importance, and file usage frequencies. A duplicate file checking unit for determining similarity based on type information between the files stored in the plurality of file servers,
A similar file group construction unit for generating the similar files judged as a group and assigning the generated similar file group to the file server,
And a duplicate removal unit for removing duplicated files in each similar file group based on the similarity determined through the duplication file checking unit
Lt; / RTI >
Wherein the similar file group constituent unit comprises:
Wherein at least one similar file group is allocated to each file server and stored. A duplicate file checking unit for determining similarity based on type information between the files stored in the plurality of file servers,
A similar file group construction unit for generating the similar files judged as a group and assigning the generated similar file group to the file server,
And a duplicate removal unit for removing duplicated files in each similar file group based on the similarity determined through the duplication file checking unit
Lt; / RTI >
Wherein the de-
And removing a file exceeding a predetermined number of duplicated files in each of the similar file groups. A duplicate file checking unit for determining similarity based on type information between the files stored in the plurality of file servers,
A similar file group construction unit for generating the similar files judged as a group and assigning the generated similar file group to the file server,
And a duplicate removal unit for removing duplicated files in each similar file group based on the similarity determined through the duplication file checking unit
Lt; / RTI >
Wherein the de-
And removing the same block exceeding a predetermined number when at least one block is identical in each pseudo file in each pseudo file group. Determining similarity based on type information of each file distributed and stored in a plurality of file servers in a duplicate file checking unit;
Creating a group of files determined to be similar through the duplicate file checking unit in the similar file grouping unit and assigning similar files in the same similar file group among the generated similar file groups to the same file server;
Determining whether there is a duplicated file or a duplicated file block in each similar file group for each file server based on the similarity determined through the duplicate file checking unit in the duplicate removal unit,
A method for deduplication in a distributed file system. 9. The method of claim 8,
The type information includes:
A file name, a file extension name, a file size, a file type, and metadata information. 9. The method of claim 8,
The process of determining similarity includes:
When the file name and the file size are the same, the file name is different but the file size is the same, the file size or the file name is different, or at least one block is the same. A method of deduplication in a distributed file system. Determining similarity based on type information of each file distributed and stored in a plurality of file servers in a duplicate file checking unit;
Creating a group of files determined to be similar through the duplicate file checking unit in the similar file grouping unit and assigning the generated similar file group to the file server;
Removing duplicated files in each similar file group on the basis of similarity determined through the duplicate file checking unit in the duplicate removal unit
/ RTI >
Wherein the step of assigning the generated similar file group to the file server comprises:
Wherein at least one similar file group is allocated to each file server and stored. Determining similarity based on type information of each file distributed and stored in a plurality of file servers in a duplicate file checking unit;
Creating a group of files determined to be similar through the duplicate file checking unit in the similar file grouping unit and assigning the generated similar file group to the file server;
Removing duplicated files in each similar file group on the basis of similarity determined through the duplicate file checking unit in the duplicate removal unit
/ RTI >
Wherein the step of assigning the generated similar file group to the file server comprises:
Wherein the similar file groups are sequentially allocated or allocated based on at least one of a file extension, a file type, a file importance, and a file usage frequency. Determining similarity based on type information of each file distributed and stored in a plurality of file servers in a duplicate file checking unit;
Creating a group of files determined to be similar through the duplicate file checking unit in the similar file grouping unit and assigning the generated similar file group to the file server;
Removing duplicated files in each similar file group on the basis of similarity determined through the duplicate file checking unit in the duplicate removal unit
/ RTI >
Wherein the step of removing the duplicate file comprises:
And removing a file exceeding a preset number of duplicated files in each pseudo file group. Determining similarity based on type information of each file distributed and stored in a plurality of file servers in a duplicate file checking unit;
Creating a group of files determined to be similar through the duplicate file checking unit in the similar file grouping unit and assigning the generated similar file group to the file server;
Removing duplicated files in each similar file group on the basis of similarity determined through the duplicate file checking unit in the duplicate removal unit
/ RTI >
Wherein the step of removing the duplicate file comprises:
Wherein the same block exceeding a predetermined number of blocks is removed if at least one block is identical in each pseudo file in each pseudo file group.

Images