KR101441059B1 - Method for effective data storage in distributed file system - Google Patents

Abstract

In the distributed file system according to the present invention, a large size data is divided into a plurality of slices by using an erasure code method and is divided into at least two or more servers, And a local storage having a storage space for storing data, the method comprising the steps of: (a) storing a copy of the original file; A method of operating a distributed file system, the method comprising the steps of: (A) comparing a size of an original file to be input with a predetermined threshold; and (B) if the size of the original file is greater than a predetermined threshold, (slice) by using an erasure code method and divided into at least two or more servers If the size of the original file is smaller than or equal to a preset threshold value, storing a data copy having the same size as that of the original file using the mirror method is further included in the step (C) have.

Description

[0001] The present invention relates to an efficient data storage method in a distributed file system,

The present invention relates to a method for storing data in a distributed file system, and more particularly, to a method for efficiently storing user data by using a mirror method and an erasure code method in a distributed file system.

A distributed file system that provides a consolidated file system environment to a user by connecting a plurality of computers using the network is used in various internet service environments. With the participation of users, new and diverse Internet services are increasingly required for distributed file systems. Some of these services require not only the reading of files, but also the contents of files and the addition of contents, and are constantly changing to meet the needs of such users. In particular, it is very important to ensure that the distributed file system does not break the consistency of the metadata by multiple clients accessing the file at the same time, while providing the change and additional functions without compromising file read performance.

Since there are a number of clients accessing the same file simultaneously in the distributed file system, the metadata information must be consistently maintained so that the client can access the file data existing in various data servers correctly.

For example, if a file A is stored in the data servers D1, D2, and D3 and the client C1 that wants to read file A and the clients C2 and C3 that want to add to file A are present at the same time, And clients C2 and C3 will try to change the information of the metadata in the metadata server M. At this time, if the client C1 reads the block identifier of the last block information while the client C2 writes the block identifier to the last block information, the client C2 will be confused by the erroneous block identifier.

Accordingly, in order to solve such a problem, a method for controlling a plurality of clients simultaneously accessing the same file is required, and at the same time, the degradation of the performance should be minimized.

On the other hand, the distributed file system uses a mirror method to maintain the constant of the data. In this case, at least one copy of another data is stored in addition to the original file, and if an error or loss occurs in the original file, the original file is recovered using the previously stored copy. Therefore, the existing distributed file system must store additional copies with the original file, and these replicas have the same data size as the original file, so that the storage space of the original file in the distributed file system is at least twice as many A storage space is required.

As a method for solving such a problem, Japanese Laid-Open Patent Publication No. 10-1128998 (distributed file operation method using parity data) divides an original file to be input into at least one data without using a further copy, A plurality of pieces of divided data obtained by combining parity data with data are divided and stored into at least two or more servers so that waste of storage space according to replicas can be prevented. In addition, when an error or loss occurs in the original file, An erasure code method is disclosed in which the original file can be restored by using the parity data of the divided data stored in the erase code storage unit and the constants of the data can be maintained.

However, in the case of the erasure code method, it is possible to prevent waste of storage space according to the replica when the original file is large to some extent. However, when the original file is small, In addition, since the data is divided and stored in a plurality of servers in combination with the parity data, a complicated and many procedures are required to store the data, while the effect of the storage space is insignificant. Also, in the case of the erasure code method in the recovery of the original file, since it is necessary to receive all the corresponding data from a plurality of divided servers, the amount of information of the metadata stored in the metadata server (MDS) There is a problem that the process becomes complicated, for example, the data becomes large and the divided data needs to be combined again and recovered.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in an effort to solve the problems as described above, and it is an object of the present invention to provide a distributed file system that divides a large size data into a plurality of slices using an erasure code method, The present invention also provides a method of efficiently storing data in a distributed file system in which a copy having the same size as an original file is additionally stored using a mirror method.

According to an aspect of the present invention, there is provided an efficient data storage method in a distributed file system, including: distributing a distributed file system comprising at least two servers including a local storage having a storage space for storing data; A method for operating a file, the method comprising: (A) comparing a size of an original file to be input with a predetermined threshold; and (B) if the size of the original file is greater than a preset threshold, Dividing the file into a plurality of slices using at least two or more servers and storing the divided files in at least two servers; and (C) if the size of the original file is smaller than or equal to the set threshold, And further storing a small size replica of the same size as the file.

Preferably, the step (B) includes the steps of: dividing the input original file into at least one piece of data; generating at least one piece of divided data obtained by combining parity data with each of the divided pieces of data; Storing the metadata in a local storage in a server, and storing meta information including location information of the divided data stored in the local storage in a separate metadata server (MDS) .

Preferably, the generated partitioned data is generated with a codeword size corresponding to one frame of local storage including parity data through encoding.

Preferably, the step (C) includes the steps of: generating at least one copy of the original file input through the client; and transmitting the original file and the generated copy to two or more servers, respectively, And storing the meta information including location information of the original files and the replicas stored in the local storage in a separate metadata server (MDS).

The method of efficiently storing data in the distributed file system according to the present invention as described above has the following effects.

First, the large-sized data is divided into several slices by using an erasure code method, and is divided into at least two servers. By storing the data, You can eliminate the extra storage space associated with large replicas of data. Also, in this case, because there are a plurality of data servers involved, the throughput is increased due to the parallel effect.

Secondly, the data of small size can be saved by using the mirror method to save a copy of data having the same size as the original file, thereby minimizing the waste of storage space by the copy. In this case, since the number of participating data servers is smaller than the number of participating data servers in the erasure code, the read / write latency is shortened.

Third, it is possible to maintain the constant of the data by using the deletion code method for the large data and the mirror method using the small size data, so that it is possible to recover the error of the original file or the lost file flexibly have.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing the overall structure of a distributed file system according to an embodiment of the present invention; FIG.
2 is a flowchart illustrating an efficient data storage method in a distributed file system according to an embodiment of the present invention.
FIG. 3 is a view for explaining a distributed storage method of data using an erasure code method according to the present invention; FIG.
FIG. 4 is a diagram for explaining encoding for generating divided data in units of codeword sizes in FIG. 3; FIG.
5 is a view for explaining a method of distributing and storing data using a mirror system according to the present invention;

Other objects, features and advantages of the present invention will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

A preferred embodiment of an efficient data storage method in a distributed file system according to the present invention will be described with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. It is provided to let you know. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention. Therefore, various equivalents It should be understood that water and variations may be present.

FIG. 1 is a block diagram showing the overall structure of a distributed file system according to an embodiment of the present invention.

1, the distributed file system 100 includes a plurality of servers 110a to 110n and local storages 120a to 120n each having a storage space for storing data in the respective servers 100a to 11n do.

For example, each of the servers 110a to 110n constituting the distributed file system 100 may detect an event occurring on data stored in its local storages 120a to 120n. At this time, the data may mean a file or a directory. An event also means a change, deletion or creation of a file or directory.

Accordingly, when an event for the data stored in the second local storage 120b occurs, the second server 110b transmits the changed data through the event to the other servers of the distributed file system 110 excluding the second server 110b, The same data state can be maintained.

When an event is detected in the local storages 120a to 120n, which are hardware, through data input / output, the contents of the data changed by the event are notified to the application. The application may be a program running on the plurality of servers 110a to 110n constituting the distributed file system 100. [ That is, the distributed file system 100 senses an event through data input / output using a dedicated application implemented in each service-specific programming language provided by each of the servers 100a to 110n.

The distributed file system 100 may then use the already implemented data input / output based on the data type of the local storage 120a-120n determined by the operating system.

As described above, the distributed file system 100 can transmit data changed through the application to another server in order to reflect data change according to an event to another server. Conversely, the distributed file system 100 may receive data that has changed through the application from another server. At this time, the application can reflect the changed data received from the other server to the local storage of the server. The process of transmitting data and receiving data can be performed by each of the plurality of servers 110a to 110n constituting the distributed file system 100 in common.

An efficient data storage method in the distributed file system according to the present invention will now be described in detail with reference to the accompanying drawings. The same reference numerals as those in Fig. 1 designate the same members performing the same function.

2 is a flowchart illustrating an efficient data storage method in a distributed file system according to an embodiment of the present invention.

2, when the original file is input (S10), the size of the original file input from the storage API (not shown) in the distributed file system 100 is defined in advance And compares it with a threshold value (S20). In this case, the threshold may be a case where the original file is divided into a plurality of slices and divided into at least two or more servers, and a case where a replica of the same size as the original file is additionally stored in at least one server As a comparison, it is a value that can optimize the amount of calculation due to storage and recovery while minimizing the waste of storage space (local storage, metadata server (MDS)), preferably 512 Kb.

If the size of the original file is larger than the preset threshold value, the controller 20 divides the slice into a plurality of slices using an erasure code method and divides the divided slices into at least two or more servers (S30 to S60) .

Referring to FIG. 3, the original file input through the client is stored in a codeword size corresponding to one frame of the local storage in each server. (S30). Then, one or more divided data obtained by combining parity data with the divided data is generated (S40). That is, as shown in FIG. 4, when the codeword size unit of the local storage is 'i', the original file is divided into 'k' units except for the size 'm' Respectively. In this case, the size of 'm' for including the parity data in the divided data is increased according to the degree of importance of the data, and the size of 'm' is decreased when the importance is low , Which can be changed by the administrator. For reference, the large size of 'm' means that the number of bits of parity data increases, and the more parity bits, the better the data recovery capability.

The division data thus generated is transmitted to at least two or more servers and stored in the local storages 120a to 120n in the server (S50). Then, as shown in FIG. 3, the divided data sb1, sb2, ..., sbn generated by combining the parity data are transmitted to the plurality of servers 110a to 110n, respectively, The first local storage 120a stores the first partitioned data sb1 and the second local storage 120b stored in the second server 110b stores the second partitioned data sb2 and the third server 110c The third local storage 120c stores the third divided data sb3. In this manner, the n-th divided data sbn is stored in the nth local storage 120n in the n-th server 110n. At this time, parity data is stored in the last nth local storage 120n depending on the size of the parity data.

The distributed file system 100 stores meta information including location information of the divided data stored in the local storages 120a to 120n of the servers 110a to 110n in a separate metadata server (MDS) 130 (S60). If the size of the original file is larger than the set threshold value, the original file is divided into n pieces by using the erasure code method. The divided data obtained by combining the divided data with the parity data is stored in the local storage Respectively.

If the size of the original file is smaller than or equal to the preset threshold value (S20), a data copy having the same size as the original file is further stored using the mirroring method (S70 to S90).

5, at least one replica of the original file input through the client is generated at step S70. In the present invention, one original file and two copies are generated by applying a 1 + 2 mirror method. However, the present invention is not limited thereto. That is, as the number of replicas increases, the ability to recover data becomes better. Therefore, the more important the data is, the more the number of replicas is generated. Therefore, it should be noted that this can be changed by the administrator.

The original file and the generated replica are transmitted to two or more servers, respectively, and stored in the local storages 120a to 120c in the server (S80). 5, the original file stores the original file in the first local storage 120a in the first server 110a and the original file is stored in the second local storage 120b in the second server 110b, Stores the first copy in the third local storage 120c in the third server 110c, and stores the second copy in the third local storage 120c in the third server 110c.

The distributed file system 100 then transmits meta information including location information of original files and replicas stored in the local storages 120a to 120c of the servers 110a to 110c to a separate metadata server (MDS) (S90). If the size of the original file is smaller than or equal to the set threshold value, the original file and the replica of the original file are divided and stored in the local storage in the respective servers using the mirror method.

Accordingly, it is possible to maintain the constant of the data, thereby allowing a smooth recovery of the original file or the lost file, while minimizing the waste of the storage space.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It will be apparent to those skilled in the art that various modifications may be made without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (4)

A method for operating a distributed file system in a distributed file system comprising at least two servers including local storage having a storage space for storing data,
(A) comparing the size of an original file to be input with a predefined threshold,
(B) dividing the original file into a plurality of slices by using an erasure code method if the size of the original file is larger than a predetermined threshold, and dividing the divided files into at least two or more servers;
(C) if the size of the original file is smaller than or equal to the set threshold value as a result of the comparison, further storing the data copy having the same size as the original file using the mirror method,
At this time, the step (C)
Generating at least one replica of the original file input through the client,
Transmitting the original file and the generated replica to two or more servers, respectively, and storing the original files and local replicas in a local storage in the server;
And storing meta information including location information of original files and replicas respectively stored in the local storage in a separate metadata server (MDS). 2. The method of claim 1, wherein step (B)
Dividing the input original file into at least one or more data;
Generating at least one piece of divided data obtained by combining parity data with each of the divided data;
Transmitting the generated divided data to at least two or more servers and storing the divided data in local storage in a server,
And storing meta information including location information of the divided data stored in the local storage in a separate metadata server (MDS). 3. The method of claim 2,
Wherein the generated partitioned data is generated in a codeword size corresponding to one frame of local storage including parity data through encoding. delete

Images