KR101679303B1 - Asymmetric distributed file system and data processing method therefor - Google Patents

Abstract

The present invention relates to a metadata server, a data server, a replication server, an asymmetric distributed file system, and a data processing method thereof. The disclosed asymmetric distributed file system includes a data server for performing a write operation on original data, And a metadata server for allocating chunks (blocks) corresponding to the write locations of the original data and the replica data, and storing metadata of the original data and the replica data, wherein the meta data server and the data server And replication servers can be placed on separate layers and the original data write operations and replica data write operations can be dualized to support low latency and different RAIDs can be applied by placing the data server and the replica server on different layers The data server and the replica server are the same When compared to the prior art placed in an ear and to support more storage space by using a smaller number of computers, to the support for data error recovery facilitates there is an advantage to enhance the data security.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an asymmetric distributed file system,

The present invention relates to an asymmetric distributed file system, and more particularly, to an asymmetric distributed file system in which a metadata server, a data server, and a replica server are disposed on separate layers, data processing for data write operations and data read operations ≪ / RTI >

As noted, the asymmetric distributed file system is a system for separately storing and managing metadata and actual data of a file. Metadata is managed by a metadata server, and actual data is stored in a plurality of data servers. The metadata managed by the metadata server includes information about the data server in which the actual data is stored.

The metadata server and the data server are connected to each other via a network to have a distributed structure. Therefore, the client terminal is separated from the metadata of the file and the path for accessing the data. That is, in order to access the file, the client terminal first accesses the metadata of the file in the metadata server and obtains information about the data servers in which the actual data is stored. Input / output to the actual data is then performed through the data servers.

In an asymmetric distributed file system, file data is divided into data chunks of fixed size and distributed to data servers.

In order to solve the problem that the server can not input or output data when the server or the network fails, a replica of the chunk is created and stored in the other data servers. The number of replicas is decided considering the storage cost. By keeping replicas on different data servers, it is possible to distribute the access load from the users.

In the asymmetric distributed file system according to the related art, the metadata server and the data server are arranged in separate layers, respectively, and the replica is stored in data servers different from the data server in which the original is stored. In addition, we focused on the distributed generation of replicas in consideration of batch operations such as large-capacity data analysis, and operated an asymmetric distributed file system.

Therefore, when the data writing operation is performed according to the request of the client terminal, not only the time required for the write operation of the original data but also the additional time required for the write operation of the replica data occurs, so that it can not support the low latency There was a problem.

Disclosure of Invention Technical Problem [8] Accordingly, the present invention has been made in order to solve the problems of the related art, and it is an object of the present invention to provide a data server, a data server, and a replication server in a separate layer, Asymmetric distributed file system.

The present invention provides a metadata server, a data server, and a replica server constituting such an asymmetric distributed file system.

The present invention provides an asymmetric distributed file data processing method capable of supporting low latency.

According to a first aspect of the present invention, there is provided an asymmetric distributed file system comprising a data server for performing a write operation on original data, a replica server for performing a write operation on the replica data, And a metadata server that allocates the corresponding chunks (blocks) and stores metadata of the original data and the replica data.

Here, the data server and the replica server can perform the write operation on the original data and the write operation on the replica data asynchronously.

The data server and the replica server can apply different levels of RAID.

The replication server may apply RAID level 5 or RAID level 6.

As a second aspect of the present invention, a metadata server allocates a chunk (block) corresponding to a write location of a data server to perform a write operation on original data, and writes And stores metadata of the original data and the replica data.

Here, the metadata server may transmit a data write completion message to the client terminal when the data server completes a write operation on the original data.

As a third aspect of the present invention, a data server records the original data in the allocated chunk (block) when a meta data server allocates a chunk (block) corresponding to a write location for performing a write operation on the original data And when the metadata server allocates the chunk (block) corresponding to the write location for performing a write operation on the replica data, the metadata server transmits the original data to the replica server, The write operation can be performed.

According to a fourth aspect of the present invention, there is provided a duplication server, wherein when a metadata server allocates a chunk (block) corresponding to a write location for performing a write operation on replica data, And may perform a write operation on the replica data by receiving the data.

Here, the replica server may apply RAID at a level different from that of the data server.

The replication server may apply RAID level 5 or RAID level 6.

According to a fifth aspect of the present invention, there is provided a method of processing data in an asymmetric distributed file system, the method comprising: allocating a chunk (block) corresponding to a writing location of original data and replica data to a metadata server; Write operation of the replica data; performing a write operation on the replica data by the replica server; and storing the metadata of the original data and the replica data by the metadata server.

Here, the writing operation for the original data and the writing operation for the replica data can be performed asynchronously.

According to a sixth aspect of the present invention, there is provided a data processing method of a metadata server, comprising: allocating a chunk (block) corresponding to a write location of a data server to perform a write operation on original data; Allocating the chunk (block) corresponding to a write location of a replica server to be performed, and storing metadata of the original data and the replica data.

The data processing method may further include transmitting a data write completion message to the client terminal when the data server completes a write operation on the original data.

According to a seventh aspect of the present invention, there is provided a data processing method of a data server, comprising: receiving a chunk (block) corresponding to a write location for performing a write operation on original data from a metadata server; The method comprising: performing a write operation to write the original data in the replica data; and allocating the chunk (block) corresponding to the write location in which the metadata server performs the write operation to the replica data, And causing the replication server to perform a write operation on the replica data.

According to an eighth aspect of the present invention, there is provided a data processing method of a replica server, comprising: receiving a chunk (block) corresponding to a write location for performing a write operation on replica data from a metadata server; Receiving the original data from a data server which is allocated with the chunk (block) corresponding to a write location for performing a write operation to the original data and writes the original data; And performing a write operation of writing the replica data in the replica data.

According to the embodiment of the present invention, the metadata server, the data server, and the replica server are disposed on separate layers, respectively, and the original data write operation and the replica data write operation are duplexed to support low latency.

In addition, since the data server and the replica server can be arranged in different layers, different RAIDs can be applied. Therefore, compared to the prior art in which the data server and the replica server are arranged on the same layer, , And it is effective to enhance data security by facilitating support for data error recovery.

1 is a configuration diagram of an asymmetric distributed file system according to an embodiment of the present invention;
2 is a flowchart illustrating an asymmetric distributed file data processing method according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

Each block of the accompanying block diagrams and combinations of steps of the flowchart may be performed by computer program instructions. These computer program instructions may be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus so that the instructions, which may be executed by a processor of a computer or other programmable data processing apparatus, And means for performing the functions described in each step are created. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

1 is a configuration diagram of an asymmetric distributed file system according to an embodiment of the present invention.

As shown in the figure, the asymmetric distributed file system 100 includes a plurality of metadata servers 111 and 112 arranged in the layer 1, n data servers 120-1, 120-2, 120-3, ..., 120-n and m duplication servers 130-1, ..., 130-m arranged in the layer 3 and k client terminals 10-1, 10-2, 10-3, 10- 4,..., 10-k) through the network. For example, the number of nodes can be arranged as "k " n > (2 x m)" Although FIG. 1 shows an example in which two metadata servers 111 and 112 are arranged, one or more metadata servers may be arranged.

The function of each terminal or server constituting the asymmetric distributed file system will be described as follows.

The client terminal 10-1, 10-2, 10-3, 10-4, ..., 10-k accesses the file metadata from the metadata servers 111 and 112 while performing the client application, -1, 120-2, 120-3, ..., 120-n and / or the replica servers 130-1, ..., and 130-m.

The metadata servers 111 and 112 store and manage the metadata of all the files in the file system and manage all the data servers 120-1, 120-2, 120-3, ..., 120- 130-1, ..., 130-m.

In the asymmetric distributed file system 100 according to the embodiment of the present invention, original file data is divided into chunks of a predetermined size and distributed to the data servers 120-1, 120-2, 120-3, ..., 120- And replica file data is divided into chunks of a predetermined size and distributed to the replica servers 130-1, ..., and 130-m. The metadata servers 111 and 112 select a data server on which the original file data chunk is to be generated and also select the replica servers 130-1,..., And 130-m for which a replica file data chunk is to be generated. The file data chunks are distinguished by unique identifiers, and the identifier information for the generated file data chunks is stored and held in the metadata servers 111 and 112 as metadata information of the files.

The data servers 120-1, 120-2, 120-3, ..., 120-n store and manage original file data chunks, and the replica servers 130-1, ..., 130- . The data servers 120-1, 120-2, 120-3, ..., 120-n and the replica servers 130-1, ..., and 130-m periodically inform the metadata servers 111, Report the information.

The client terminals 10-1, 10-2, 10-3, 10-4, ..., 10-k, the metadata servers 111 and 112, the data servers 120-1, 120-2, ..., and 130-m are interconnected through a network and apply one RAID level among a plurality of Redundant Array of Inexpensive Disks (RAID) levels to create RAID volumes volume can be configured. For example, the data servers 120-1, 120-2, 120-3, ..., and 120-n can configure a volume by applying a level of excellent data read / write performance such as RAID level 0, (130-1, ..., 130-m) can constitute a volume by applying a level having an excellent reliability such as a RAID level 5, a RAID level 6, and the like.

RAID was first introduced as a requirement to create a large storage space by bundling several cheap disks, but in recent years the price has become cheaper and the term meaning has been faded and recently it is also called Redundant Array of Independent Disks.

RAID is aimed at disk redundancy and performance improvement, and can be divided into six levels (0, 1, 2, 3, 4, 5, 6) depending on the configuration. Different levels show different reliability and performance improvements. Because the disks that are grouped according to the level are used as one volume, they are called RAID volumes. The characteristics according to the RAID level are as follows.

(RAID 0)

RAID level 0 is the configuration of two or more volumes using two or more disks. It is simply called striping mode because the disks are listed for each volume. Generally, in RAID 0, when writing specific data, the number of volumes is divided and written in parallel to the same sector of the same disk. Therefore, the data read / write performance is greatly improved.

However, because the data is partitioned, only one disk that constitutes a volume can fail to recover even if it fails. You can think of systems that need fast speeds, but level 0 is undesirable if the system requires data stability.

(RAID 1)

RAID level 1 is an additional configuration of the same RAID volume. This is called mirroring mode because the volumes you configure are identical to each other. At this level, each time you read and write data to volume 0, the same operation is performed on volume 1 as well. Therefore, if a disk in a volume fails or fails, it will help to read / write normally by using disk of another volume.

The read performance can be improved because it can be divided and read when using multi-thread, but the performance is degraded because the write must be written in duplicate. They are often used when data reliability is important, such as for servers, but they are not preferred because they cost twice as much disk configuration costs.

(RAID 2)

RAID level 2 is a way to increase reliability while taking advantage of RAID level 0. Four volumes are configured in striping mode and an additional three volumes are used to store parity, the error correction code for the previous four volumes. Parity is generally calculated as a hamming code. At least three disks are required to configure RAID level 2. This is because additional disks are required for parity storage. A total of seven disks are used because they store separate parity between each volume (Volume 0 - Volume 1, Volume 1 - Volume 2, Volume 2 - Volume 3). With this configuration, if multiple disks fail, they can be recovered normally.

(RAID 3)

RAID level 3 is a slight improvement on the disadvantages of RAID level 2. Data storage for parity storage An additional volume of 1 was needed, but RAID 3 stores parity with only one volume. Thus, the additional cost of the disk is less. However, if the same location on each volume fails or fails at the same time, it is vague to recover.

A feature of RAID level 3 is that it generates parity at the byte level. Therefore, it is possible to perform parity calculation by simply XOR operation on the bytes of each volume. Also, at least three disks are required as the volume for parity storage.

(RAID 4)

RAID level 4 does not seem to be much different from RAID level 3. In reality, the configuration of the two levels is the same. The difference is that RAID level 3 divides the data in bytes and calculates the parity, while RAID level 4 divides the data in predetermined blocks and calculates the parity. Since the data is processed in block units, performance can be improved more than level 3. Level 4 also requires parity storage, so at least three disks are needed.

(RAID 5)

RAID level 5 is designed to improve the disadvantages of RAID level 4. In Level 2, the parity disk volume is under heavy load when data changes frequently because the parity disks belong to the same volume. Therefore, RAID level 5 is the partitioning of parity disks into each volume for performance efficiency. Level 5 requires parity, so you need at least three disks.

(RAID 6)

RAID level 6 is a more reliable configuration. In RAID level 5, it is hard to recover if two disks fail at the same time, but level 6 adds one additional parity disk to make it possible to recover even if there are no errors at the same time. Actually, there are two parity disks for each volume. This allows the generation of parity in the horizontal and vertical directions of the volume. Since additional parity disks are required, a minimum of four disks can be configured.

As we have seen, RAID level 5 and RAID level 6 provide high reliability, but the performance of the speed for read / write is relatively low. However, since the replica servers 130-1, ..., and 130-m of the present invention require higher reliability than the high-speed, RAID level 5 or RAID level 6 can be applied.

2 is a flowchart illustrating an asymmetric distributed file data processing method according to an embodiment of the present invention. In FIG. 2, the asymmetric distributed file data processing method is divided into a plurality of steps and listed in a time series in order to facilitate the understanding of the description, but each step may be performed in the same order or in the same order. For example, steps S203 and S211 may be performed simultaneously in the same order, and steps S207 and S215 may be performed simultaneously in the same order.

As described above, the asymmetric distributed file data processing method includes steps (S203 and S211) of allocating chunks (blocks) corresponding to the writing locations of the original data and the replica data to the metadata server arranged in the layer 1, A step S209 of the data server performing a write operation on the original data, and a step S209 of the metadata server transmitting a data write completion message to the client terminal when the data server completes the write operation on the original data, A step S213 of performing a write operation on the replica data by the replica server arranged in the layer 3, and a step S207 and S215 of storing the metadata of the original data and the replica data by the metadata server have.

Hereinafter, a data processing method using an asymmetric distributed file system will be described in a time-wise manner with reference to FIGS. 1 and 2. FIG.

First, when the client terminals 10-1, 10-2, 10-3, 10-4, ..., 10-k request storage of file data to the metadata servers 111 and 112, , 112) allocates a chunk (block) corresponding to the original data writing position (S203).

Then, the client terminals 10-1, 10-2, 10-3, 10-4, ..., 10-k and the data servers 120-1, 120-2, 120-3, ..., The original data is written in chunks (blocks) of the data servers 120-1, 120-2, 120-3, ..., and 120-n assigned by the metadata servers 111 and 112 (S205).

When the original data writing operation by the data servers 120-1, 120-2, 120-3, ..., and 120-n is completed, the metadata servers 111 and 112 store the metadata including the original data chunk (block) And stores the data (S207).

In addition, the metadata servers 111 and 112 generate a data write completion message indicating that data writing has been completed and send the data write completion message to the client terminals 10-1, 10-2, 10-3, 10-4, ..., 10-k (S209).

Next, the metadata servers 111 and 112 allocate chunks (blocks) corresponding to the replica data writing locations (S211).

The data servers 120-1, 120-2, 120-3, ..., 120-n and the duplication servers 130-1, ..., and 130- The replica data writing operation for recording the replica data in the chunks (blocks) of the servers 130-1, ..., and 130-m is performed (S213).

As described above, the write operation for the original data and the write operation for the replica data are performed in an asynchronous manner. When the write operation for the original data is completed, the client terminals 10-1, 10-2, 10-3, The data servers 120-1, 120-2, 120-3, ..., 120-n and the replica servers 130-1, ..., 130- m) send and receive data directly to each other to generate a duplicate.

Finally, when the original data writing operation by the data servers 120-1, 120-2, 120-3, ..., and 120-n is completed, the metadata servers 111 and 112 store replica data chunk (block) information And stores metadata including the metadata (S215).

As described above, in the embodiment of FIG. 2, the original data writing process of steps S203 to S207 and the replica data writing process of steps S211 to S215 are sequentially performed. However, And some sequences involved in the process of writing replica data may be performed concurrently. For example, chunks (blocks) corresponding to the original data writing position and chunks (blocks) corresponding to the replica data writing positions are simultaneously allocated (S203 and S211), and the original data writing operation and the replica data writing operation are sequentially performed S205, and S213), metadata including original data chunk (block) information and metadata including replica data chunk (block) information can be simultaneously stored (S207, S215).

On the other hand, when the client terminal 10-1, 10-2, 10-3, 10-4, ..., 10-k wants to perform a read operation on file data, the metadata server 111, (120-1, 120-2, 120-3, ..., 120-n) and / or the replication server (120-1) based on the chunk (block) 130-1, ..., 130-m.

The embodiment of the present invention is applicable to a data storage service server called a web hard and an internet hard, a service system using the same, a data processing method thereof, a cloud storage server providing virtualized storage resources through a network, And a data processing method using the same, and a plurality of data centers are operated in a wired communication or a wireless communication service system and a service method thereof. In order to improve data stability and performance, And the like.

10-1, 10-2, 10-3, 10-4, ... , 10-k: client terminal
111, 112: Metadata server
120-1, 120-2, 120-3, ... , 120-n: data server
130-1, ... , 130-m: Replication Server

Claims (16)

A data server for performing a write operation on the original data,
A replica server that performs a write operation to the replica data,
A metadata server,
The metadata server includes:
Wherein when a write operation of the original data is performed after storing a chunk (block) corresponding to a writing position of the original data to the data server, the metadata of the original data is stored, (Block) to the copy server, and when the write operation of the copy data is performed, the metadata of the copy data is stored,
Wherein the data server and the replication server are arranged in different layers,
Asymmetric distributed file system. The method according to claim 1,
Wherein the data server and the replica server execute asynchronously a write operation on the original data and a write operation on the replica data
Asymmetric distributed file system. delete The method according to claim 1,
The replication server may be configured to use RAID level 5 or RAID level 6
Asymmetric distributed file system. delete delete delete delete delete delete A step in which the metadata server allocates a chunk (block) corresponding to the writing position of the original data to the data server disposed in the first layer;
The data server performing a write operation on the original data;
The metadata server storing metadata of the original data;
The metadata server assigning a chunk (block) corresponding to a write location of the replica data to a replica server disposed in a second layer;
Performing a write operation on the replica data by the replica server;
Wherein the metadata server stores metadata of the replica data,
Wherein the data server and the replica server have different levels of RAID,
A method for processing data in an asymmetric distributed file system. 12. The method of claim 11,
A write operation for the original data and a write operation for the replica data are performed asynchronously
A method for processing data in an asymmetric distributed file system. delete delete delete delete

Images