KR20120027786A - Meta-data server, data server, replica server, asymmetric distributed file system, and data processing method therefor - Google Patents

Abstract

PURPOSE: A metadata server, a data server, a replicated server, an asymmetric distributed file system, and a data processing method are provided to support storing spaces through small number of computers by arranging a data server and a replicated server to different layers. CONSTITUTION: Data servers(120-1, 120-2, 120-3, 120-n) performs a writing operation about original data. Copy servers(130-1, 130-m) perform a writing operation about replicated data. Metadata servers allocate chunks corresponding to the writing location of replicated data and original data. The metadata servers store the original data and the metadata of the replicated data. The data servers and the replicated servers perform a writing operation about the original data.

Description

Metadata Servers, Data Servers, Replication Servers, Asymmetric Distributed File Systems, and Methods for Processing Data {META-DATA SERVER, DATA SERVER, REPLICA SERVER, ASYMMETRIC DISTRIBUTED FILE SYSTEM, AND DATA PROCESSING METHOD THEREFOR}

The present invention relates to an asymmetric distributed file system, and more particularly, an asymmetric distributed file system having a metadata server, a data server, and a replication server on separate layers, a metadata server, a data server, and a replication server constituting the same. And it relates to a data processing method for these data write operation and data read operation.

As is well known, an asymmetric distributed file system is a system that stores and manages metadata and actual data of a file separately. The metadata is managed in a metadata server and the actual data is distributed and stored in multiple data servers. The metadata managed by the metadata server includes information about the data server where the actual data is stored.

These metadata servers and data servers are distributed over a network. Therefore, the client terminal is separated from the file metadata and the path for accessing the data. That is, in order to access the file, the client terminal first accesses the metadata of the file in the metadata server to obtain information about the data servers in which the actual data is stored. Then input and output to the actual data is done through the data servers.

In an asymmetric distributed file system, file data is divided into data chunks of fixed size and distributed and stored in data servers.

In addition, if a server or network fails, in order to solve the problem of not being able to input / output data, a copy of the chunk is made and stored in other data servers. The number of copies is determined in consideration of the storage cost. By keeping the replica on multiple data servers, you can distribute the load of access from users.

The asymmetric distributed file system according to the prior art arranges the metadata server and the data server in separate layers, and the replicas are stored in different data servers from the data server where the original is stored. In addition, an asymmetric distributed file system was operated with a focus on distributed generation of replicas, mainly considering batch operations such as analyzing large amounts of data.

Therefore, when performing the data write operation according to the request of the client terminal, not only the time required for the write operation of the original data but also additional time required for the write operation of the replica data are generated and thus low-latency cannot be supported. There was a problem.

The present invention has been proposed to solve the above problems of the prior art, and the metadata server, the data server and the replication server are arranged in separate layers, and the original data write operation and the duplicate data write operation to support low latency Provide an asymmetric distributed file system.

The present invention provides a metadata server, a data server, and a replication server constituting such an asymmetric distributed file system.

The present invention provides an asymmetric distributed file data processing method capable of supporting low delay.

As a first aspect of the present invention, an asymmetric distributed file system includes a data server performing a write operation on original data, a replication server performing a write operation on replica data, and a write position of the original data and the replica data. It may include a metadata server that allocates a corresponding chunk (block) and stores metadata of the original data and the replica data.

Here, the data server and the replication server may perform a write operation on the original data and a write operation on the copy data asynchronously.

The data server and the replication server may apply different levels of RAID.

The replication server may apply RAID level 5 or RAID level 6.

As a second aspect of the present invention, the metadata server allocates a chunk (block) corresponding to a write position of a data server to perform a write operation on original data, and writes a replication server to perform a write operation on replica data. The chunk (block) corresponding to a location may be allocated, and metadata of the original data and the replica data may be stored.

Here, when the data server completes the write operation on the original data, the metadata server may transmit a data write completion message to a client terminal.

As a third aspect of the invention, a data server writes the original data to the allocated chunk (block) when a metadata server allocates a chunk (block) corresponding to a write position to perform a write operation on the original data. When the metadata server allocates the chunk (block) corresponding to the write position to perform the write operation on the replica data, the metadata server transmits the original data to the replica server so that the replica server sends the replica data to the replica data. To perform a write operation.

As a fourth aspect of the present invention, if the metadata server allocates a chunk (block) corresponding to a write position to perform a write operation on the replica data, the replication server receives the original data from the data server performing the write operation on the original data. The data may be received to perform a write operation on the copy data.

Here, the replication server may apply a RAID having a different level from that of the data server.

The replication server may apply RAID level 5 or RAID level 6.

According to a fifth aspect of the present invention, a data processing method of an asymmetric distributed file system includes a method in which a metadata server allocates chunks (blocks) corresponding to write positions of original data and replica data, and the data server is configured for the original data. The method may include performing a write operation, performing a write operation on the replica data by the replication server, and storing the original data and metadata of the replica data by the metadata server.

Here, the write operation on the original data and the write operation on the copy data may be asynchronously performed.

According to a sixth aspect of the present invention, a data processing method of a metadata server may include allocating a chunk (block) corresponding to a write position of a data server to perform a write operation on original data, and performing a write operation on replica data. The method may include allocating the chunk (block) corresponding to a write position of a replication server to be performed, and storing metadata of the original data and the replica data.

The data processing method may further include transmitting a data write completion message to a client terminal when the data server completes a write operation on the original data.

According to a seventh aspect of the present invention, a data processing method of a data server includes: receiving a chunk (block) corresponding to a write position to perform a write operation on an original data from a metadata server, and the allocated chunk (block). Performing a write operation of recording the original data in the; and if the metadata server allocates the chunk (block) corresponding to a write position to perform a write operation on the replica data, the original data is transmitted to a replication server. And causing the replication server to perform a write operation on the replica data.

According to an eighth aspect of the present invention, there is provided a data processing method of a replication server, the method comprising: receiving a chunk (block) corresponding to a write position to perform a write operation on a replica data from a metadata server, and source data from the metadata server. Receiving the original data from a data server performing a write operation of receiving the chunk (block) corresponding to the write position to perform a write operation on the original data, and the allocated chunk (block). And performing a write operation to record the copy data at.

According to the embodiment of the present invention, the metadata server, the data server, and the replication server may be disposed on separate layers, and the low latency may be supported by dualizing the original data write operation and the copy data write operation.

In addition, different RAIDs can be applied by placing data servers and replica servers on different layers, thus using more storage space with fewer computers compared to the prior art of placing data servers and replica servers on the same layer. It is possible to support the data error recovery, and it is easy to support data error recovery, thereby enhancing the data safety.

1 is a block diagram of an asymmetric distributed file system according to an embodiment of the present invention.
2 is a flowchart illustrating a method for processing asymmetric distributed file data according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and only the embodiments make the disclosure of the present invention complete, and the general knowledge in the art to which the present invention belongs. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the embodiments of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be based on the contents throughout this specification.

Each block of the accompanying block diagrams and combinations of steps of the flowchart may be performed by computer program instructions. These computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment such that instructions executed through the processor of the computer or other programmable data processing equipment may not be included in each block or flowchart of the block diagram. It will create means for performing the functions described in each step. These computer program instructions may be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular manner, and thus the computer usable or computer readable memory. It is also possible for the instructions stored in to produce an article of manufacture containing instruction means for performing the functions described in each block or flowchart of each step of the block diagram. Computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operating steps may be performed on the computer or other programmable data processing equipment to create a computer-implemented process to create a computer or other programmable data. Instructions that perform processing equipment may also provide steps for performing the functions described in each block of the block diagram and in each step of the flowchart.

In addition, each block or step may represent a portion of a module, segment or code that includes one or more executable instructions for executing a specified logical function (s). It should also be noted that in some alternative embodiments, the functions noted in the blocks or steps may occur out of order. For example, the two blocks or steps shown in succession may in fact be executed substantially concurrently or the blocks or steps may sometimes be performed in the reverse order, depending on the functionality involved.

1 is a block diagram of an asymmetric distributed file system according to an embodiment of the present invention.

As shown in FIG. 4, the asymmetric distributed file system 100 includes a plurality of metadata servers 111 and 112 disposed in layer 1, n data servers 120-1, 120-2, 120-3, ..., 120-n), including m replication servers 130-1, ..., 130-m arranged in Layer 3, and k client terminals 10-1, 10-2, 10-3, and 10-. 4, ..., 10-k) provides the storage and management of data through the network. For example, the number of nodes can be arranged under the condition of "k? N> (2 x m)". Although FIG. 1 illustrates an example in which two metadata servers 111 and 112 are arranged, one or more metadata servers may be arranged.

As described above, the functions of terminals or servers constituting the asymmetric distributed file system are as follows.

The client terminals 10-1, 10-2, 10-3, 10-4,..., 10-k access the file metadata from the metadata server 111, 112 while executing the client application and the data server 120. -1, 120-2, 120-3, ..., 120-n) and / or file data is input and output from the replication servers 130-1, ..., 130-m.

Metadata servers 111 and 112 store and manage metadata for all files in the file system, and all data servers 120-1, 120-2, 120-3, ..., 120-n and replication servers ( 130-1, ..., 130-m) to manage the status information.

In the asymmetric distributed file system 100 according to the embodiment of the present invention, the original file data is divided into chunk units having a predetermined size and distributed to the data servers 120-1, 120-2, 120-3, ..., 120-n. The replica file data is divided into chunk units having a predetermined size and distributed to the replica servers 130-1,..., 130-m. The meta data servers 111 and 112 select data servers on which the original file data chunks are to be generated, and also select replica servers 130-1,..., 130-m on which the duplicate file data chunks are to be generated. The file data chunks are distinguished by unique identifiers, and the identifier information for the generated file data chunks is stored and maintained in the metadata servers 111 and 112 as metadata information of the file.

Data servers 120-1, 120-2, 120-3, ..., 120-n store and manage the original file data chunks, and replica servers 130-1, ..., 130-m store replica file data chunks. Store and manage it. The data servers 120-1, 120-2, 120-3,..., 120-n and the replication servers 130-1,..., 130-m periodically transmit their status to the metadata server 111, 112. Report the information.

Client terminals 10-1, 10-2, 10-3, 10-4, ..., 10-k, metadata servers 111, 112, data servers 120-1, 120-2, 120-3, ..., 120-n), the replication servers 130-1, ..., 130-m are interconnected via a network, and apply a RAID volume by applying one RAID level among a plurality of levels of redundant array of inexpensive disks (RAID). volume). For example, the data servers 120-1, 120-2, 120-3,..., 120-n may configure a volume by applying a level at which data read / write performance is excellent, such as RAID level 0, and a replication server. 130-1,..., 130-m may configure a volume by applying a level of high reliability such as RAID level 5 or RAID level 6.

RAID first appeared as a request to bundle a lot of inexpensive disks to create a large amount of storage space. However, the terminology has faded in recent years as the price of disks has become cheaper, and it is also called Redundant Array of Independent Disks.

RAID is intended for disk redundancy and performance improvement, and can be divided into six levels (0, 1, 2, 3, 4, 5, 6) depending on the configuration. Different levels show different reliability and performance improvements. Grouped disks of different levels are called RAID volumes because they are used as a single volume. The characteristics of RAID levels are as follows.

(RAID 0)

RAID level 0 uses two or more disks to form two or more volumes. It is called striping mode because it simply lists the disks by volume. In general, when RAID 0 writes specific data, it divides by the number of volumes and writes them in parallel to the same sector on the same disk. This greatly improves the read / write performance of the data.

However, because the data is partitioned, you cannot recover data if only one disk that constitutes the volume fails. You may want to consider a system that requires high speed, but level 0 is not recommended for systems that require data stability.

(RAID 1)

RAID level 1 is an additional configuration of the same RAID volume. It is called mirroring mode because the configured volumes are the same. At this level, whenever you read and write data to volume 0, the same operation is performed on volume 1. Therefore, if a disk in a volume fails or fails, the disk in another volume can be used for normal read / write operations.

Read performance can be improved because multithreaded reads can be split, but writes are redundant, resulting in poor performance. It is often used when data reliability is important, such as for servers, but it is not preferred because it costs twice the disk configuration.

(RAID 2)

RAID level 2 is a way to increase reliability while taking advantage of RAID level 0. Four volumes are configured in striping mode and an additional three volumes are used to store parity, the error correction code for the previous four volumes. Parity is generally calculated as a hamming code. At least three disks are required to configure RAID level 2. This is because additional disks are required for parity storage. A total of seven disks are used because they separately store parity between each volume (volume 0-volume 1, volume 1-volume 2, volume 2-volume 3). With this configuration, you can recover normally if multiple disks fail.

(RAID 3)

RAID level 3 is a minor improvement over RAID level 2. Although parity storage required an additional volume of data storage volume -1, RAID 3 stores parity with only one volume. Therefore, the additional cost of the disk is less. However, if the same location on each volume fails or fails at the same time, it is difficult to recover.

A feature of RAID level 3 is that it creates parity at the byte level. Therefore, parity can be calculated by simply XORing the bytes of each volume. Also, at least three disks are required for the parity storage.

(RAID 4)

RAID level 4 looks no different than RAID level 3. In fact, the two levels of configuration are identical. The difference is that RAID level 3 partitions data in bytes and calculates parity, whereas RAID level 4 partitions data in predetermined blocks and calculates parity. Because the data is processed in units of blocks, performance can be improved more than level 3. Level 4 also requires parity storage, so at least three disks are required.

(RAID 5)

RAID level 5 is designed to improve the shortcomings of RAID level 4. At level 2, parity disks belong to the same volume, so if the data changes frequently, the parity disk volume will be heavily loaded. Therefore, partitioning parity disks into each volume is RAID level 5 for performance efficiency. Level 5 also requires parity, so at least three disks are required.

(RAID 6)

RAID level 6 is a more based configuration. In the case of RAID level 5, if two disks fail at the same time, it is difficult to recover, but level 6 adds one more parity disk to make it possible to recover at the same time. In fact, each volume has two parity disks. This allows parity in the horizontal and vertical directions of the volume. An additional parity disk is required, so at least four disks are required for configuration.

As we have seen, RAID level 5 and RAID level 6 provide high reliability, but the performance in terms of speed for read / write is relatively low. However, the replication servers 130-1,..., 130-m of the present invention can apply RAID level 5 or RAID level 6 because high reliability is required rather than high speed.

2 is a flowchart illustrating a method for processing asymmetric distributed file data according to an embodiment of the present invention. In FIG. 2, the asymmetric distributed file data processing method is listed in time series for dividing the process into a plurality of stages for clarity of understanding, but each stage may be changed in order or may be performed simultaneously in the same order. For example, step S203 and step S211 may be performed simultaneously in the same order, and step S207 and step S215 may also be performed simultaneously in the same order.

As described above, the asymmetric distributed file data processing method includes a step (S203, S211) of assigning chunks (blocks) corresponding to write positions of original data and replica data by a metadata server arranged in Layer 1, and in Layer 2; (S205) the arranged data server performs a write operation on the original data, and when the data server completes the write operation on the original data, the metadata server transmits a data write completion message to the client terminal (S209). And performing a write operation on the replica data by the replication server arranged in the layer 3 (S213), and storing the metadata of the original data and the replica data by the metadata server (S207 and S215). have.

Hereinafter, a data processing method using an asymmetric distributed file system will be described in time series with reference to FIGS. 1 and 2.

First, when the client terminal 10-1, 10-2, 10-3, 10-4,..., 10-k requests the metadata server 111, 112 to store file data, the metadata server 111 (112) allocates a chunk (block) corresponding to the original data write position (S203).

Then, the client terminals 10-1, 10-2, 10-3, 10-4, ..., 10-k and the data servers 120-1, 120-2, 120-3, ..., 120-n Performs an original data write operation of writing original data into chunks (blocks) of the data servers 120-1, 120-2, 120-3, ..., 120-n allocated by the metadata servers 111 and 112. (S205).

When the original data write operation by the data servers 120-1, 120-2, 120-3,..., 120-n is completed, the metadata servers 111 and 112 perform meta-data including original data chunk (block) information. The data is stored (S207).

In addition, the metadata servers 111 and 112 generate a data write completion message indicating that data writing is completed, and send the generated data write completion message to the client terminals 10-1, 10-2, 10-3, 10-4, ..., 10-k. It transmits (S209).

Next, the metadata servers 111 and 112 allocate chunks (blocks) corresponding to the copy data write positions (S211).

Then, the data servers 120-1, 120-2, 120-3,..., 120-n and the replication servers 130-1,..., 130-m are replicated assigned by the metadata servers 111, 112. The replica data write operation for recording the replica data in the chunk (block) of the servers 130-1, ..., 130-m is performed (S213).

As such, the write operation on the original data and the write operation on the replica data are performed asynchronously by dualizing, and as soon as the write operation on the original data is completed, the client terminal 10-1, 10-2, 10-3, 10- 4, ..., 10-k to inform the completion of the data write operation, the data server (120-1, 120-2, 120-3, ..., 120-n) and the replication server (130-1, ..., 130- m) transmits and receives data directly to each other to create a copy.

Finally, when the original data write operation by the data servers 120-1, 120-2, 120-3, ..., 120-n is completed, the metadata servers 111 and 112 may copy the replica data chunk (block) information. Including the meta data is stored (S215).

As described above, the embodiment of FIG. 2 illustrates a case in which the original data writing process of steps S203 to S207 and the copy data writing process of steps S211 to S215 are sequentially performed, but some orders included in the original data writing process are illustrated. And some of the sequences included in the copy data write process may be performed simultaneously. For example, allocating a chunk (block) corresponding to the original data write position and a chunk (block) corresponding to the replica data write position (S203, S211), and performing the original data write operation and the replica data write operation sequentially ( S205 and S213 may simultaneously store metadata including original data chunk (block) information and metadata including duplicate data chunk (block) information (S207 and S215).

On the other hand, when the client terminal 10-1, 10-2, 10-3, 10-4, ..., 10-k wants to perform a read operation on the file data, the corresponding file is transmitted to the metadata server 111 or 112. Request metadata, and access the metadata and based on the chunk (block) information contained therein, the data server 120-1, 120-2, 120-3, ..., 120-n and / or the replication server ( 130-1, ..., 130-m) to read the file data.

An embodiment of the present invention, a data storage service server referred to as terms such as web hard and internet hard, a service system using the same, a data processing method thereof, and a cloud storage server providing virtualized storage resources according to a user's request. And the service system, its data processing method, and multiple data centers, which are common to operate a wired or wireless communication service system and its service method. It can be used in the technical field to create and manage.

10-1, 10-2, 10-3, 10-4,... , 10-k: client terminal
111, 112: metadata server
120-1, 120-2, 120-3,... , 120-n: data server
130-1,... , 130-m: replication server

Claims (16)

A data server performing a write operation on the original data,
A replication server that writes to replica data;
A metadata server for allocating chunks (blocks) corresponding to write positions of the original data and the replica data, and storing metadata of the original data and the replica data;
Asymmetric Distributed File System. The method of claim 1,
The data server and the replication server perform asynchronous write operations on the original data and write operations on the replica data.
Asymmetric Distributed File System. The method of claim 1,
The data server and the replication server, applying different levels of RAID
Asymmetric Distributed File System. The method according to claim 1 or 3,
The replication server, applying a RAID level 5 or a RAID level 6
Asymmetric Distributed File System. Allocate a chunk (block) corresponding to a write location of a data server to perform a write operation on the source data, and allocate the chunk (block) corresponding to a write location of a replication server to perform a write operation to replica data. Storing metadata of the original data and the replica data.
Metadata Server. The method of claim 5, wherein
The metadata server transmits a data write completion message to a client terminal when the data server completes a write operation on the original data.
Metadata Server. When the metadata server allocates a chunk (block) corresponding to a write position to perform a write operation on the original data, the metadata server performs a write operation of writing the original data to the allocated chunk (block). Allocating the chunk (block) corresponding to a write position to perform a write operation on the replica data transmits the original data to a replication server so that the replication server performs a write operation on the replica data.
Data server. When the metadata server allocates a chunk (block) corresponding to a write position to perform a write operation on the replica data, the original data is received from the data server performing the write operation on the original data, and the write operation is performed on the replica data. To do
Replication server. The method of claim 8,
The replication server applies a RAID level different from that of the data server.
Replication server. The method according to claim 8 or 9,
The replication server, applying a RAID level 5 or a RAID level 6
Replication server. The metadata server assigns chunks (blocks) corresponding to the write locations of the original data and the replica data;
A data server performing a write operation on the original data;
A replication server performing a write operation on the replica data;
Storing, by the metadata server, metadata of the original data and the replica data.
How data is processed in an asymmetric distributed file system. The method of claim 11,
Asynchronously performing a write operation on the original data and a write operation on the copy data.
How data is processed in an asymmetric distributed file system. Allocating a chunk (block) corresponding to a write location of a data server to perform a write operation on the original data;
Allocating the chunk (block) corresponding to the write position of the replica server to perform a write operation on the replica data;
Storing metadata of the original data and the replica data;
How the metadata server handles data. The method of claim 13,
The data processing method may further include transmitting a data write completion message to a client terminal when the data server completes a write operation on the original data.
How the metadata server handles data. Receiving a chunk (block) corresponding to a write position to perform a write operation on the original data from the metadata server;
Performing a write operation of writing the original data into the allocated chunk (block);
When the metadata server allocates the chunk (block) corresponding to a write position to perform a write operation on the replica data, the metadata server transmits the original data to the replica server to perform a write operation on the replica data. Comprising the steps of
How data servers process data. Receiving a chunk (block) corresponding to a write position to perform a write operation on the replica data from the metadata server;
Receiving the chunk (block) corresponding to a write position to perform a write operation on the original data from the metadata server, and transmitting the original data from a data server performing a write operation of writing the original data;
Performing a write operation to write the replica data to the allocated chunk (block).
How data is processed by the replication server.

Images