KR20120090320A - Method for effective data recovery in distributed file system - Google Patents

Abstract

PURPOSE: An efficient data recovering method in a distributed file system is provided to efficiently reduce recovery task time by configuring data servers as a pipeline. CONSTITUTION: If a first data server is detected through a metadata server, location information of data servers is detected for storing recovery data of the first data server(S200). A recovery task requests recovery data to the data servers(S300). If a recovery task of the data server is completed through recovery operation, the recovery completion of the data server is reported to the metadata server(S400).

Description

Method for effective data recovery in distributed file system

The present invention relates to a data recovery method in a distributed file system, and more particularly, to a method for efficient data recovery in a distributed file system in which data is divided into slices and the divided data are stored in at least two servers. It is about.

Most of the data stored in the conventional storage environment was business-related data generated by companies or institutions. However, due to the rapid development of Internet technology, the storage rate of multimedia data such as blogs, pictures, and videos is rapidly increasing. . In particular, large portal companies that provide Internet services at home and abroad are newly generating, storing, and managing several terabytes (TBs) of data every month. However, the existing storage structure environment has many problems in storage scalability and ease of management, so it is not sufficient to replace the ever-changing service environment.

Recent technological advances in storage systems or file systems are due to improvements in scalability and performance of storage systems. Specifically, in terms of the file system structure, some systems have separated the data input / output path of the file and the metadata management path of the file to increase the scalability and performance of the distributed storage system. By applying this structure, the client system can directly access the storage devices, and by distributing the metadata, the bottleneck caused by frequent file metadata access can be eliminated to increase the storage scalability.

Enterprise-class storage solutions built on this architecture include IBM's StorageTank, Panasas' ActiveScale Storage Cluster, and cluster filesystems luster, and Google's Google Filesystem. In particular, Google Filesystem has increased availability by replicating block data for one file to multiple data servers.

In this network-based distributed file system environment, client file systems, metadata servers, and data servers communicate over a network to provide input and output of data. To access a specific file, the client obtains the location information of the block in which the actual data of the file is stored from the metadata server, and then accesses the data server where the block is located and reads the data of the block.

On the other hand, in the case of data corruption and failures such as server (data server, metadata server) or disk failure in such a distributed file system, detecting and recovering it quickly and accurately greatly affects the performance of the distributed file system. It is an important issue.

In order to prepare for such a failure in the distributed file system, generally, in a distributed file system environment that supports multiple replication, the failed blocks are repaired using the following methods within the range to ensure the minimum availability.

In the first method, all block information is loaded into the memory of the metadata server, and when a failure situation occurs, the block is recovered from the memory after collecting the failed block information.

The second method is to store all block information in a separate database, recover the block after collecting the block information from the database when a failure occurs. In detail, a method of constructing a dedicated database for storing block information in a metadata server and editing and managing a database whenever a change in a block occurs.

The third method is to recover blocks after collecting all block information by searching all metadata whenever a failure occurs without managing separate block information.

As such, a variety of recovery methods have been studied in preparation for a failure of distributed file systems, and a more efficient failure recovery will be possible only when a recovery method suitable for such a new structure is applied to a table structure of a newly researched and designed distributed file system.

Accordingly, the present invention has been made to solve the above problems, and efficiently divides the data into several slices and efficiently distributes the divided data to at least two servers. Its purpose is to provide a method for restoring data.

Another object of the present invention is a data server participating in data recovery constitutes one pipeline, recovers data input from the previous data server, and outputs the data to the data server of the next stage. It is to provide an efficient data recovery method in a distributed file system that can save time.

A feature of an efficient data recovery method in a distributed file system according to the present invention for achieving the above object is (A) if a failed first data server is detected through the metadata server, Detecting location information of data servers storing recovery data to be reconstructed for recovery; (B) requesting recovery data from the data servers where the location information is detected, receiving recovery data, and receiving recovery data; Performing a recovery operation through a recovery operation using the data; and (C) completing a recovery operation of the failed data server through the recovery operation, and notifying the metadata server that the recovery of the data server is completed. It consists of including.

Preferably, the step (B) includes providing the location information of the detected recovery data servers to any data server in charge of failure recovery, and the recovery data servers holding the recovery data based on the provided location information. Receiving a recovery data request command to receive the recovery data held in the local storage of the corresponding recovery data server, receiving the recovery data until failure recovery of the first data server is possible, and receiving the received recovery data. After reconstructing, characterized in that it comprises a step of performing a recovery operation.

Preferably, the step (B) includes transmitting a recovery data request command to the recovery data servers from which the location information is detected, and recovering data held in local storage in each recovery data server to which the recovery data request command has been transmitted. And receiving the recovery data until the failure of the first data server is possible, reconstructing the input recovery data, and performing a recovery operation.

Preferably, the step (B) includes providing location information of the detected recovery data servers to the failed first data server and providing each recovery data server based on location information of the provided recovery data servers. Configuring one pipeline, transmitting pipeline information including a recovery data request command to a next stage recovery data server based on the configured pipeline, and based on the transmitted pipeline information Reading the recovery data held in local storage and transmitting the recovery data and pipeline information together to a recovery data server, which is the next stage of the pipeline, and having the recovery data transmitted from the previous recovery data server. After reconstructing the retained data, performing a recovery operation Transmitting the recovery operation data and the pipeline information together as a result of the recovery operation to the data server, which is the next stage of the pipeline, through the pipeline processing; and reconstructing the transmitted recovery operation data with the retained data owned by And transmitting the recovery operation data and the pipeline information, which are the result of the recovery operation, to the data server, which is the next stage of the pipeline, through the pipeline processing while performing the recovery operation, and the next stage based on the configured pipeline information. And continuing until the data server does not exist.

Preferably, the step (B) includes providing the location information of the detected recovery data server to any one of the recovery data servers from which the location information is detected, and the location information of the provided recovery data servers. Comprising a step of configuring a pipeline consisting of each recovery data server based on the, and based on the configured pipeline information to read the recovery data held in its own local storage and the recovery data server, which is the next stage of the pipeline Transmitting the recovered data and pipeline information together, and reconstructing the transmitted recovered data with retained data owned by the user, and performing a recovery operation to recover to the next server of the pipeline through a pipeline process. Restore operation data and pipeline as the result of the operation Transmitting the information together, reconstructing the transmitted recovery operation data with the retained data owned by the user, and performing a recovery operation to perform a recovery operation to a data server that is the next stage of the pipeline. And transmitting the recovery operation data and the pipeline information together, and continuing until the next stage of the data server does not exist based on the configured pipeline information.

An efficient data recovery method in the distributed file system according to the present invention as described above is divided into a plurality of slices (slice), and in a distributed file system having a new structure for storing the divided data to at least two servers Can recover data efficiently. In particular, it is possible to efficiently reduce the recovery work time by performing a simultaneous work by configuring a single pipeline of multiple data servers participating in data recovery.

In addition, the recovery time, as well as the efficient use of network resources can reduce the effect of the load on the service due to the recovery operation.

1 is a block diagram showing the overall structure of a distributed file system according to an embodiment of the present invention
2 is a diagram illustrating a distributed storage method of data using parity data according to the present invention.
3 is a diagram illustrating a distributed storage method of data according to an embodiment of the present invention.
4A, 4B and 5A, 5B are schematic diagrams illustrating an efficient data recovery apparatus of a distributed file system according to the present invention.
4C and 5C are timing diagrams showing a recovery operation time during a data recovery operation according to the embodiment.
6 is a block diagram showing a data server structure in a distributed file system according to the present invention.
7 is a flowchart illustrating an efficient data recovery method in a distributed file system according to an embodiment of the present invention.
8 to 9 are flowcharts for explaining a method for receiving and recovering recovered data in the method for recovering data according to the present invention.
10 to 11 are diagrams for explaining a recovery operation method applying Cosi Reed-Solomon in the data recovery method of the present invention

Other objects, features and advantages of the present invention will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

A preferred embodiment of an efficient data recovery method in a distributed file system according to the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various forms, and only the present embodiments are intended to complete the disclosure of the present invention and to those skilled in the art to fully understand the scope of the invention. It is provided to inform you. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiment of the present invention and do not represent all of the technical idea of the present invention, various modifications that can be replaced at the time of the present application It should be understood that there may be equivalents and variations.

1 is a block diagram showing the overall structure of a distributed file system according to an embodiment of the present invention.

As shown in FIG. 1, the distributed file system 100 includes a plurality of data servers 110a through 110n and local storage 120a through a separate storage space for storing data in each of the data servers 110a through 110n. 120n).

For example, each of the data servers 110a to 110n constituting the distributed file system 100 may detect an event occurring for data stored in its local storage 120a to 120n. In this case, the data may mean a file or a directory. In addition, an event means a change, deletion, or creation of a file or directory.

Therefore, when an event for data stored in the second local storage 120b occurs, the second data server 110b transmits the changed data through the event to other servers of the distributed file system 100 except for the entire data, thereby providing overall data. The same data state can be maintained between the servers.

When an event is detected in hardware local storage 120a to 120n through data input / output, the application notifies the contents of data changed by the event. In this case, the application may mean a program that operates in the plurality of data servers 110a to 110n constituting the distributed file system 100. That is, the distributed file system 100 detects an event through data input / output using a dedicated application implemented in a programming language for each service provided by each of the data servers 110a to 110n.

Then, the distributed file system 100 may use data input / output already implemented based on the data type of the local storages 120a to 120n determined by the operating system.

As such, the distributed file system 100 may transmit the changed data to another server through an application in order to reflect the data change according to an event in another data server. In contrast, the distributed file system 100 may receive data from another data server about the changed data through an application. In this case, the application may reflect the changed data received from another data server to the local storage of the corresponding data server. The process of transmitting data and receiving data may be performed by each of the plurality of data servers 110a to 110n constituting the distributed file system 100 in common.

In the data distribution method, as shown in FIG. 2, the original file input through the client is divided into codeword size units corresponding to one frame of local storage in each data server.

3 is a diagram illustrating a distributed storage method of data according to an embodiment of the present invention.

As shown in Fig. 3, one original file is divided into d pieces so that d ₁ to d _d divided data are sequentially arranged. And p parity data (three (p1 to p3) parity data are described herein for ease of explanation) according to the divided data. At this time, the d + p is preferably equal to or less than the number of data servers.

As described above, the d + p pieces of divided data sb1, sb2, .., and sbn divided into parity data are transmitted to the plurality of data servers 110a to 110n, respectively, to generate the first data server 110a. The first divided data sb1 is stored in the first local storage 120a, the second divided data sb2 is stored in the second local storage 120b in the second data server 110b, and the third data server ( Third partitioned data sb3 is stored in the third local storage 120c in 110c. In this manner, the nth divided data sbn is stored in the nth local storage 120n in the nth data server 110n at the end.

In addition, the distributed file system 100 may store meta information including location information of partitioned data stored in the local storages 120a through 120n of the data servers 110a through 110n, in a separate metadata server (MDS) 130. Save it.

As described above, in the distributed storage method of data using parity data according to the present invention, one original file is divided into d pieces and stored in local storage in each data server.

Referring to the accompanying drawings, an efficient data recovery method in the distributed file system according to the present invention configured as described above is as follows.

7 is a flowchart illustrating an efficient data recovery method in a distributed file system according to an embodiment of the present invention.

Referring to FIG. 7, first, a data server 110 having a failure is detected through the metadata server 130 (S100). In this case, failure detection of the data server 110 may include a case in which damaged data is found during a read operation, a failure occurs due to a failure in the data server 110 or the disk (local storage) 120 itself, and metadata. Periodic checks with the data server 110 through the server 130 may result in data that does not match the 'checksum', network disconnection, abnormal termination of the data server process, power failure, and the like.

Subsequently, when the failed data server 110 is detected, the metadata server 130 detects locations of the data servers that store the recovery data to be reconstructed for recovery from the failure of the detected data server 110 (S200). ). At this time, the metadata server 130 is an area for managing a file namespace tree of a file and expresses a hierarchical structure of each directory and file, and attributes of a file such as name, size, authority, and location information of each file And stores and manages information about files. Therefore, the metadata server 130 manages all data information managed by all data servers, and may collect recovery information on failure of a specific data server.

The data server detected for recovery by the metadata server 130 transmits normal recovery data to be used for recovery, and the data server to be recovered receives it. Then, the recovery operation is performed through the recovery operation using the received recovery data (S300). In this case, a method of receiving and recovering recovery data may be performed through various methods, and a detailed description thereof will be described below with reference to the accompanying drawings.

The recovery operation performed in this manner completes the recovery operation of the failed data server 110, and notifies the metadata server 130 that the recovery of the corresponding data server 110 is completed (S400).

Meanwhile, a method of receiving and restoring recovery data will now be described with reference to the accompanying drawings.

1st Example

8 is a flowchart illustrating a method of receiving and restoring restoration data according to the first embodiment in a data restoration method of the present invention, and FIG. 4A illustrates a method of receiving and restoring restoration data in a data restoration method of the present invention. A first embodiment for explanation.

Referring to the drawings, first, the metadata server 130 is a recovery data server (second data server, storing recovery data for recovering a failure of the failure data server (first data server) 110a). When the positions of the fourth data server, the fifth data server, ..., the mth data server, ..., the nth data server 110b, 110d, 110e, 110f and 110h are detected (S200), a failure occurs. The first data server 110a provides location information of the detected recovery data servers. In this case, when the location information is requested from the first data server 110a to the metadata server 130, the location information detected by the first data server 110a may be provided, or a separate information may be provided. If a failure is detected without requesting location information, the detected location information may be provided to the corresponding first data server 110a.

Subsequently, the first data server 110a is connected to the recovery data servers 110b, 110d, 110e, 110f, and 110h that hold the recovery data based on the location information provided through the metadata server 130. Send a command for requesting recovery data.

Then, each recovery data server reads the recovery data held in the local storage in the server using the recovery data request command transmitted from the first data server 110a and transmits it to the first data server 100a (S301). In this case, the transmission of the recovery data may be performed step by step in consideration of the system situation of each data server. That is, the step-by-step progression is to first complete all of the recovery data held by the second data server 110b to the first data server 110a, and then to recover all the recovery data held by the fourth data server 110d. The data is transmitted to the first data server 110a. As such, when the transmission of the recovery data held in one data server 110a is completed, the recovery data held in the next data server is sequentially performed. In this case, the transmission may proceed in parallel. However, since network transmission capacity becomes a bottleneck, it proceeds sequentially or in parallel, but the difference is not so great.

On the other hand, when the recovery data is sequentially transmitted from each of the recovery data servers, the first data server 110a searches for recovery data as much as necessary to recover its failure, and inputs all the recovery data necessary for the failure recovery. Until the recovery data is received (S302).

After reconstructing the inputted recovery data, a recovery operation is performed (S303). In this case, the recovery operation is performed through the Cauchy Reed-Solomon method. Meanwhile, the recovery operation may be performed by the Reed-Solomon method.

Referring to the recovery operation method in more detail, as shown in FIG. 10, the first data server 110a generates a Cauchy bit matrix for recovery based on the received recovery data, and then generates the Cauchy bit matrix. The matrix and the input data are read from the decoder to perform a recovery operation, and then divided into appropriate chunks. The deleted data is then recovered using all the data partitioned to the appropriate size. This is a well-known technique for Kosi Reed-Solomon's recovery operation method, which effectively divides data into multiple slices and efficiently recovers data in a distributed file system with a new structure that stores the divided data on at least two servers. It is characterized by using a known recovery operation method such as Kosi Reed-Solomon.

Second Example

8 is a flowchart illustrating a method of receiving and restoring restoration data according to a second embodiment in a data restoration method of the present invention, and FIG. 4B is a method of receiving and restoring restoration data in a data restoration method of the present invention. A second embodiment for explanation.

Referring to the drawings, first, the metadata server 130 is a recovery data server (second data server, storing recovery data for recovering a failure of the failure data server (first data server) 110a). When the positions of the fourth data server, the fifth data server, ..., the mth data server, ..., the nth data server 110b, 110d, 110e, 110f and 110h are detected (S200), the detected recovery is performed. The recovery data request command is transmitted to the data servers 110b, 110d, 110e, 110f, and 110h.

Then, each recovery data server reads the recovery data held in the local storage in the server using the recovery data request command transmitted from the metadata server 130 and transmits it to the first data server 100a (S301). In this case, the transmission of the recovery data may be performed step by step in consideration of the system situation of each data server. That is, the step-by-step progression is to first complete all of the recovery data held by the second data server 110b to the first data server 110a, and then to recover all the recovery data held by the fourth data server 110d. The data is transmitted to the first data server 110a. As such, when the transmission of the recovery data held in one data server 110a is completed, the recovery data held in the next data server is sequentially performed. In this case, the transmission may proceed in parallel. However, since network transmission capacity becomes a bottleneck, it proceeds sequentially or in parallel, but the difference is not so great.

On the other hand, when the recovery data is sequentially transmitted from each of the recovery data servers, the first data server 110a searches for recovery data as much as necessary to recover its failure, and inputs all the recovery data necessary for the failure recovery. Until the recovery data is received (S302).

After reconstructing the inputted recovery data, a recovery operation is performed (S303). In this case, the recovery operation is performed through the Cauchy Reed-Solomon method described in FIG. Meanwhile, the recovery operation may be performed by the Reed-Solomon method.

FIG. 4C is a timing diagram showing a recovery operation time during a data recovery operation in the first and second embodiments. FIG.

As described above, as the first data server 110a receives the recovery data sequentially from each of the recovery data servers, the recovery work time may be represented by Equation 1 as shown in FIG. 4C.

[Equation 1]

Recovery work time = (Recovery data transfer time per recovery server) × (Participation recovery data server) + (Recovery time)

As shown in Equation 1, the total recovery operation time is inputted from each recovery data server in the first data server 110a in the total data transmission time having the product of the number of participating recovery data servers and each recovery data transmission time. The time required for the recovery operation processing time to recover the failure using the recovery data is required.

A method for further reducing the recovery operation processing time in the failure recovery method described in the first and second embodiments will be described in the third and fourth embodiments below.

Third Example

9 is a flowchart illustrating a method of receiving and recovering recovered data according to a third embodiment in a data recovery method of the present invention, and FIG. 5A illustrates a method of receiving and recovering recovered data in a data recovery method of the present invention. A third embodiment for explanation.

Referring to the drawings, first, the metadata server 130 is a recovery data server (second data server, storing recovery data for recovering a failure of the failure data server (first data server) 110a). When the positions of the fourth data server, the fifth data server, ..., the mth data server, ..., the nth data server 110b, 110d, 110e, 110f and 110h are detected (S200), a failure occurs. The first data server 110a provides location information of the detected recovery data servers. In this case, when the location information is requested from the first data server 110a to the metadata server 130, the location data of the recovery data servers may be provided with the location information detected by the corresponding first data server 110a, or separately. If a failure is detected without the request for the location information, the detected location information may be provided to the corresponding first data server 110a.

Subsequently, the first data server 110a configures one pipeline composed of each recovery data server based on the location information provided through the metadata server 130. The first data server 110a transmits pipeline information including a recovery data request command to a next recovery data server (second data server) 110b based on the configured pipeline (S310).

Then, the second data server 110b reads the recovery data held in the local storage in the server using the pipeline information delivered by the first data server 110a, and then the fourth data server 100d which is the next stage of the configured pipeline. In step S320, the recovery data and the pipeline information are transmitted together.

The fourth data server 100d reconstructs the recovered data transmitted by the second data server 110b with retained data owned by the fourth data server 100b and then performs a recovery operation (S330). In this case, the recovery operation is performed through a recovery operation method that can maintain an intermediate result in erasure code, such as the Cauchy Reed-Solomon method. However, in the third embodiment, since data transmission and recovery operations are simultaneously performed through pipeline processing, bit operations for pipelined recovery must be performed.

That is, in the bit operation method for pipelined recovery, the combining law is established for the XOR operation in each data server, so that the result of partial operation on the data stored in each node is accumulated so that the pipeline overhead processing is combined after stepwise calculation. do.

In more detail, as shown in FIG. 11, each data server generates a Cauchy bit matrix for recovery based on the received recovery data, and reads the generated cocy bit matrix and the input data. After performing the recovery operation, partition into appropriate chunks. The first data divided into the appropriate size is collected to perform a decoder to generate temporary parity data of the same size. Then, the data server performs a decoder using the partitioned data used in each step and the temporary parity data generated in the previous step to generate new temporary parity data again. This process is repeated with a data server located at the next stage of the pipeline. In this case, the transmission process may use buffering to increase efficiency.

Subsequently, by using the pipeline information input from the second data server 110b of the previous stage, the recovery operation data and the pipeline information, which are the result of performing the recovery operation, are transmitted together to the fifth data server 100e, which is the next stage of the pipeline. (S350).

At this time, the recovery data server participating in the recovery may simultaneously perform a task of receiving data, a recovery operation, and exporting the data. In particular, the task of receiving data and exporting data can make full use of the network's full-duplex attribute, which makes efficient use of network resources. Of course, recovery operations can efficiently use modern multicore CPU resources. Therefore, the recovery operation data is transmitted simultaneously with the recovery operation in one data server.

The fifth data server 100e reconstructs the recovery operation data transmitted by the fourth data server 100d with the retained data owned by the fourth data server 100d, and then, as illustrated in FIG. 11, Cauchy Reed-Solomon. Like the method, a recovery operation is performed through a recovery operation method capable of maintaining an intermediate result in an erasure code (S330). Meanwhile, the recovery operation may be performed by the Reed-Solomon method.

This recovery operation continues until the transmission to the data server through the configured pipeline is no longer the data server that is the next stage of the pipeline (S340). In this case, if there are no more data servers in the preceding stage, it is the case that the data server is a failure and the beginning of the pipeline.

In this way, the failed first data server 110a does not receive the recovery data sequentially from each of the recovery data servers holding the recovery data, but the result of the recovery operation accumulated through the pipeline is located at the last stage. The input is received only through the n th data server 110a.

Fourth Example

9 is a flowchart illustrating a method of receiving and restoring recovery data according to a third embodiment in a data recovery method of the present invention, and FIG. 5B is a method of receiving and restoring recovery data in a data recovery method of the present invention. A fourth embodiment for explanation.

Referring to the drawings, first, the metadata server 130 is a recovery data server (second data server, storing recovery data for recovering a failure of the failure data server (first data server) 110a). When the positions of the fourth data server, the fifth data server, ..., the mth data server, ..., the nth data server 110d, 110e, 110f, and 110h are detected (S200), the detected recovery data server ( The location information of the detected recovery data server is transmitted to the recovery data server (in this case, designated as the second data server 110b) of any one of 110b, 110d, 110e, 110f, and 110h.

Then, the second data server 110b configures one pipeline composed of each recovery data server based on the location information provided through the metadata server 130 (S310). At this time, the last stage of the pipeline is composed of the first data server 100a having a failure.

The second data server 110b reads the recovery data held in its local storage and transmits the recovery data and the pipeline information together to the fourth data server 100d, which is the next stage of the configured pipeline (S320). .

Then, the fourth data server 110d reconstructs the recovered data delivered by the second data server 110b with retained data owned by the fourth data server 110b and then, as shown in FIG. 11, as in the Cauchy Reed-Solomon method. The recovery operation is performed through a recovery operation method capable of maintaining an intermediate result in the erasure code (S330). Meanwhile, the recovery operation may be performed by the Reed-Solomon method.

Subsequently, the fourth data server 110d uses the pipeline information input from the second data server 110b of the previous stage to restore the data to the fifth data server 100e, which is the next stage of the pipeline. The data and the pipeline information are transmitted together (S350).

At this time, the recovery data server participating in the recovery may simultaneously perform a task of receiving data, a recovery operation, and exporting the data. In particular, the task of receiving data and exporting data can make full use of the network's full-duplex attribute, which makes efficient use of network resources. Of course, recovery operations can efficiently use modern multicore CPU resources. Therefore, the recovery operation data is transmitted simultaneously with the recovery operation in one data server.

The fifth data server 100e reconstructs the recovery operation data transmitted by the fourth data server 100d with the retained data owned by the fourth data server 100d, and then, as illustrated in FIG. 11, Cauchy Reed-Solomon. Like the method, a recovery operation is performed through a recovery operation method capable of maintaining an intermediate result in an erasure code (S330). Meanwhile, the recovery operation may be performed by the Reed-Solomon method.

This recovery operation continues until the transmission to the data server through the configured pipeline is no longer the data server that is the next stage of the pipeline (S340). In this case, if there are no more data servers in the preceding stage, it is the case that the data server is a failure and the beginning of the pipeline.

In this way, the failed first data server 110a does not receive the recovery data sequentially from each of the recovery data servers holding the recovery data, but the result of the recovery operation accumulated through the pipeline is located at the last stage. Is input only through the n th data server 110a.

FIG. 5C is a timing diagram showing a recovery operation time during a data recovery operation in the third and fourth embodiments. FIG.

As described above, as the first data server 110a simultaneously transmits and recovers the recovery operation data from each of the recovery data servers based on the pipeline, the recovery operation time is represented by Equation 2 as shown in FIG. Can be represented.

[Equation 2]

Recovery operation time = (one data transfer time) + (pipeline overhead) × (number of participating data nodes)

As shown in Equation 2, as the data receiving operation, the recovery operation operation, and the data export operation are performed at the same time, one data transmission time, the participating recovery data server, and the recovery operation time are pipelined to reduce the recovery operation time. Can be reduced. In this case, the pipeline overhead means a time that does not overlap. The recovery operation time is mostly performed in parallel within the data transmission time and overlapped.

Although the technical spirit of the present invention described above has been described in detail in a preferred embodiment, it should be noted that the above-described embodiment is for the purpose of description and not of limitation. In addition, those skilled in the art will understand that various embodiments are possible within the scope of the technical idea of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (9)

(A) if a failed first data server is detected through the metadata server, detecting location information of data servers storing recovery data to be reconstructed for recovery of the detected first data server failure;
(B) requesting recovery data from the data servers where the location information is detected, receiving recovery data, and performing a recovery operation through a recovery operation using the received recovery data;
(C) completing the recovery operation of the failed data server through the recovery operation, and notifying the metadata server that the recovery of the data server is completed; . The method of claim 1, wherein step (B)
Providing location information of the detected recovery data servers to any data server that is responsible for failover;
Receiving a recovery data request command stored in local storage in the recovery data server by transmitting a recovery data request command to the recovery data servers holding the recovery data based on the provided location information;
Efficient data recovery method in a distributed file system comprising the step of receiving the recovery data until the failure recovery of the first data server, reconstructing the input recovery data, and performing a recovery operation . The method of claim 1, wherein step (B)
Transmitting a recovery data request command to the recovery data servers from which the location information is detected;
Receiving recovery data held in local storage in each recovery data server to which the recovery data request command is transmitted;
And receiving the recovery data until the failure of the first data server is possible, reconstructing the input recovery data, and performing a recovery operation. The method according to claim 2 or 3,
The input of the recovery data is efficient data recovery method in a distributed file system, characterized in that the transmission of the recovery data held in the next data server is sequentially performed when the transmission of the recovery data held in one data server is completed . The method according to claim 2 or 3,
The input of the recovery data is effective data recovery method in a distributed file system, characterized in that the detected recovery data server in parallel based on the network transmission capacity. The method according to claim 2 or 3,
The recovery operation is any one of Reed-Solomon, Cauchy Reed-Solomon method, efficient data recovery method in a distributed file system. The method of claim 1, wherein step (B)
Providing location information of the detected recovery data servers to the failed first data server;
A pipeline including a recovery data request command to a next recovery data server based on the configured pipeline by configuring one pipeline composed of each recovery data server based on the location information of the provided recovery data servers. Transmitting the information,
Reading recovery data held in its local storage based on the transmitted pipeline information, and transmitting the recovery data and pipeline information together to a recovery data server, which is the next stage of the pipeline;
A reconstruction operation that is a result of performing a reconstruction operation to a data server, which is the next stage of the pipeline, by performing a reconstruction operation after reconstructing the reconstruction data transmitted from the reconstruction data server of the previous stage with retained data owned by the residing data. Sending data and pipeline information together;
After reconstructing the transmitted recovery operation data with the retained data owned by the user, the recovery operation data and the pipeline information, which are the result of the recovery operation, are performed to the data server, which is the next stage of the pipeline, by performing a repair operation while performing a recovery operation. Sending together,
And continuing until the next stage, the data server, based on the configured pipeline information, does not exist. The method of claim 1, wherein step (B)
Providing location information of the detected recovery data server to any one of the recovery data servers from which the location information has been detected;
Constructing a pipeline consisting of respective recovery data servers based on the location information of the provided recovery data servers;
Reading recovery data held in its local storage based on the configured pipeline information and transmitting the recovery data and pipeline information together to a recovery data server, which is the next stage of the pipeline;
After reconstructing the transmitted recovery data from the retained data owned by the owned data, the recovery operation data and the pipeline information, which are the result of performing the recovery operation together with the data server which is the next stage of the pipeline through the pipeline processing while performing the recovery operation, together Transmitting,
After reconstructing the transmitted recovery operation data with the retained data owned by the user, the recovery operation data and the pipeline information, which are the result of the recovery operation, are performed to the data server, which is the next stage of the pipeline, by performing a repair operation while performing a recovery operation. Sending together,
And continuing until the next stage, the data server, based on the configured pipeline information, does not exist. 9. The method according to claim 7 or 8,
The recovery operation is any one of Reed-Solomon, Cauchy Reed-Solomon method, efficient data recovery method in a distributed file system.

Images