KR20130133989A - System and method for parallel file transfer between file storage clusters - Google Patents

Abstract

The present invention discloses a system and method for parallel file transfer between file storage clusters. The parallel file transmission system between the file storage clusters according to the embodiment of the present invention can receive the information of electrical transmission object files stored in the first file storage cluster from the master server of the first file storage cluster and also the system for transmitting the preserved file in the first file storage cluster with the second file storage cluster. The invention also includes the file transfer manager assigning the electrical transmission object files to multiple file transmission parts based on the information of the electrical transmission object files, and multiple file transmission parts receiving the electrical transmission object files allocated from the file transfer manager from the first file storage cluster stores the received electrical transmission object files in the second file storage cluster. [Reference numerals] (110) File Transfer Manager;(112) 1DFS client;(114,D1,D2) File Transfer unit;(116,E1,E2) 1DFS client;(118,F1,F2) 2DFS client;(120) Network;(AA) Relay server 1;(BB) Relay server 2;(CC) Relay server N;(GG) 1DFS Network;(II,RR) Slave 2;(JJ,QQ) Slave 1;(KK,PP) Slave N;(L1,L2,L3,U1,U2,U3) Data node;(M1,M2,M3,V1,V2,V3) Disk;(NN,TT) Metadata node;(OO) 2DFS Network

Description

System and method for parallel file transfer between file storage clusters {SYSTEM AND METHOD FOR PARALLEL FILE TRANSFER BETWEEN FILE STORAGE CLUSTERS}

The present invention relates to techniques for transferring files efficiently between file storage clusters.

A file storage cluster is a system in which a plurality of servers are bundled into clusters to provide a file storage service. The file storage cluster is characterized in that the server in the cluster is configured to process in parallel when there is a request to read or write a file from the client. Such file storage clusters include systems such as Distributed File System (DFS) and Object Storage (DFS), including Hadoop HDFS, GlusterFS, Luster, and Swift. File storage clusters are widely used in big data because they can easily store and manage tens of petabytes of large data.

In the enterprise market using file storage clusters as described above, the stored data needs to be transferred to another file storage cluster for reasons such as backup of the stored data, replacement of the file storage cluster, and analysis of the stored data. However, in the case of file storage clusters, the capacity of the data is very large. Therefore, when using a general data migration method, the data movement takes too much time. As a result, there is a need for technology to efficiently support data transfer between file storage clusters.

It is an object of the present invention to provide a means for effectively transferring data between file storage clusters using the characteristics of the file storage cluster.

The parallel file transfer system between file storage clusters according to an exemplary embodiment of the present invention is a system for transferring a file stored in a first file storage cluster to a second file storage cluster, the master file of the first file storage cluster. A file transfer manager that receives information of transfer target files stored in a first file storage cluster and allocates the transfer target files to a plurality of file transfer units based on the transfer target file information; And the plurality of file transfer units that receive a transfer target file allocated from the file transfer manager from the first file storage cluster and store the received transfer target file in the second file storage cluster.

In addition, the parallel file transfer method between file storage clusters according to an embodiment of the present invention is a method for transferring a file stored in a first file storage cluster to a second file storage cluster, and in the file transfer manager, the first file. Receiving information of transfer target files stored in the first file storage cluster from a master server of a storage cluster; Assigning, by the file transfer manager, the transfer target files to a plurality of file transfer units based on the received information of the transfer target file; And a plurality of file transfer units, receiving a transfer target file allocated from the file transfer manager from the first file storage cluster and storing the received transfer target file in the second file storage cluster.

According to embodiments of the present invention, the bottleneck may be prevented by increasing the number of nodes for file transfer between file storage clusters. In addition, the file transfer time can be shortened by efficiently distributing the files to the transfer nodes in consideration of the file storage characteristics of the distributed file system and processing the file transfer operations in parallel.

1 is a diagram illustrating a parallel file transfer system 100 between file storage clusters according to a first exemplary embodiment of the present invention.
2 is a flowchart illustrating a file transfer method 200 in a file storage cluster-to-cluster parallel file transfer system 100 according to a first embodiment of the present invention.
3 is a diagram illustrating a parallel file transfer system 300 between file storage clusters according to a second exemplary embodiment of the present invention.
4 is a diagram illustrating a parallel file transfer system 400 between file storage clusters according to a third exemplary embodiment of the present invention.
5 is a flowchart illustrating a file transfer method 500 in the parallel file transfer system 300 and 400 between file storage clusters according to the second and third exemplary embodiments of the present invention.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, this is merely an example and the present invention is not limited thereto.

In the following description, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification.

The technical idea of the present invention is determined by the claims, and the following embodiments are merely a means for effectively explaining the technical idea of the present invention to a person having ordinary skill in the art to which the present invention belongs.

1 is a diagram illustrating a parallel file transfer system 100 between file storage clusters according to a first exemplary embodiment of the present invention. As shown, the file storage inter-cluster parallel file transfer system 100 according to the first embodiment of the present invention is configured to transfer files stored in the first file storage cluster 102 to the second file storage cluster 104. As a system, a file relay device 106 and a plurality of relay servers 108 are included, and each of the above-described components are connected to each other through a network 120 to transmit and receive data. In an embodiment of the invention, the network 120 is configured to ensure sufficient network bandwidth between the first file storage cluster 102 and the second file storage cluster 104 for smooth data transfer. For example, in the illustrated embodiment, if there are 10 slave servers in the first file storage cluster 102 and each has 1 Gbps of network performance, then at least 10 Gbps (in bold) between the first and second DFS networks, Guaranteed performance can be achieved.

The first file storage cluster 102 is a file storage cluster in which data to be transmitted to the second file storage cluster 104 is stored. The first file storage cluster 102 is a file system in which a plurality of servers are clustered through a first DFS network and distributed files are stored in multiple nodes. For example, HDFS of Hadoop.

As shown, the first file storage cluster 102 includes one master server and one or more slave servers. The master server includes a metadata node, which manages information of files stored in each slave server. That is, which slave server to store the file is determined by the metadata node of the master server, and the client to store the file stores the file on the disk through the data node of the slave server determined by the metadata node. . In contrast, when a client reads a file, the client first obtains information on the file from the metadata node, and then reads the file stored in the disk through the data node of the slave server according to the information. Meanwhile, in the illustrated embodiment, although the metadata server is included in the master server, in the case of a file storage cluster without a master server such as GlusterFS, the metadata node also does not exist in configuration. However, even in this case, other components in the system are configured to perform the same function as the metadata node. In this case, it is logically regarded that the metadata node exists.

 The second file storage cluster 104 is a file storage cluster for storing data read into the first file storage cluster 102. The second file storage cluster 104 also includes one master server and one or more slave servers in the same manner as the first file storage cluster 102. The second file storage cluster 104 may use the same type of file storage cluster as the first file storage cluster 102 or may be implemented using another type of file storage cluster. In other words, the invention is not limited to any particular type of file storage cluster.

The file relay device 106 is a device for totally controlling file transfer between file storage clusters performed by a plurality of relay servers 108 to be described later, and includes a file transfer manager 110 and a first DFS client 112. .

The file transfer manager 110 receives the information of the transfer target files stored in the first file storage cluster 102 from the metadata node of the master server of the first file storage cluster 102, and receives the information of the received transfer target file. Based on this, the transfer target files are assigned to each file transfer unit 114 in the plurality of relay servers 108. In addition, the file transfer manager 110 continuously schedules and monitors the file transfer operation of the relay server 108 even after allocating the file.

The information of the transfer target file includes a URL and file size information in the first file storage cluster 102 of the transfer target file. Using this, the file transfer manager 110 transmits the plurality of file transfer units 114 to the transfer target files such that the sum of the sizes of transfer target files allocated to each file transfer unit 114 is equalized for each file transfer unit 114. Is assigned to).

For example, file allocation in the file transfer manager 110 will be described below. First, suppose that the sizes of the respective files stored in the first file storage cluster 102 are sufficiently similar in size, although they are not all the same or completely the same. In this case, the total capacity is obtained by using file size information of individual files, and the file is allocated to each file transfer unit 114 as much as the capacity corresponding to the total capacity / number of file transfer units. For example, if there are 100 files each having a size of 1 GB and the file transfer unit 114 is composed of 10 files, the total file size is 100 GB, so that 10 files per file transfer unit 114 are divided and allocated. The allocation method may be configured to divide the list of 100 file URLs by 10 and transmit the list to the file transfer unit 114.

On the contrary, when the size of each file stored in the first file storage cluster 102 is severe, the file transfer manager 110 may arrange the transfer target files according to the size, and arrange the aligned transfer target files in the file transfer unit 114. ), But the file allocation order of a specific round is assigned to the files to be transferred so that the file allocation order of the previous round is reversed. This is explained more easily as follows.

For example, assume that there are 100 files in the first file storage cluster 102, and that each capacity increases from 1MB to 100MB in 1MB intervals. In order to assign these to ten file transfer units 114 each having a name from a to j, the files are first sorted in descending order (or ascending order) by file size. Then, the sorted files are arranged in each file transfer unit 114 in order, first in the order of a to j, then in the order of j to a, then again in the order of a to j, that is, Allocate files to be transferred so that a file rounding order of a specific round is reversed from the previous file ordering. This is shown in the table below.

File transfer name a b c d e f g h i j One 2 3 4 5 6 7 8 9 10 20 19 18 17 16 15 14 13 12 11 21 22 23 24 25 26 27 28 29 30 40 39 38 37 36 35 34 33 32 31 41 42 43 44 45 46 47 48 49 50 60 59 58 57 56 55 54 53 52 51 61 61 61 61 61 61 61 61 61 61 80 79 78 77 76 75 74 73 72 71 81 82 83 84 85 86 87 88 89 90 100 99 98 97 96 95 94 93 92 91 Sum of size by file transfer part 505 504 503 502 501 500 499 498 497 496

When the files are allocated in this way, as shown in Table 1, the sum of the sizes of the files allocated to each file transfer unit 114 is almost the same, so that the operation of the specific file transfer unit 114 ends too quickly. This allows you to perform file transfers efficiently and without ending too late.

The first DFS client 112 is an agent used by the file transfer manager 110 to connect to the first file storage cluster 102 to receive information of the transfer target files. That is, the file transfer manager 110 accesses the master server of the first file storage cluster 102 through the first DFS client 112 and receives the information of the transfer target files.

Next, the relay server 108 includes a file transfer unit 114, a first DFS client 116, and a second DFS client 118. In the embodiment of the present invention, the number of relay servers 108 is not limited, but it is preferable to have the same number of relay servers 108 as the number of slave servers in a file storage cluster to which files are to be transferred.

The file transfer unit 114 receives the transfer target file allocated from the file transfer manager 110 from the first file storage cluster 102 using the first DFS client 116, and receives the received transfer target file in a second manner. The file is stored in the second file storage cluster 104 using the DFS client 118, and the file transfer manager 110 reports the file transfer when the file transfer is completed.

The first DFS client 116 is an agent used by the relay server 108 to connect to the first file storage cluster 102 to read the transfer destination file, and the second DFS client 118 is the relay server 108. Is the agent used to connect to the second file storage cluster 104 to store the transfer target file.

2 is a flowchart illustrating a file transfer method 200 in a file storage cluster-to-cluster parallel file transfer system 100 according to a first embodiment of the present invention.

First, the file transfer manager 110 requests information of transfer target files stored in the first file storage cluster 102 from the metadata node of the master server of the first file storage cluster 102 through the first DFS client 112. To receive it (202). As described above, the information of the transfer target file includes a URL and a file size in the first file storage cluster 102 of the transfer target file.

If the first file storage cluster 102 is HDFS-based, the file transfer manager 110 may obtain file information through an API provided by namenode. In addition, when the client is mounted by supporting a Portable Operating System Interface (POSIX) such as GlusterFS, a command for obtaining file information such as ls may be used to receive information about files to be transferred.

Next, the file transfer manager 110 allocates the transfer target files to the plurality of file transfer units 114 based on the received information of the transfer target file (204). At this time, the file transfer manager 110 transmits the transfer target files to the plurality of file transfer units 114 such that the sum of sizes of transfer target files allocated to each file transfer unit 114 is equalized for each file transfer unit 114. Can be assigned. Since the file allocation method has been described in detail with reference to FIG. 1, a detailed description thereof will be omitted.

Next, each file transfer unit 114 receives a transfer target file allocated from the file transfer manager 110 from the first file storage cluster 102 through the first DFS client 116 (206). At this time, the first DFS client 116 obtains the information of the slave server in which the actual file data is stored through the metadata node, and then reads the transmission target file through the data node of the slave server in which the actual data is stored. Step 206 and steps 208 and 210 to be described later are performed in parallel through a plurality of file transfer units 114.

Next, each file transmitter 114 requests the metadata node of the second file storage cluster 104 to allocate a URL for storing the received file to be transmitted to the second file storage cluster 104 (208). ). If the first file storage cluster 102 and the second file storage cluster 104 are file storage clusters of the same type so that the URL of the first file storage cluster 102 can be used as it is, the step may be omitted. have. However, since the URL notation may be different between heterogeneous file storage clusters, in this case, a new URL allocation should be requested to the second file storage cluster 104.

Next, each file transmission unit 114 determines whether or not the received URL is available, and if available, the second file through the second DFS client 118 using the URL to be received. Stored in storage cluster 104 (210). In this case, when the second file storage cluster 104 stores a striping method, one file is stored in groups of a plurality of slave servers in chunks, and when the second file storage cluster 104 is stored in a distributed manner, the second file storage cluster 104 is stored in file units on a designated slave server. .

Next, each file transmitter 114 determines whether a file to be transmitted remains (212), and if the remaining file exists, repeats steps 206 to 210.

If no file remains to be transmitted, each file transfer unit 114 reports the file transfer result to the file transfer manager 110 (214). Then, the file transfer manager 110 determines whether or not the transfer failed file exists from the file transfer result (216), and if so, returns to step 204 and attempts to retransmit the file. However, in order to prevent an infinite loop, when a file transfer fails more than a predetermined number of times in this step, it is preferable to stop work assignment for the file.

Finally, if the transfer failed file does not exist or if the transfer of the file is canceled because the file transfer fails for a predetermined number of times, the file transfer is terminated (218).

3 is a diagram illustrating a parallel file transfer system 300 between file storage clusters according to a second exemplary embodiment of the present invention. As can be seen from the figure, in the present embodiment, there is no separate relay server 108, and each component of the relay server 108 is in the same hardware as each slave server of the first file storage cluster 102. For example, the file transfer unit 114, the first DFS client 116 and the second DFS client 118 is different from the first embodiment. That is, in the present embodiment, each slave server of the first file storage cluster 102 serves as a relay server. In this case, the actual transfer target file is directly transferred from the first DFS network to the second DFS network, and since only a small amount of data such as information of each transfer target file is transmitted to the file transfer manager 110, Sufficient bandwidth (indicated by bold lines in the drawing) is required only between the DFS network and the second DFS network.

In addition, since the relay server and the slave server exist in one hardware, the file transfer unit 114 directly uses the internal server communication without using a network when receiving a file from a data node existing in the same hardware. This allows you to read files much faster than when using a network.

4 is a diagram illustrating a parallel file transfer system 400 between file storage clusters according to a third exemplary embodiment of the present invention. As can be seen in the figure, in this embodiment, each node of the first file storage cluster 102, each node of the second file storage cluster 104, and the plurality of relay servers 108 are all one piece of hardware. It is provided in. That is, in the present exemplary embodiment, when the hardware resources of the first file storage cluster are sufficient, the second file storage cluster 104 and the relay server are configured in the same hardware, and the files of the first file storage cluster 102 are deleted through the internal communication network. Two file storage cluster 104.

According to this embodiment, there are advantages as follows. First, the existing file storage cluster may be used as it is without having to provide new hardware resources to configure the second file storage cluster 104.

In addition, if the data node of the first file storage cluster 102 to read the file and the data node of the second file storage cluster 104 to store the file exist in the same hardware, the data transmission may be performed through the internal communication network without using the network at all. As a result, data transmission is possible at a much higher speed than the previous embodiment.

5 is a flowchart illustrating a file transfer method 500 in the parallel file transfer system 300 and 400 between file storage clusters according to the second and third exemplary embodiments of the present invention.

First, the file transfer manager 110 requests information of transfer target files stored in the first file storage cluster 102 from the metadata node of the master server of the first file storage cluster 102 through the first DFS client 112. To receive it (502).

At this time, the information of the transfer target files may be divided into two cases according to the file storage type of the first file storage cluster (02).

First, the file is stored in a distributed manner. The distributed method is to store files in one slave server without splitting them. In this case, the information of the transfer target file that the file transfer manager 110 receives from the first file storage cluster 102 corresponds to a URL other than a URL and a file size of the transfer target file in the first file storage cluster 102. The file further includes identification information of the slave server in which the file is stored.

Secondly, the file is stored by striping. The striping method refers to a method of dividing a file into a plurality of chunks and storing them in a plurality of slave servers. In this case, the information of the transfer target file may include identification of a slave server in which a URL, a file size, and each chunk constituting each transfer target file are stored in the first file storage cluster 102 of the transfer target file. Contains information.

Next, the file transfer manager 110 allocates the transfer target files to the plurality of file transfer units 114 based on the received information of the transfer target file (504).

First, when the first file storage cluster 102 stores the files in a distributed manner, the file transfer manager 110 divides the transfer target file into a plurality of groups by the number of file transfer units 114 and each of the divided files. Groups of are allocated to each file transfer unit 114, but may be divided so that the sum of the size of the transfer target files included in each group is equal for each group. That is, the total capacity may be obtained according to the file transfer capacity described in the above information, and each file transfer unit 114 may be divided into groups so that the files are distributed as much as the capacity corresponding to " total capacity / number of file transfer units. &Quot;

Also, in this case, the file transfer manager 110 may be configured to assign a group including the largest number of transfer target files stored in the same slave server as the specific file transfer unit 114 to the specific file transfer unit 114. That is, the group having the most files stored in the slave server at the same location as the file transfer unit 114 is assigned to the file transfer unit 114 first.

On the other hand, when the first file storage cluster 102 stores the files in a striping manner, the file transfer manager 110 may divide the transfer target files into a plurality of groups as many as the number of the file transfer units 114 in the same manner as in the distributed scheme. Each of the divided groups may be divided and allocated to each file transfer unit 114. The divided files may be divided so that the sum of the size of the transfer target files included in each group is equalized for each group. In this case, however, the file transfer manager 110 assigns the group including the most pieces of the transfer target file stored in the same slave server as the specific file transfer unit 114 to the specific file transfer unit 114. That is, the group having the most fragments stored in the slave server at the same location as the file transfer unit 114 is assigned to the file transfer unit 114 first.

Next, each file transfer unit 114 receives (506) the transfer target file allocated from the file transfer manager 110 through the first DFS client 116 from the first file storage cluster 102. In this case, the first DFS client 116 obtains the information of the slave server in which the actual file data is stored through the metadata node, and then reads the transmission target file through the data node of the slave server in which the actual data is stored. Step 206 and steps 208 and 210 to be described later are performed in parallel through a plurality of file transfer units 114.

Next, each file transfer unit 114 requests the metadata node of the second file storage cluster 104 to allocate a URL for storing the received file to be transmitted to the second file storage cluster 104 (508). ). If the first file storage cluster 102 and the second file storage cluster 104 are file storage clusters of the same type, and thus the URL of the first file storage cluster 102 can be used as it is, this step may be omitted. have. However, since the URL notation may be different between heterogeneous file storage clusters, in this case, a new URL allocation should be requested to the second file storage cluster 104.

Next, each file transmission unit 114 determines whether or not the received URL is available, and if available, the second file through the second DFS client 118 using the URL to be received. Stored in storage cluster 104 (510). In this case, when the second file storage cluster 104 stores a striping method, one file is stored in groups of a plurality of slave servers in chunks, and when the second file storage cluster 104 is stored in a distributed manner, the second file storage cluster 104 is stored in file units on a designated slave server. .

Next, each file transmitter 114 determines whether a file to be transmitted remains (512), and if the remaining file exists, repeats steps 206 to 210.

If no file remains to be transmitted, each file transfer unit 114 reports the file file transfer result to the file transfer manager 110 (514). Then, the file transfer manager 110 determines whether or not the transfer failed file exists from the file transfer result (516), and if so, returns to step 204 and attempts to retransmit the file. However, in order to prevent an infinite loop, when a file transfer fails more than a predetermined number of times in this step, it is preferable to stop work assignment for the file.

Finally, if the transfer failed file does not exist or if the transfer of the file is canceled because the file transfer has failed for a predetermined number of times, the file transfer is terminated (518).

On the other hand, an embodiment of the present invention may include a computer-readable recording medium including a program for performing the methods described herein on a computer. The computer-readable recording medium may include a program command, a local data file, a local data structure, or the like, alone or in combination. The media may be those specially designed and constructed for the present invention or may be known and available to those of ordinary skill in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and magnetic media such as ROMs, And hardware devices specifically configured to store and execute program instructions. Examples of program instructions may include machine language code such as those generated by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the present invention. I will understand.

Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by equivalents to the appended claims, as well as the appended claims.

100: parallel file transfer system between file storage clusters
102: first file storage cluster
104: second file storage cluster
106: file relay device
108: relay server
110: file transfer manager
112: First DFS Client
114: file transfer unit
116: First DFS Client
118: Second DFS Client
120: Network

Claims (24)

A system for transferring a file stored in a first file storage cluster to a second file storage cluster,
A file that receives information of transfer target files stored in the first file storage cluster from a master server of the first file storage cluster, and allocates the transfer target files to a plurality of file transfer units based on the transfer target file information. Transmission manager; And
A file inter-cluster parallel file including the plurality of file transfer units for receiving a transfer target file allocated from the file transfer manager from the first file storage cluster and storing the received transfer target file in the second file storage cluster. Transmission system.
The method according to claim 1,
And wherein the information of the transfer target file includes a URL and a file size in the first file storage cluster of the transfer target file.
The method according to claim 2,
And the file transfer manager assigns the transfer target files to the plurality of file transfer units such that the sum of sizes of transfer target files allocated to each file transfer unit is equalized for each file transfer unit.
The method according to claim 3,
The file transfer manager arranges the transfer target files according to size and sequentially allocates the sorted transfer target files to the file transfer unit, wherein a specific round file allocation order is the same as the previous conference file allocation order. Allocating the transfer destination files to be reversed.
The method according to claim 2,
The plurality of file transfer units request a URL allocation in the second file storage cluster of the transfer target file to a master server of the second file storage cluster, and from the first file storage cluster according to the allocated URL. And a file transfer system for storing the received transfer target file.
The method according to claim 1,
The plurality of file transfer units are disposed in a plurality of slave servers included in the first file storage cluster.
The method of claim 6,
The information of the transfer target file includes a URL, a file size, and identification information of a slave server in which the file is stored, in the first file storage cluster of the transfer target file.
The method of claim 7,
The file transfer manager divides the transfer target file into a plurality of groups by the number of the file transfer units and allocates each of the divided groups to each file transfer unit, but the sum of the size of the transfer target files included in each group is increased. A file transfer system that divides evenly into groups.
The method according to claim 8,
And the file transfer manager assigns to the specific file transfer unit a group containing the most transfer target files stored in the same slave server as the specific file transfer unit.
The method of claim 6,
The information of the transfer target file includes identification information of a slave server in which a URL, a file size, and each chunk constituting each transfer target file are stored in the first file storage cluster of the transfer target file. File transfer system.
The method of claim 10,
The file transfer manager divides the transfer target file into a plurality of groups by the number of the file transfer units and allocates each of the divided groups to each file transfer unit, but the sum of the size of the transfer target files included in each group is increased. A file transfer system that divides evenly into groups.
The method of claim 11,
And the file transfer manager assigns to the specific file transfer unit a group including the most pieces of a transfer target file stored in the same slave server as the specific file transfer unit.
A method for transferring a file stored in a first file storage cluster to a second file storage cluster, the method comprising:
Receiving, at a file transfer manager, information of transfer target files stored in the first file storage cluster from a master server of the first file storage cluster;
Assigning, by the file transfer manager, the transfer target files to a plurality of file transfer units based on the received information of the transfer target file; And
In the plurality of file transfer unit, File storage cluster comprising the step of receiving a transfer target file allocated from the file transfer manager from the first file storage cluster, and storing the received transfer target file in the second file storage cluster Parallel file transfer method.
The method according to claim 13,
And the information of the transfer target file comprises a URL and a file size in the first file storage cluster of the transfer target file.
The method according to claim 14,
The step of allocating the transfer target files to the plurality of file transfer units may include transferring the transfer target files to the plurality of file transfer units such that the sum of sizes of transfer target files allocated to each file transfer unit is equalized for each file transfer unit. File transfer method.
16. The method of claim 15,
The step of assigning the transfer target files to a plurality of file transfer unit,
Sorting the transfer target files according to a size; And
Allocating the transferred target files sequentially to the file transfer unit, allocating the transfer target files such that a specific round file allocation order is opposite to a previous conference file allocation order. Way.
The method according to claim 14,
Storing the transfer target file in the second file storage cluster,
Requesting a URL allocation in the second file storage cluster of the transfer target file to a master server of the second file storage cluster; And
And storing the transfer target file received from the first file storage cluster in the second file storage cluster according to the assigned URL.
The method according to claim 14,
The information of the transfer target file includes a URL, a file size, and identification information of a slave server in which the file is stored, in the first file storage cluster of the transfer target file.
19. The method of claim 18,
The step of allocating the transfer target files to a plurality of file transfer units may include dividing the transfer target files into a plurality of groups by the number of the file transfer units and assign each divided group to each file transfer unit, wherein each group And dividing the sum of the size of the transfer target files included in the evenly into groups.
The method of claim 19,
The allocating of the transfer target files to the plurality of file transfer units may include: allocating a group including the most transfer target files stored in the same slave server as the transfer file to the specific file transfer unit.
The method according to claim 14,
The information of the transfer target file includes identification information of a slave server in which a URL, a file size, and each chunk constituting each transfer target file are stored in the first file storage cluster of the transfer target file. File transfer method.
The method of claim 10,
The step of allocating the transfer target files to a plurality of file transfer units may include dividing the transfer target files into a plurality of groups by the number of the file transfer units and assign each divided group to each file transfer unit, wherein each group And dividing the sum of the size of the transfer target files included in the evenly into groups.
23. The method of claim 22,
The allocating of the transfer target files to the plurality of file transfer units may include: allocating a group including the most fragments of the transfer target files stored in the same slave server as the specific file transfer unit to the specific file transfer unit. .
A computer-readable recording medium having recorded thereon a program for performing the method according to any one of claims 13 to 23 on a computer.

Images