KR20180058096A - Distributed file sharing system and method of duplicating file in distributed file sharing system - Google Patents

Abstract

The present invention relates to a distributed file sharing system including a plurality of servers and a plurality of customer terminals communicating with the plurality of servers. One server among the plurality of servers comprises: a server interface receiving a request of the customer terminals including a file search, a file upload request and a file download request through communication between any one server and the plurality of customer terminals, and including an event handler which performs duplication and distribution of a file for processing according to the request of the customer terminals; and a database storing and managing metadata for file search and log data which is a result of processing the request of the customer terminals. A file sharing service can be smoothly provided without addition of a separate server for file search by properly distributing and storing the file in the plurality of servers in a distributed environment.

Description

METHOD OF DISTRIBUTED FILE SHARING SYSTEM AND DISTRIBUTED FILE SHARING SYSTEM BACKGROUND OF THE INVENTION 1. Field of the Invention < RTI ID = 0.0 >

The present invention relates to a distributed file sharing system and a method for replicating files in a distributed file sharing system. More particularly, the present invention relates to a method of distributing and storing metadata in a distributed network such as P2P, And a distributed file sharing system for processing requests of customer terminals including uploading.

P2P file sharing systems that operate in a distributed environment require file data as a method for sharing stored files and additionally use a server for searching the location of files. Examples of P2P file sharing systems include Napster, Guntella, and KaZaA. Because Napster uses one centralized server to store and manage file data, the entire system is unavailable when one centralized server goes down. Gnutella does not use an additional server to share file data but sends a broadcast message to all the servers that make up the system to obtain file data. Gnutella is inevitably burdened by the network as the number of servers making up the system and the number of requests for file downloads increase. Kazaa manages file data by using several servers called Super Nodes, so the system performance decreases as the number of super nodes and the total number of servers increase.

As described above, systems for providing a file sharing service based on a distributed environment consume a lot of latency to retrieve file data as the number of files to be shared increases. In addition, systems for providing a file sharing service based on a distributed environment have a problem in that unnecessary communication in the network is excessively generated due to an increase in the number of customer terminals connecting to the system, Shuntdown).

For example, a P2P file sharing system such as Torrent manages file data instead of using the web servers that use the name "Tracker" for file retrieval, so it can provide services without worrying about file data. have. However, when the size of the file data held by the server becomes large, the search time for obtaining the information of the file becomes long. Also, the torrent is very slow because it downloads the entire file by fetching fragments of the file from specific computers scattered around the globe.

Therefore, a file sharing system for efficiently storing and managing file data in a distributed environment such as a P2P network is required.

Techniques that constitute the background of the present invention are disclosed in the following two patent documents.

(1) United States Patent Publication No. 5635683 A (MCDERMOTT, R. M.) 1997.06.03.

(2) Japanese Patent Registration No. JP 10-105775 A 1998.04.24, page 5, line 3-15, drawing 1

According to one side, the metadata of a file is stored in distributed servers based on a distributed based NoSQL (Not-Structured Query Language) storage system (or distributed based NoSQL database) to provide a file sharing service, Can solve the file search problem that existed in.

According to one side, it replicates files based on the popularity of files to be shared among distributed servers and the status information of servers, and distributes and stores files to distributed servers, thereby eliminating the network bottleneck that was concentrated in one server. And provide fault tolerance of the system.

According to one side, even if a server is added or deleted in a distributed environment, it can provide high availability and scalability in which the file sharing service is not interrupted.

According to one aspect of the present invention, in a distributed file sharing system including a plurality of servers and a plurality of client terminals communicating with the plurality of servers, any one of the plurality of servers includes the server Receiving a request of the customer terminals including a file search, a file upload request, and a file download request through communication between the customer terminals of the client terminal, performing an event for performing file replication and distribution for processing according to a request of the customer terminals A server interface including a handler; And a database for storing and managing the metadata for the file search and the log data which is a processing result of the request of the customer terminals.

The server interface may select at least one server to distribute the replication factor of the file to be replicated and the file to be replicated, based on the popularity of the file and the request of the client terminals for the file.

Wherein the event handler processes an event including a request for the client terminals, an event including the file copying and distribution, and the server interface periodically transmits the disk capacity of the one server and the status information of the one server A status updater to collect and update; And a query processor for processing a query including reading, inserting, and storing data for the database based on the information collected by the status updater.

When the uploading of a file is requested by the client terminal, the event handler searches another server adjacent to the one server based on whether any one of the servers has a sufficient space for storing the file to be uploaded , And may transmit the information about the other server to the client terminal requesting the upload.

Wherein the event handler collects the log data, calculates a total number of requests of the client terminals for the file processed by the one server based on the collected log data, The server can measure the popularity of the processed file and calculate the copying coefficient of the processed file based on the popularity of the file.

Wherein the event handler calculates a ratio of the popularity of the file in any one of the servers and a network cost required for the one server to replicate the file, and calculates a ratio of the popularity of the file and the network cost It is possible to determine whether to copy the file to any one of the servers based on the information.

Wherein the database comprises: a file data storage for storing and managing data generated upon request of the customer terminals and data for searching the file; A log storage for storing log data which is a result of a request of the client terminals processed by any one of the servers; A routing information storage for storing a result of measuring a data movement time from one of the servers to the other servers; And a cluster status store for storing status information including a cluster status of any one of the servers and a disk usage capacity of any one of the servers.

According to one aspect, a method for replicating a file in a distributed file sharing system including a plurality of servers and a plurality of client terminals communicating with the plurality of servers, includes the steps of: ; A replication coefficient of the file to be duplicated and at least one server for distributing the file among the plurality of servers based on the popularity of the file to be duplicated and the network cost of the server for duplicating the file to be duplicated Determining; And replicating the file to the determined at least one server according to the replication factor.

Wherein determining at least one server to distribute the file comprises: calculating a percentage of popularity of the file at the at least one server of the plurality of servers; Calculating a network cost for the at least one server to replicate the file; And determining the at least one server as a server to distribute the file based on the ratio of the popularity of the file and the network cost.

The step of calculating the popularity of the file to be duplicated may include collecting log data that is a processing result of the request of the customer terminals; Calculating a total number of requests of the client terminals for the file processed by any one of the plurality of servers based on the collected log data; And calculating the popularity of the file processed by any one of the servers using the total number of requests.

According to an embodiment, a file sharing service can be smoothly provided without adding a separate server for file search by appropriately distributing and storing files in a plurality of servers in a distributed environment.

According to an embodiment, a server for maintaining an actual file may be distinguished from a separate server for searching for a file location, or a client terminal may provide a file sharing service by a single server without communicating with a separate server can do.

According to an embodiment, a customer terminal can acquire a desired file (or data) by only one query for retrieving a file.

According to an embodiment, even if a server constituting a distributed file sharing system is added and / or deleted by using the distributed-based NoSQL storage system, it is possible to cope with flexibility without affecting the entire system.

According to an embodiment of the present invention, by detecting state information at a server constituting a distributed file sharing system, it is possible to quickly detect a change in a server and provide a service suited to the state of the corresponding sub, and to provide scalability, Flexibility, and consistency of metadata.

According to one embodiment, it is possible to provide a fast response and processing for a request of a customer terminal by copying and distributing a file based on the distribution of the request of the customer terminal to the specific file and the popularity of the specific file.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram of a distributed file sharing system according to an embodiment; Fig.
2 is a diagram illustrating a configuration of any one of servers constituting a distributed file sharing system according to an exemplary embodiment;
3 is a flowchart illustrating a method of replicating a file in a distributed file sharing system according to an exemplary embodiment;
FIG. 4 is a flowchart illustrating a method of calculating a popularity of a file to be duplicated in a distributed file sharing system according to an exemplary embodiment; FIG.
5 is a flow diagram illustrating a method for determining at least one server to distribute a file in a distributed file sharing system according to one embodiment.
6 is a flowchart illustrating a method of processing a file upload request of a customer terminal in a server of a distributed file sharing system according to an exemplary embodiment.
FIG. 7 is a flowchart illustrating an operation when a file upload request can not be processed in a server of a distributed file sharing system according to an exemplary embodiment; FIG.
8 is a flowchart illustrating a method of processing a file download request of a customer terminal in a distributed file sharing system according to an exemplary embodiment.
9 is a diagram illustrating the meaning of symbols used in a file replication method applied to a server of a distributed file sharing system according to an exemplary embodiment;
10 is a flowchart illustrating a file copying method in a distributed file sharing system according to an embodiment.
FIG. 11 is a diagram for explaining the operation of the distributed file sharing system according to one embodiment; FIG.

In the following, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

Various modifications may be made to the embodiments described below. It is to be understood that the embodiments described below are not intended to limit the embodiments, but include all modifications, equivalents, and alternatives to them.

The terms used in the examples are used only to illustrate specific embodiments and are not intended to limit the embodiments. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this embodiment belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. In the following description of the embodiments, a detailed description of related arts will be omitted if it is determined that the gist of the embodiments may be unnecessarily blurred.

1 is a block diagram of a distributed file sharing system according to an embodiment of the present invention. 1, a distributed file sharing system 100 according to an embodiment includes a plurality of servers 200, 200-1 and 200-2 and a plurality of servers 200, 200-1, 200-2, ..., and a plurality of customer terminals (300, 300-1, 300-2, ...) communicating with each other via a network (50). For example, the plurality of servers 200, 200-1, 200-2, .. are N, and the plurality of client terminals 300, 300-1, 300-2, M number. The network 50 may include a wired and wireless communication network.

The server 200, which is a server of any one of the plurality of servers 200, 200-1, 200-2, ... may include a server interface 210, a database 220, and a file sharing module 230 have. The specific configuration of the server 200 will be described with reference to Fig.

The server interface 210 is connected to the server 200 through the communication between the server 200 and the customer terminals 300, 300-1, 300-2, And receives the request. The request of the customer terminals 300, 300-1, 300-2, .. includes, for example, a file search, a file upload request, and a file download request.

The server interface 210 includes an event handler (see 212 in FIG. 2) for performing file replication and distribution for processing according to the request of the customer terminals 300, 300-1, 300-2,.

The server interface 210 determines a replication coefficient of a file to be duplicated and a replication coefficient of a file to be duplicated based on a request of the client terminals 300, 300-1, 300-2, You can choose one server. A method for the server interface 210 to calculate the popularity of a file, determine at least one server to distribute a file to be duplicated and replicate the file will be described with reference to FIGS. 3 to 5. FIG.

The database 220 stores and manages the metadata for the file search and the log data, which is the processing result of the requests of the customer terminals 300, 300-1, 300-2,. The database 220 may be, for example, a distributed based NoSQL storage system or a distributed based NoSQL database. The NoSQL database is a space for storing highly optimized key values for simple retrieval and addition operations, with significant performance gains associated with latency and throughput. NoSQL databases are widely used for large data and commercial use of real-time Web applications, and can also be referred to as "Not only SQL" in that they can use the SQL-based query language.

The file sharing module 230 may store files to be shared among the plurality of customer terminals 300, 300-1, 300-2, ....

The customer terminal 300 of any one of the customer terminals 300, 300-1, 300-2,... May include a customer interface 305. The customer interface 305 may communicate with a plurality of servers and may generate a customer request.

The customer terminals 300, 300-1, 300-2,... May be personal communication terminals such as a smart phone, a PDA, and the like, a PC, and a notebook. The customer interface 305 may be a software program running on the customer terminal 300. The customer interface 305 can communicate with a plurality of servers 200, 200-1, 200-2, ..., for example, based on the HTTP protocol.

2 is a diagram illustrating a configuration of any one of servers that constitute a distributed file sharing system according to an exemplary embodiment of the present invention. Referring to FIG. 2, the configuration and operation of the server interface 210, the database 220, and the file sharing module 230 constituting the server 200 are shown.

The server interface 210 may include a status updater 211, an event handler 212, and a query processor 213.

The status updater 211 may periodically collect and update the disk capacity of the server 200 and the status information of the server 200. [

The event handler 212 may process events involving processing of requests of customer terminals, file replication and distribution. For example, when the upload of a file is requested by the customer terminal, the event handler 212 extracts the metadata of the file requested to be uploaded, and determines whether the metadata of the file requested to be uploaded exists in the file sharing module The metadata of the file requested to be uploaded to the database can be transmitted. A method in which the event handler 212 processes a file upload request of the customer terminal will be described with reference to FIG.

When the uploading of a file is requested by the client terminal, the event handler 212 searches for another server adjacent to the server 200 based on whether or not the server 200 has a sufficient space for storing the file to be uploaded , And the information about the other server can be transmitted to the client terminal requesting the upload. The operation when the event handler 212 can not process the file upload request of the customer terminal will be described with reference to FIG.

When the download of a file is requested by the customer terminal, the event handler 212 determines the storage location of the file to be downloaded based on the metadata of the file to be downloaded, acquired from the database, It can be checked whether or not it exists. A method in which the event handler 212 processes a file download request of the customer terminal will be described with reference to FIG.

The event handler 212 may collect log data and calculate the total number of requests of the client terminals for the file processed by the server 200 based on the collected log data. The event handler 212 can measure the popularity of the file processed in the server 200 based on the total number of requests and calculate the copy count of the processed file. The 'file replication count' can be understood as the number of copies of the file.

The event handler 212 can calculate the ratio of the popularity of the file in the server 200 and the network cost required for the server 200 to replicate the file. The event handler 212 can determine whether to copy the file to the server 200 based on the ratio of the popularity of the file and the network cost. A method for copying a file by the distributed file sharing system according to an embodiment will be described with reference to FIG.

The query processor 213 may process a query that includes reading, inserting, and storing data for the database 220 based on the information collected by the state updater 211.

The database 220 may include at least one of a cluster state store 221, a log store 222, a file data store 223, and a routing information store 224.

The cluster status store 221 stores status information including the cluster status of the server 200 and the disk usage capacity of the server 200. [ The cluster state store 221 may store the cluster state of the entire distributed file sharing system.

The log storage 222 stores log data which is a result of a request of the customer terminals processed by the server 200. [ At this time, the log data may be generated by the event handler 212 in response to the request of the customer terminals.

The file data storage 223 stores and manages data generated at the request of the customer terminals and data for file retrieval. The file data storage 223 can store and manage data for searching for a file, for example, a file name, a size of a file, and a storage location of a file.

The routing information storage 224 stores results of measuring data movement time and / or data movement speed from the server 200 to the other servers 200-1, 200-2, ....

Each of the repositories 221, 222, 223, and 224 included in the database 220 may be distributed and stored at a specific location.

3 is a flowchart illustrating a method of copying a file in a distributed file sharing system according to an exemplary embodiment. Referring to FIG. 3, a distributed file sharing system according to an exemplary embodiment calculates a popularity of a file to be cloned based on a request of customer terminals (310). A method for calculating the popularity of a file to be copied by the distributed file sharing system will be described with reference to FIG.

The distributed file sharing system is based on the replication cost of a file to be cloned and at least one server to distribute a file among a plurality of servers based on the popularity of the file to be cloned and the network cost of the server required to clone the file to be cloned (320). A method by which the distributed file sharing system determines a server to distribute a file will be described with reference to FIG.

The server replicates 330 the file to at least one server determined in step 320 according to the replication count.

4 is a flowchart illustrating a method of calculating a popularity of a file to be duplicated in a distributed file sharing system according to an exemplary embodiment. Referring to FIG. 4, the distributed file sharing system according to an exemplary embodiment may collect log data, which is a processing result of requests of client terminals, from a database of each server using an event handler of each server (410).

Based on the log data collected in step 410, the distributed file sharing system may calculate the total number of requests of the client terminals for files processed by any one of the plurality of servers included in the distributed file sharing system (420). The distributed file sharing system can calculate the total number of requests of the customer terminals using Equation (1) described later, for example.

The distributed file sharing system may calculate the popularity of the processed file in any one of the servers using the total number of requests calculated in operation 420. The distributed file sharing system can calculate the popularity of files processed in any one server by using, for example, Equation (2) described later.

5 is a flowchart illustrating a method for determining at least one server to distribute a file in a distributed file sharing system according to an exemplary embodiment of the present invention. Referring to FIG. 5, a distributed file sharing system according to an exemplary embodiment may calculate a ratio of a popularity of a file in at least one server among a plurality of servers (510). The distributed file sharing system can calculate the ratio of the popularity of the file on at least one server using, for example, Equation (5) and Equation (6) to be described later.

The distributed file sharing system may calculate the network cost of at least one server to replicate the file (520).

The distributed file sharing system may determine 530 the server to distribute the file to at least one server based on the ratio of the popularity of the file calculated in step 510 and the network cost calculated in step 520. [ The distributed file sharing system can determine a server to distribute a file using, for example, Equation (7) to be described later.

6 is a flowchart illustrating a method of processing a file upload request of a customer terminal in a server of a distributed file sharing system according to an exemplary embodiment. The upload request generated by the client terminal can be processed by the server, more specifically by the event handler of the server interface.

After the communication connection 610 with the client terminal, the server extracts the metadata of the file requested by the client terminal (620).

The server can determine whether the same metadata as the file requested by the client terminal exists in the database using the query processor (630).

If it is determined in step 630 that the same metadata exists in the database, the server considers that the same file exists, and transmits a file confirmation message requesting confirmation to the client terminal of the file to be uploaded (step 680). After transmitting the file confirmation message, the server may terminate the communication with the customer terminal (670).

If it is determined in step 630 that the same metadata does not exist in the database, the server may transmit metadata extracted in step 620 to the database by the query processor in operation 640.

The server can receive a file to be shared from the client terminal (650).

The server may store the file transmitted in step 650 to the file sharing module of the server (660) and then terminate the communication with the customer terminal (670).

7 is a flowchart illustrating an operation when a file upload request can not be processed in a server of a distributed file sharing system according to an exemplary embodiment. 7, a distributed file sharing according to one embodiment of the system if it is determined that the previous file with the same metadata, the client terminal at step 630 requesting an upload of Figure 6 is present, be uploaded to a current access server i It may determine whether there is enough space to store the file (710).

If it is determined in step 710 that there is a sufficient space for storing the file to be uploaded to the currently connected server i , the distributed file sharing system performs a file upload process of steps 640 to 670 of FIG. 6 between the server i and the client terminal . ≪ / RTI > The distributed file sharing system can determine whether there is enough space to store the file to be uploaded to the currently connected server i based on information about the disk capacity of the server, for example.

If it is determined in step 710 that there is not enough space to store the file to be uploaded to the server i currently connected to, the distributed file sharing system searches the server i closest to the currently accessed server i based on the information stored in the routing information storage of each server j (720).

The distributed file sharing system may transmit a message including the information of the server j to the client terminal (730).

The distributed file sharing system terminates the communication (740) between the client terminal and the currently connected server i, and can connect the communication between the client terminal and the newly searched server j . The server j, which is connected to the client terminal, can perform the file uploading process of steps 640 to 670 of FIG.

8 is a flowchart illustrating a method of processing a file download request of a customer terminal in a distributed file sharing system according to an exemplary embodiment. The download request generated by the client terminal may also be performed by a server, more specifically an event handler of the server interface.

Referring to FIG. 8, the server performs a communication connection with a customer terminal (810), and extracts metadata of a file to be downloaded from the database using the query processor (820).

In operation 830, the server determines whether there is a file to be downloaded to the file sharing module of the current server by checking the location where the file to be downloaded is stored based on the metadata extracted in operation 820.

If it is determined in step 830 that there is a file to be downloaded to the file sharing module, the server transmits the file directly to the client terminal (840) and terminates the communication between the server and the client terminal (850).

If it is determined in step 830 that there is no file to be downloaded to the file sharing module, the server may transmit the location of the file to be downloaded to the client terminal (860). At this time, the server can transmit a message informing the location of the file to be downloaded to the client terminal.

Upon receiving the location of the file to be downloaded, the customer terminal establishes a new communication connection with another server that owns the file to be downloaded (870), and receives the file from another server (880).

After receiving the file, the customer terminal terminates the communication between the other server and the customer terminal (890), and can also terminate the communication with the server that initially connected to the customer terminal (850).

As described above, the customer terminal generates two types of requests, such as upload and download, and transmits them to the server. In this process, the client terminal can communicate with the server using the HTTP protocol. The HTTP protocol is easy to use and has the advantage of being fast when transferring large files.

In the distributed file sharing system according to an exemplary embodiment, the number of requests from the client terminals connected to the servers storing the files may vary depending on the popularity of the files stored in the servers. A server with a high popularity may cause an excessive network load due to a request from a client terminal. Also, in systems operating on a distributed environment, it is generally determined that the amount of movement of data through the network is small and high depending on a storage location of data and a location of data, and thus the performance of the system can be determined.

Accordingly, in one embodiment, the performance of the distributed file sharing system can be improved by using a file replication method that minimizes the movement of data through the network and reduces the load on the network. The server according to an exemplary embodiment calculates a replication count (the number of files to be duplicated) of a file and a place to replicate the file (that is, a duplication of duplication) based on the popularity of the file and the request distribution data of the client terminal of the corresponding file processed by each server A server to distribute a file to be distributed), thereby reducing the load on the network while minimizing the movement of data over the network.

Hereinafter, a method of copying a file applied to a server of a distributed file sharing system will be described with reference to FIGS. 9 to 10. FIG. Prior to the description, the meaning of symbols used in the following file replication method will be described with reference to FIG. 9 below.

10 is a flowchart illustrating a file copying method in a distributed file sharing system according to an embodiment. Referring to FIG. 10, a process of the distributed file sharing system replicating the file j is shown. The following file duplication process can be performed by the event handler included in the server interface of the server.

The distributed file sharing system may load data for calculating the popularity of file j (1005). At this time, the event handler of each server can collect data for analyzing the popularity of the file j using the query processor of the server. The data for analyzing the popularity of the file j may be, for example, data stored in a log storage for storing log data that is a result of a request of customer terminals processed by the server. The data for analyzing the popularity of the file j may include, for example, the node where the file is used and the number of times the file was used in the node.

The distributed file sharing system may calculate 1010 the popularity of the file j to be replicated throughout the distributed file sharing system based on the data loaded in step 1005. In step 1010, the distributed file sharing system may calculate the total number of requests R of the client terminals for file j processed in each of the servers using Equation (1) below.

The distributed file sharing system can measure the popularity (FPj) of the file j in each server using Equation (2).

)를 산출할 수 있다. The distributed file sharing system may determine 1015 the number of files to be replicated in the distributed file share, i . E., The replica count of the file, based on the popularity (FPj) of the file j at each server calculated at step 1010. A distributed file sharing system, for example, by using the formula 3 below in the file j replication factor (

) Can be calculated.

는 최대 복제될 파일의 수를 의미하고

은 최소 복제될 파일의 수를 의미한다.

,

는 파일 공유 서비스를 구동하기 위한 분산 파일 공유 시스템을 운영하는 관리자(또는 사용자)가 직접 적절하게 설정할 수 있다. In Equation (3)

Means the maximum number of files to be replicated

Means the minimum number of files to be replicated.

,

(Or a user) who operates the distributed file sharing system for driving the file sharing service can directly set it appropriately.

)를 가까운 정수로 반올림하여 정수화할 수 있다(1020).The distributed file sharing system calculates the replication coefficient of the file j calculated by Equation (3)

) May be rounded to a nearest integer to perform integer conversion (1020).

)를 파악하여

와 같이 정수화된 복제 계수

와 비교할 수 있다(1025). 여기서, X_i,j는 0 또는 1의 이진 값을 가질 수 있다. X_i,j는 파일 j가 서버 i에 저장되어 있는 경우에 '1'을, 그렇지 않는 경우에 '0'을 가질 수 있다. X_i,j = 1 인 경우는 해당 노드 i로 파일 j(혹은 데이터 j) 요청 시에 추가적인 네트워크 비용이 필요 없다는 의미로 이해될 수 있다. X_i,j = 0인 경우에는 노드 i에 특정 파일 j가 요청될 시 추가적인 네트워크 비용이 소모된다는 의미로 이해될 수 있다. The distributed file-sharing system determines the number of file js existing in the entire system

)

Lt; RTI ID = 0.0 >

(1025). ≪ / RTI > Here, X _{i, j} may have a binary value of 0 or 1. X _{i, j} may have a value of '1' if file j is stored in server i, and '0' otherwise. In the case of X _{i, j} = 1, it can be understood that no additional network cost is required when a file j (or data j) is requested to the corresponding node i. If X _{i, j} = 0, it means that additional network cost is consumed when a specific file j is requested to node i.

보다 많은 수의 파일을 유지하고 있는 경우, 분산 파일 공유 시스템은 파일 복제를 수행하지 않고 동작을 종료할 수 있다. In step 1025, the entire system (distributed file sharing system)

If a larger number of files are maintained, the distributed file sharing system can terminate operations without performing file replication.

보다 작은 수의 파일을 유지하고 있는 경우, 분산 파일 공유 시스템은 그 차이만큼 파일 복제를 수행하기 위해 특정 서버에서 파일의 인기도를 계산하기 위한 데이터를 로딩할 수 있다(1030). 단계(1030)에서 로딩되는 데이터는 단계(1005)에서 로딩된 데이터를 이용하여 계산된 파일 복제 계수(n_j) 등을 포함할 수 있다. 분산 파일 공유 시스템은 파일 복제 계수(n_j)를 로딩하여 몇 개의 파일을 복제해야 하는지를 결정할 수 있다. In step 1025, the entire system is multiplied with the integerized replication coefficient

If a smaller number of files are being maintained, the distributed file sharing system may load 1030 data for computing the popularity of the files at a particular server to perform file replication by that difference. The data loaded in step 1030 may include the file replication factor n _j calculated using data loaded in step 1005, and so on. The distributed file sharing system may load the file replication factor (n _j ) to determine how many files should be replicated.

)를 계산할 수 있다.The distributed file sharing system may calculate 1035 the popularity of a particular file j at a particular server i based on the data loaded at step 1030. The distributed file sharing system uses, for example, the following equation (4) to determine the popularity of a particular file j in a particular server i

) Can be calculated.

)을 구할 수 있다(1040). 분산 파일 공유 시스템은 예를 들어, 아래의 [수학식 5]를 이용하여 인기도 집합에서의 최소 값(

)을 구할 수 있다. The distributed file sharing system determines, based on the computation results in step 1035, the minimum value in the popularity set of a particular file j at a particular server i

(1040). For example, the distributed file sharing system uses the following formula (5) to calculate the minimum value in the popularity set

) Can be obtained.

)을 이용하여 분산 파일 공유 시스템을 구성하는 각 서버에서 특정 파일 j을 처리하는 인기도의 비율(

)을 산출할 수 있다(1045). 분산 파일 공유 시스템은 예를 들어, 아래의 [수학식 6]을 이용하여 각 서버에서 특정 파일 j을 처리하는 인기도의 비율(

)을 산출할 수 있다. The distributed file sharing system determines the minimum value in the popularity set (1040)

) Is used to determine the ratio of the popularity of processing a specific file j in each server constituting the distributed file sharing system

(1045). For example, the distributed file sharing system calculates the ratio of the popularity of processing a specific file j in each server using Equation (6) below

) Can be calculated.

A distributed file sharing system is a set of servers on which file j to be copied is stored V _j And determines a server having a minimum cost based on the ratio of the popularity obtained in step 1045 and the network cost (NC) used for processing the file as a server to distribute (store) the file to be finally copied (1050). V _j Can be understood as a set of servers in which file j is stored when X _{i, j} = 0.

The network cost (NC) can be calculated, for example, to cost about $ 0.12 per month to utilize a network of 1 GB per month, according to the definition of network cost to utilize Google's cloud platform. The network cost can be understood as the network cost in case of X _{i, j} = 0 in the Google cloud platform.

)를 결정할 수 있다. In step 1050, for example, a server having a minimum cost using Equation (7) below

Can be determined.

는 각 서버의 한계 용량을 나타내고, b_j는 파일 j의 사이즈를 나타낸다. V in Equation (7) means a set of servers that do not store a specific file,

Represents the limit capacity of each server, and b _j represents the size of the file j.

을 만족할 때까지 계산을 수행하고, 최종적으로 결정된 서버로 파일 j를 복제할 수 있다. 이때, 분산 파일 공유 시스템은 각 서버의 한계 용량인

를 확인한 후, 파일 j의 복제가 수행되도록 할 수 있다. The distributed file sharing system uses Equation (7) in step 1050

, And then copy the file j to the finally determined server. At this time, the distributed file sharing system has a limit capacity

The file j can be reproduced.

11 is a diagram for explaining the operation of the distributed file sharing system according to an embodiment. 11, the number of servers constituting the distributed file sharing system according to an exemplary embodiment | M | = 4, and the file to be copied stored in the server M ₄ is F ₁ .

Assuming that the popularity (FP ₁ ) of the file F _{1 to} be copied is 0.2, the maximum number of files to be copied (C_max) = 4, and the minimum number of files to be copied (C_min) = 2, The replication coefficient n ₁ of the file of ₁ can be calculated by the above-described equation (3) as follows.

n ₁ = (C_max - C_min) * FP _{1 +} C_min = (4-2) * 0.2 + 2 = 2.4

In the distributed file sharing system, the calculated replication factor of 2.4 is rounded up to a natural number of 2, and then 2 can be selected as a replication factor.

Then, the distributed file sharing system can calculate the popularity MP _{i, 1} for F ₁ for each server based on the request processed for F ₁ in each server.

The distributed file-sharing system uses the popularity of F ₁ for each server _The weight (a _{i, 1} ) can be calculated by selecting the minimum value (Z ₁ ) based on _{i and 1} . At this time, the set of servers that do not store F ₁ can be expressed as V = {M ₁ , M ₂ , M ₃ }. The servers (M ₁ , M ₂ , M ₃ ) belonging to V are X _i , ₁ = 0 (in other words, X _1,1 = 0, X _2,1 = 0, X _3,1 = 0), and the server M ₄ not belonging to V can be set to X ₄ , ₁ = 1.

The distributed file sharing system uses X _i , ₁ = The server is set to 0, the weights (a _{i, 1)} and the file F ₁ on the basis of the result of multiplying the network cost (NC) in the respective other words, the server belongs to _{V (M 1, M 2,} M 3) V ₁ *, which indicates the location of the server to be replicated, can be selected.

The embodiments described above may be implemented in hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, such as an array, a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the drawings, various technical modifications and variations may be applied to those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

50: Network
100: Distributed File Sharing System
200: Server
210: Server Interface
220: Database
230: File Sharing Module
300: Customer terminal
305: Customer interface

Claims (10)

A distributed file sharing system including a plurality of servers and a plurality of client terminals communicating with the plurality of servers,
One of the plurality of servers
Receiving a request of the customer terminals including a file search, a file upload request, and a file download request through communication between any one of the servers and the plurality of customer terminals, A server interface including an event handler for performing file replication and distribution; And
A database for storing and managing the metadata for the file search and the log data which is a processing result of the request of the customer terminals
A distributed file sharing system. The method according to claim 1,
The server interface
And selects at least one server to distribute the replica coefficient of the file to be replicated and the file to be replicated, based on the popularity of the file and the request of the customer terminals for the file. The method according to claim 1,
The event handler
Processing the request of the customer terminals, processing the event including the file copying and distribution,
The server interface
A status updater that periodically collects and updates disk capacity of any one of the servers and status information of any one of the servers; And
A query processor for processing a query that includes reading, inserting, and storing data for the database based on the information collected by the state updater
The distributed file sharing system further comprising: The method according to claim 1,
When a file upload is requested by the customer terminal,
The event handler
And searching for another server adjacent to the one server based on whether there is sufficient space for storing the file to be uploaded to the one server and transmitting the information about the other server to the client terminal requesting the upload, Distributed File Sharing System. The method according to claim 1,
The event handler
The method according to claim 1, further comprising: collecting the log data; calculating a total number of requests of the client terminals for a file processed by the one server based on the collected log data; The popularity of the file, and calculates the replication coefficient of the processed file based on the popularity of the file. The method according to claim 1,
The event handler
A ratio of the popularity of the file in any one of the servers and a network cost required for one of the servers to replicate the file is calculated, and the ratio of the popularity of the file and the network cost A distributed file sharing system that determines whether to replicate the file to a single server. The method according to claim 1,
The database
A file data storage for storing and managing data generated upon request of the customer terminals and data for searching for the file;
A log storage for storing log data which is a result of a request of the client terminals processed by any one of the servers;
A routing information storage for storing a result of measuring a data movement time from one of the servers to the other servers; And
A cluster state storage for storing state information including a cluster state of any one of the servers and a disk usage capacity of any one of the servers,
The distributed file sharing system. A method for replicating a file in a distributed file sharing system including a plurality of servers and a plurality of client terminals communicating with the plurality of servers,
Calculating a popularity of a file to be cloned based on a request of the customer terminals;
A replication coefficient of the file to be duplicated and at least one server for distributing the file among the plurality of servers based on the popularity of the file to be duplicated and the network cost of the server for duplicating the file to be duplicated Determining; And
Replicating the file to the determined at least one server according to the replication factor
A method for replicating a file in a distributed file sharing system. 9. The method of claim 8,
The step of determining at least one server to distribute the file
Calculating a ratio of the popularity of the file at the at least one server among the plurality of servers;
Calculating a network cost for the at least one server to replicate the file; And
Determining the at least one server as a server to distribute the file based on the ratio of the popularity of the file and the network cost
A method for replicating a file in a distributed file sharing system. 9. The method of claim 8,
The step of calculating the popularity of the file to be duplicated
Collecting log data which is a processing result of a request of the customer terminals;
Calculating a total number of requests of the client terminals for the file processed by any one of the plurality of servers based on the collected log data; And
Calculating the popularity of the file processed by any one of the servers using the total number of requests
A method for replicating a file in a distributed file sharing system.

Images