KR101272877B1 - Apparatus and method for seperating partition in distributed file system - Google Patents

Abstract

It provides a technique for partitioning in a large file distribution management system that distributes and manages a large amount of data to each partition. According to an embodiment of the present invention, a partitioning apparatus of a distributed file system includes a frequency measuring unit configured to measure a frequency of connection to a partition for distributing and storing data every first period, and a second frequency interval every second period that is an integer multiple of the first period. Partitioning the partition based on a load calculation unit that generates a first load by using connection frequencies corresponding to two cycles, and a second load that calculates first loads corresponding to a third period for every third period that is an integer multiple of the second period. Characterized in that it comprises a split control unit for controlling whether or not.

Description

Partitioning device and method for distributed file system {APPARATUS AND METHOD FOR SEPERATING PARTITION IN DISTRIBUTED FILE SYSTEM}

The present invention relates to a method for effectively partitioning a partition for managing data in a distributed data system for managing a large file. More specifically, the present invention relates to a technique of dividing a partition having a high access frequency based on a user's access frequency to a partition, and determining a position to divide by reflecting the access frequency even when partitioning.

With the development of wired and wireless communication technologies and the development of computer-related technologies, researches have been conducted on technologies for effectively managing data. With the advent of user-generated data such as UCC and user-centric applications, the amount of data to be managed at once is also increasing rapidly.

In addition, with the increase in the capacity of multimedia data and the development of computer processing speed, the size of individual data is also very large. Therefore, there is an urgent need for data management techniques that are rapidly increasing in both size and quantity.

Distributed large data system exists as a system for managing large data. Distributed large data systems consist of a master server and multiple partition servers. The master server manages partition servers and manages the location of partition servers to which data belongs. The partition server manages the partition to which the actual data belongs, and the data is arranged and managed sequentially based on the key.

When retrieving data from the user side, the master server first provides information on which partition server exists, and the user directly accesses the partition server based on the provided information and uses the data. In this case, when a large number of clients access a specific partition or a specific data, the partition server that needs to manage a specific partition is heavily loaded. Thus, in this case, partitions are partitioned and allocated to other available partition servers, thereby avoiding overloading the partition servers.

However, in the case of high-capacity data having a very high frequency of connection and a very large size, even when partitioning, there is a part of data existing in the existing partition. Therefore, it is difficult to effectively distribute a connection to a partition having a high frequency of access only by the function of dividing the partition collectively. Accordingly, there is a need for a technique that actively controls the partitioning and reflects the access frequency in partitioning. Growing.

In order to cope with the above problems and needs, an object of the present invention is to provide a technique capable of effectively controlling partitioning and controlling the frequency of access to each partition even when partitioning.

In order to achieve the above-mentioned object, a partition partitioning apparatus of a distributed file system according to an embodiment of the present invention, the frequency measuring unit for measuring the frequency of connection to the partition for distributing and storing data for each first period, and the first A load calculation unit generates a first load by using connection frequencies corresponding to a second period for every second period that is an integer multiple of the period, and calculates first loads that correspond to a third period for each third period that is an integer multiple of the second period. And a partition control unit for controlling whether to partition the partition based on the second load.

According to an embodiment of the present invention, a partitioning method of a distributed file system may include: measuring, by a frequency measuring unit, a connection frequency to a partition that stores and distributes data every first period; Generating, by the load computing unit, the first load by using connection frequencies corresponding to the second period for every second period that is an integer multiple of the first period; And controlling, by the division controller, whether to partition the partition based on a second load that calculates first loads corresponding to the third period for every third period that is an integer multiple of the second period.

In addition, the partition control unit may calculate a load based on the frequency of each data element included in the partition based on the third period when the partition target partition is divided, and determine a position to partition the partition based on the calculated load. It may further include.

According to the partitioning apparatus and method of the distributed file system according to an embodiment of the present invention, a partition to be partitioned is selected according to a user's access frequency. Therefore, since partitions are not collectively partitioned, efficient partitioning can be performed, and the load of a specific partition server can be distributed. In addition, since the load for each data element is calculated and divided at the time of partition partitioning, there is an effect of effectively distributing the connection to the data of high capacity and high connection frequency.

Hereinafter, a partition partitioning apparatus of a distributed file system according to an exemplary embodiment of the present invention will be described.

1 is a block diagram of a partitioning apparatus 100 of a distributed file system according to an exemplary embodiment of the present invention.

Referring to FIG. 1, a partition partitioning apparatus 100 of a distributed file system according to an exemplary embodiment of the present invention includes a frequency measuring unit 101, a load calculating unit 103, and a splitting control unit 102.

In an embodiment of the present invention, a large amount of data is stored in one logical table by sorting by key value. The table is divided into partitions as the data grows. The partitions P1, P2 to Pn, hereinafter, will be identified by being represented by P1 unless a specific partition is designated, are a subset of horizontally partitioned data of the table, and are units of connection distribution and load measurement.

The frequency measuring unit 101 measures the connection frequency of the partition P1 every first period (for example, 1 second). The access frequency is also called the frequency of each partition. The frequency measuring unit 101 is connected to the partition server 104 via the network 106.

In the frequency measuring unit 101, the connection frequency of the partition P1 is measured for partitions included in all partition servers. Therefore, the frequency measuring unit 101 may be used in parallel connection frequency measurement that can measure the connection frequency of the partitions included in all partition servers.

The load calculator 103 may generate a first load by using connection frequencies corresponding to the second period every second period (for example, four seconds) that is an integer multiple (for example, four times) of the first period. To perform.

The access frequency may change over time, and when partitioning based only on the access frequency, the partitioning purpose for a large amount of data may not be properly achieved. This is because partition partitioning and management cannot effectively select a partition to be partitioned for a large amount of data. Therefore, the first load, which is a linear function of the connection frequency, is calculated and used for selecting partitions to be partitioned.

The load calculator 103 uses connection frequencies corresponding to the second period for every second period that is an integer multiple of the first period when calculating the first load. Corresponding to the second period means using the access frequencies measured in the plurality of first periods included in the second period.

However, if the connection frequency measured in the plurality of first periods is treated the same, the connection frequency or load of the current partition is also affected by the previously generated connection frequency or load, so that no effective load calculation is made. Therefore, using a simple average of the access frequency in units of time is not effective in determining which partition to partition.

Therefore, when calculating the first load, the load calculation unit 103 calculates the first load by assigning a weight to each unit of time. That is, the first load is calculated by applying a weight to each first period and reflecting the weight on each connection frequency corresponding to each first period. The following equation is used to calculate the first load.

L1 = (aFt1 + bFt2 + cFt3 + dFt4) / 4

In an embodiment of the present invention, the access frequency for four first periods is calculated. L1 denotes a first load of the partition, and Ft1, Ft2, Ft3, and Ft4 are arranged in chronological order of the connection frequencies measured for each first period. The coefficients a, b, c, and d, which are multiplied by the respective access frequencies, are variables that are multiplied so as to give a weight in time order for each measured first period. In an embodiment of the present invention, a = 1, b = 2, c = 3, and d = 4. However, any calculation method that can weight the connection frequency in chronological order would be possible.

The division control unit 102 generates a second load in which the first load calculated by the load operation unit 103 is calculated once again, and determines a partition to be divided. The partitioning is controlled based on a second load obtained by calculating first loads corresponding to the third period every third period (for example, 16 seconds) that is an integer multiple (for example, four times) of the second period. The first loads corresponding to the third periods refer to a plurality of first loads calculated for every second periods included in the third period.

Calculating the load value applied to the partition in two steps is because it is relatively easy to determine the threshold for partition partitioning when and when it is not necessary, i.e. In addition, in order to effectively partition a partition for access to a large file, a two-step load value calculation is required to determine a partition to be partitioned correctly.

Similar to the calculation in the load calculating section 103, if the first loads measured in the plurality of second periods are treated the same, the load of the current partition is affected by the load that occurred previously, so that the effective load calculation is not performed. Thus, using a simple average of the first load in units of time is not effective in determining which partition to partition.

Therefore, when calculating the second load, the division controller 102 calculates the first load by assigning a weight to each unit of time to generate the second load. That is, the second load is calculated by applying a weight to each second period and reflecting the weight on each load corresponding to each second period. The following equation is used to calculate the second load.

L2 = (aL1T1 + bL1T2 + cL1T3 + dL1T4) / 4

In Equation 2, L2 means a second load during the third period of the measurement target partition. L1T1 to L1T4 mean a first load for each second period. The coefficients a, b, c, and d, which are multiplied corresponding to each second period, represent weights for each time zone, as in the above-described coefficients of Equation 1 above. In the embodiment of the present invention, since four second periods are set as one third period, a total of four first periods are used for the calculation. The coefficients a, b, c, and d are 1, 2, 3, and 4, respectively, as in the coefficient of Equation 1 above.

The partition controller 102 determines whether to partition the partition based on the calculated second load. If the second load exceeds the threshold, it is determined as a partition to be divided. The threshold value may vary depending on the unit of the access frequency and the coefficients a, b, c, and d, and description thereof will be omitted.

Since the partition to be divided is determined based on the connection frequency and the load calculated on the basis of a predetermined cycle, it is possible to accurately and efficiently partition the partition, unlike the conventional technique of partitioning the partition at once. Therefore, the performance degradation due to the increased load on the partition server will also be effective to effectively resolve.

In an embodiment of the present disclosure, the partition controller 102 may further include a function for partitioning the partition selected based on the above-mentioned second load. That is, the method may further include a function of determining which partition to partition the data table of which the partition is based.

The partition control unit 102 calculates the total access frequency for the data elements in the partition based on the third period. Therefore, the frequency measuring unit 101 will receive the access frequency for each first period directly from the division controller 102. The division control unit 102 adds, for example, the total number of connection frequencies in the first 16 periods for each data element. When partitioning, the load for each data element is calculated by using the total access frequency or by using Equations 1 and 2 reflecting the two-stage weights as in the method of calculating the first and second loads. Will be.

When the load for each data element is calculated, partitioning is based on data elements that can reduce the total load in half. Regardless of the access frequency, if the data element is divided based on the data element located in the middle of the data element, it is difficult to effectively achieve the object of the present invention because it is divided without reflecting the access frequency. Therefore, based on the access frequency collected for each data element, the load is based on the half. As a result, even in partitioning of a large amount of data, the efficiency can be increased, thereby effectively distributing the connection frequency.

In order to perform the above function, the frequency measuring unit 101, the load calculating unit 103, and the splitting control unit 102 may further include a function of measuring or calculating for each data element included in a partition when measuring a connection frequency or calculating a load. have.

2 illustrates an example of calculating the first load by the load calculator 103.

Referring to FIG. 2, the plurality of partitions P1, P2 ˜ Pn included in the partition server have an access frequency having a frequency measuring unit 101 for each first period t (i + 1) −ti, i being an integer. Is measured by. The number written on the time table of each partition means the frequency of access to the partition every first period. The weight is reset and weighted every second period, and 1, 2, 3, and 4 are set in order from the first first period between t0 and t1 to the last first period between t3 and t4.

In Fig. 2, partitions P1, P2 and Pn are compared. A weight is multiplied by the connection frequency for each first period, and an average according to the number of first periods 4 is calculated. As a result of the calculation of the first load 200, the first loads of the partitions P1, P2, and Pn in one second period t0 to t4 become 13, 9, and 10.75, respectively.

3 illustrates an example of calculating the second load in the division controller 102. Portions overlapping with the description of FIGS. 1 and 2 will be omitted.

Referring to FIG. 3, the first load calculated by the load calculator 103 is calculated by the division controller 102 for each second period T (i + 1) −Ti, i is an integer. The number written on the time table of each partition P1, P2, and Pn means a first load calculated for each partition every second period. The weight is reset and weighted every third period, and 1, 2, 3, and 4 are set in order from the first second period between T0 and T1 to the last second period between T3 and T4.

3, partitions P1, P2, and Pn are compared. The first load for each second period is multiplied by a weight, and an average according to the number 4 of the second periods is calculated. Referring to the result 300 of the calculation of the second load, the second loads of the partitions P1, P2, and Pn in one third period T0 to T4 become 58, 43.75, and 37.5, respectively.

It is determined whether the partition is divided for each partition P1, P2 to Pn. The partition P1 in which the second load of each partition P1, P2 to Pn exceeds a threshold value (for example, 50) is selected as the target of partition partitioning.

4 shows an example of partitioning of the partition control unit 102.

Referring to FIG. 4, the partition controller 102 determines a position to partition a partition based on the access frequency.

When the partition to be divided is selected, the partition control unit 102 determines which data element to partition the partition. In FIG. 4, it is assumed that connection frequency graphs 410 and 420 for two partitions to be divided.

Referring to the access frequency graph 410 for the first partition 430, assume a partition in which data elements (or each row of a data table) are arranged horizontally around a key value. The division control unit 102 calculates the connection frequency for each data element. In FIG. 4, the y-axis of the graphs 410 and 420 is used as the connection frequency. The x-axis represents the position of the data elements arranged horizontally.

Referring to the graph 410 of the first partition 430, the width of the graph represents the load per data element of the partition. Thus, the entire area of the graph from zero to the last data element 401 in the graph 410 for the first partition can be bisected, i.e. S1 = S2 (S1, S2 is the width of each graph portion). The first partition 430 is divided based on the data element 411 corresponding to the point 400.

Referring to the access frequency graph 420 for the second partition 440, a different access frequency distribution may be known from the graph 410 for the first partition 430. Therefore, the division control unit 102 calculates the frequency of connection of each data element sea for the second partition 440 as well.

Referring to graph 420 regarding second partition 440, the width of the graph also represents the load per data element of the partition. Thus, in the graph 420 for the second partition 440, the total width of the graph from 0 to the last data element 403 can be bisected, i.e., S3 = S4 (S3, S4 is the width of each graph portion). The second partition 440 is partitioned on the basis of the data element 412 corresponding to the point 402 to be.

5 is a flowchart illustrating a partitioning method of a distributed file system according to an exemplary embodiment of the present invention. In the following description, descriptions that overlap with the description of FIGS. 1 to 4 will be omitted.

Referring to FIG. 5, in the partition partitioning method of the distributed file system according to an exemplary embodiment of the present invention, first, when a first period arrives (S1), the frequency measuring unit 101 is divided into partitions at each arrival of the first period. The step of measuring the access frequency of (S2) is performed.

Thereafter, the load calculating unit 103 determines when the second period arrives (S3), and each time the second period arrives, the first load based on the connection frequencies corresponding to Equation 1 and the second period. Perform step S4 of calculating.

The division controller 102 determines when the third cycle arrives (S5), and each time the third cycle arrives, the division controller 102 determines the second load based on the first loads corresponding to Equation 2 and the third cycle. Performs step S6.

When the second load is calculated, the division controller 102 performs step S7 of determining whether the second load of the partition to be calculated exceeds the threshold Lth, and if not, again, the new first to third ones. The cycle is started, and if it is determined that the second load exceeds the threshold Lth, the target partition is divided.

In addition, although not shown in FIG. 5, the second part of the partition selected as the partitioning target based on the connection frequency, the first load or the second load, measured or calculated for each data element, as described above. Partitioning may be further included based on data elements that divide the load in half.

In an embodiment of the present invention, the starting points of the first to third cycles are set to be the same. This is because an error due to time difference occurs in the calculation of the above-mentioned connection frequency, the first load, and the second load when the start points of the first to third cycles are not the same.

The above description of the partitioning apparatus and method of the distributed file system according to the embodiment of the present invention mentioned above should be used only for descriptive purposes. Moreover, the above description should not limit the scope of the claims.

In addition to the above-described embodiments of the present invention, it will be obvious that equivalent inventions having the same function as the present invention will also belong to the claims of the present invention.

1 is a block diagram of a partitioning apparatus 100 of a distributed file system according to an exemplary embodiment of the present invention.

2 illustrates an example of calculating the first load by the load calculator 103.

3 illustrates an example of calculating the second load in the division controller 102.

4 shows an example of partitioning of the partition control unit 102.

5 is a flowchart illustrating a partitioning method of a distributed file system according to an exemplary embodiment of the present invention.

Claims (10)

A frequency measuring unit for measuring a frequency of connection to a partition for distributing and storing data every first period; A load calculation unit generating a first load by using connection frequencies corresponding to the second period for every second period that is an integer multiple of the first period; And a partition controller configured to control whether or not to partition the partition based on a second load that calculates first loads corresponding to the third period for every third period that is an integer multiple of the second period. Splitting device. The method according to claim 1, The load calculation unit and the division control unit, And partitioning the first load and the second load by assigning a weight to each time unit. The method according to claim 1, The load calculation unit, L1 is the first load, and Ft1, Ft2, Ft3, and Ft4 are arranged in chronological order of connection frequencies measured for each first period, and a, b, c, and d are weights for each first period. And calculating the first load by the equation L1 = (aFt1 + bFt2 + cFt3 + dFt4) / 4. The method according to claim 1, The division control unit, L2 is the second rod, L1T1, L1T2, L1T3, and L1T4 are first rods for each second period, and a, b, c, and d are weights for each second period. And computing the first load according to the equation L2 = (aL1T1 + bL1T2 + cL1T3 + dL1T4) / 4. The method according to claim 1, The division control unit, And dividing the partition to be partitioned, calculating the second load corresponding to the third period, and determining a location to partition the partition based on the second load. Measuring, by the frequency measuring unit, a frequency of access to a partition that distributes and stores data every first period; Generating, by the load computing unit, the first load by using connection frequencies corresponding to the second period for every second period that is an integer multiple of the first period; And And controlling, by the partition controller, whether to partition the partition based on a second load that calculates first loads corresponding to the third period for every third period that is an integer multiple of the second period. Split method. The method of claim 6, Generating the first rod, Calculating the first load by assigning a weight to the first period, Controlling whether or not the partition is divided, And calculating the second load by assigning weights to the second periods. The method of claim 6, Generating the first rod, L1 is the first load, and Ft1, Ft2, Ft3, and Ft4 are arranged in chronological order of connection frequencies measured for each first period, and a, b, c, and d are weights for each first period. Computing the first load according to the equation L1 = (aFt1 + bFt2 + cFt3 + dFt4) / 4. The method of claim 6, Controlling whether or not the partition is divided, L2 is the second rod, L1T1, L1T2, L1T3, and L1T4 are first rods for each second period, and a, b, c, and d are weights for each second period. Computing the first load according to the equation L2 = (aL1T1 + bL1T2 + cL1T3 + dL1T4) / 4. The method of claim 6, Controlling whether or not the partition is divided, Calculating the second load corresponding to the third period when the partition to be partitioned is divided; And And determining a location to partition the partition based on the second load.

Images