JP5575828B2 - Garbage collection execution device, garbage collection execution method, and garbage collection execution program - Google Patents

Description

  The present invention relates to a garbage collection execution device, a garbage collection execution method, and a garbage collection execution program.

  Conventionally, various systems using a distributed key-value store have been proposed with the progress of cloud computing. The distributed key-value store is a database that manages data by a pair of “key (key)” and “value (value)”. For example, Google (registered trademark) Bigtable, open source HBase, Hypertable Etc. are known.

  For example, each server group on the Bigtable PC cluster manages a table (partial table) divided into several ranges. Each server is provided with a log file for log advance writing (hereinafter referred to as a WAL (Write Ahead Logging) log) to ensure the durability of the write request. Bigtable (Distributed Key-Value Store) creates a file on GFS (Distributed File System).

  Here, in the distributed key-value store using the distributed file system, once added (Key, Value), even if an operation such as deletion is executed, (Key, Value) with a deletion flag is newly added. The original (Key, Value) is not deleted. Therefore, unnecessary (Key, Value) unnecessarily occupies the disk capacity of the distributed file system and decreases the search processing efficiency.

  Accordingly, in such a distributed key-value store, the disk occupancy is optimized by executing a garbage collection process in each server. For example, in the garbage collection process in the distributed key-value store, all the added, updated, and deleted files that have been persisted are read once, all the records to be deleted are deleted, and then a new persistent file is distributed to the distributed file system. Create on top. That is, in the garbage collection process, all information is read from the distributed file system, and almost all information (file after execution of deletion) is written to the distributed file system.

  As described above, the garbage collection process in the distributed key-value store reduces the disk occupancy of the distributed file system by deleting unnecessary files and improves the search processing efficiency. Since the processing is executed, the processing load is extremely high, and it is not possible to execute the processing simply by increasing the frequency. That is, at the time of executing the garbage collection process, a trade-off relationship between the disk occupation amount and the resource load (for example, the load on the disk I / O, the network, the CPU (Central Processing Unit), etc.) becomes important.

  So, for such problems, a method for setting the garbage collection processing execution interval, a method for providing the user with a garbage collection processing execution command, and a part that is executed as the number of records added increases. A technique for executing a garbage collection process when a table is divided is known.

F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber, “Bigtable: A Distributed Storage System for Structured Data,” OSDI (2006) S. Ghemawat, H. Gobioff, S.-T. Leung, “The Google File System,” SOSP (2003) A. Khurana, “HBase,” Hadoop Day (2010) D. Judd, “Hypertable: An Open Source, High Performance, Scalable Database,” OSCON (2008) M. Burrows, “The Chubby lock service for loosely-coupled distributed systems,” OSDI (2006)

  However, in the above-described conventional technology, there are cases where the resource load increases due to the occurrence of garbage collection processing concentrated in a short period of time. For example, in the method of setting the execution interval of the garbage collection process, only one execution interval can be set for all the partial tables. Therefore, the execution timing of the garbage collection process executed for each partial table overlaps, and load distribution is performed. I can't. In the method of providing a garbage collection execution command to the user, if the user does not have sufficient skills regarding the distributed key-value store, the garbage collection processing cannot be executed at an appropriate timing. In addition, in the method of executing the garbage collection process at the time of the partial table division described above, the partial table division occurs in a short time as the number of files to be added increases. As a result, the garbage collection process is also performed. It will occur in a short period of time and load balancing will not be possible.

  Accordingly, the technology according to the present application has been made in view of the above-described problems of the prior art, and the garbage collection execution device and the garbage that make it possible to distribute the resource load by distributing the occurrence of garbage collection It is an object to provide a collection execution method and a garbage collection execution program.

  In order to solve the above-described problems and achieve the object, the garbage collection execution apparatus according to the present application is distributed by a plurality of partial tables formed with a predetermined capacity in each of a plurality of servers included in a distributed key-value store. A determination unit that determines an execution time for each partial table so that garbage collection processing for records including managed key information and value information is distributed in time series for each partial table in each of the servers; And an execution unit that executes a garbage collection process on the target partial table at the execution time determined by the determination unit.

  The garbage collection execution apparatus according to the present application makes it possible to distribute the resource load by distributing the occurrence of garbage collection.

FIG. 1 is a diagram illustrating an example of a configuration of a distributed key-value store using the distributed file system according to the first embodiment. FIG. 2 is a diagram for explaining record management by the slave server according to the first embodiment. FIG. 3 is a diagram for explaining garbage collection processing by the slave server according to the first embodiment. FIG. 4 is a diagram for explaining a problem related to the prior art. FIG. 5 is a diagram illustrating an example of the configuration of the slave server according to the first embodiment. FIG. 6 is a diagram for explaining an example of area division and garbage collection processing executed by the slave server according to the first embodiment. FIG. 7 is a flowchart illustrating a processing procedure performed by the garbage collection determination unit according to the first embodiment. FIG. 8 is a flowchart illustrating a processing procedure performed by the garbage collection execution unit according to the first embodiment. FIG. 9 is a diagram for explaining prediction of deletion capacity by the garbage collection determination unit according to the second embodiment. FIG. 10 is a diagram for explaining an example of the garbage collection process executed by the slave server according to the second embodiment. FIG. 11 is a flowchart illustrating a processing procedure by the garbage collection determination unit according to the second embodiment. FIG. 12 is a diagram illustrating a computer that executes a garbage collection execution program according to the third embodiment.

  Exemplary embodiments of a garbage collection execution device, a garbage collection execution method, and a garbage collection execution program according to the present application will be described below in detail with reference to the accompanying drawings. In the following description, a case where a slave server included in the distributed key-value store functions as a garbage collection execution device according to the present application will be described as an example. In addition, the garbage collection execution device, the garbage collection execution method, and the garbage collection execution program according to the present application are not limited to the following embodiments.

(First embodiment)
First, an example of a distributed key-value store using a distributed file system will be described. FIG. 1 is a diagram illustrating an example of a configuration of a distributed key-value store using the distributed file system according to the first embodiment. As shown in FIG. 1, the distributed key-value store using the distributed file system according to the first embodiment includes slave servers 100A to 100C, a master server 200, and a network switch 300, and the slave server 100A. To 100C and the master server 200 are connected to each other via the network switch 300. In FIG. 1, slave servers 100 </ b> A to 100 </ b> C are illustrated, but actually, a plurality of slave servers are further connected to the network switch 300.

  The distributed key-value store stores, for example, a record including a pair of “key (key information)” and “value (value information)” on the slave server as data of a user accessed via the network switch 300. It is managed by a partial table (hereinafter referred to as an area) formed in For example, as shown in FIG. 1, the distributed key-value store divides a table of records sorted by key and manages them by slave servers 100A to 100C. Here, the division of the table may be executed based on the number of entries in the record, or may be executed based on the data size.

  The network switch 300 switches a slave server to be accessed by a client device (not shown) operated by a user. The master server 200 makes various instruction requests to the slave servers 100A to 100C. For example, the master server 200 determines which area in the table is assigned to which slave server, and issues an instruction request to each slave server. For example, as shown in FIG. 1, the master server 200 arranges an area from Key1 to Key99 and an area from Key300 to Key399 on the slave server 100A.

  The slave servers 100A to 100C manage the areas arranged by the master server 200, and execute processing for records such as search and update for the areas managed by the slave servers according to instructions from the user. For example, the slave server 100A receives a search instruction request for the areas from Key1 to Key99 from the user, and executes the received instruction request. In the distributed file system, replication of each record is stored in a plurality of servers.

  Here, in the distributed key-value store, even when an operation such as deletion or update is executed on a record once added, the record is not deleted or updated. For example, when the user performs a delete operation on (Key 302, Value 302) managed by the slave server 100A, the slave server 100A manages D (Key 302, Value 302) as shown in FIG. In other words, the distributed Key-Value store does not update the already stored (Key, Value) pair but updates (deletes) a new (Key ′) that means update or deletion when a record update or deletion is requested by the user. , Value '). Hereinafter, D (Key, Value) means that (Key, Value) is deleted. U (Key, Value ') means that (Key, Value) corresponding to Key is updated to (Key, Value'). Similar to the deletion, the stored data is not updated, but a new pair U (Key, Value ') is added internally. When there is a search, the slave servers 100A to 100C return only the necessary (Key, Value) from the (Key, Value) group indicating addition, update, and deletion related to the same Key to the user.

  Next, details of record management for the slave server will be described with reference to FIG. Since the slave servers included in the distributed key-value store perform the same management, the slave server 100A shown in FIG. 1 will be described as an example in FIG. FIG. 2 is a diagram for explaining record management by the slave server 100A according to the first embodiment.

  For example, the slave server 100A has one WAL log (log advance write log file) and a plurality of areas (area 1 and area 2) as shown in FIG. As shown in FIG. 2, area 1 and area 2 have a buffer on the memory and a plurality of sorted KeyValue files. Area 1 and area 2 are divided when a predetermined capacity is set and the capacity of the stored record exceeds the predetermined capacity. For example, when the capacity of a record stored in area 1 exceeds a predetermined capacity, area 1 is divided, and the record stored in area 1 is managed in the divided area.

  Here, as shown in (1) of FIG. 2, when receiving a request for adding (newKey, newValue) from the user, the slave server 100A first sets (NewKey, newValue) to WAL as shown in (2). Write to log. This writing to the WAL log is realized by appending to the end of the file, and when the writing is successful, the record persistence is guaranteed.

  Thereafter, as shown in (3) of FIG. 2, the slave server 100A writes (newKey, newValue) to the memory buffer in the area 1. Further, when the capacity in the memory buffer increases, as shown in (4), the slave server 100A sorts the records in the memory buffer (newKey, newValue) by Key, and then sorts them on the distributed file system. Write to KeyValue file and clear buffer in memory. In FIG. 2, the sorted KeyValue file is written in another file, but the embodiment is not limited to this. For example, the file may be additionally written in one file.

  As described above, the slave server 100A makes a record permanent by writing a temporary (Key, Value) group on the memory to the sorted KeyValue file on the distributed file system. When receiving a record search request from the user, the slave server 100A reads the buffer in memory and the sorted KeyValue file, and returns the search result to the user.

  The above-described record addition is executed in the same manner for records indicating deletion or update. For example, the slave server 100A writes D (Key10, Value10) to the memory buffer in the area 2, as shown in FIG. Then, the slave server 100A sorts the buffer D (Key10, Value10) in the memory by Key1, and then stores it in the sorted KeyValue file on the distributed file system.

  This is because (Key10, Value10) to be deleted has already been made permanent as a sorted KeyValue file, and it is not known where (Key10, Value10) to be deleted exists unless all the sorted KeyValue files are read. It is. This eliminates the need to search and update the target (Key10, Value10) when writing to the sorted KeyValue file, thereby improving the write throughput.

  Next, garbage collection processing by the slave server will be described with reference to FIG. Since slave servers included in the distributed key-value store perform similar garbage collection, FIG. 3 will be described by taking the slave server 100A shown in FIG. 1 as an example. FIG. 3 is a diagram for explaining garbage collection processing by the slave server 100A according to the first embodiment.

  For example, as shown in (1) of FIG. 3, the slave server 100A reads all the records in the area 1 (the (Key, Value) and the sorted KeyValue file in the buffer on the memory) and records to be deleted. (For example, D (Key, Value) and the target (Key, Value)) are deleted and a new file is created. Then, as shown in (2), the slave server 100A deletes all existing files and (Key, Value) on the memory.

  As described above, the garbage collection process reads all existing records for each area, deletes records to be deleted, and writes all the records after deletion. Therefore, the garbage collection process consumes a large amount of resources such as disk I / O, CPU, and network. Therefore, various methods related to the execution timing of the garbage collection processing have been disclosed, but sufficient load distribution cannot be executed by any of the methods. Hereinafter, problems related to the prior art will be described. FIG. 4 is a diagram for explaining a problem related to the prior art.

  In FIG. 4, (A) shows a method for setting an execution interval of garbage collection processing, (B) shows a method for providing a user with an execution command for garbage collection processing, and (C) shows garbage collection at the time of area division. A method for performing collection will be described. Further, in the figure indicated by the upper rectangle and the arrow in FIG. 4, the rectangle indicates the area, the length of the rectangle indicates the division size, and the area is divided as the record is added. In the middle diagram of FIG. 4, the horizontal axis indicates time (t), and the vertical axis indicates the number of occurrences of garbage collection (GC).

  First, in the method of setting the garbage collection processing execution interval, since only one garbage collection processing execution interval can be set for all areas, as shown in FIG. The execution timing of each garbage collection process overlaps, and load distribution on the time axis cannot be performed. That is, in the method of setting the execution interval of the garbage collection process, there is nothing but a state in which the load on the disk I / O, the network, and the CPU is high at a constant interval. In fact, at the initial stage of the distributed file system, the high load state when the garbage collection process is executed is completed in about one hour, but the load increases as the amount of managed data (record amount) increases. Has also been pointed out. This is because the area increases as the amount of data increases, and the number of garbage collection processes increases.

  In the method of providing the user with a garbage collection process execution command, as shown in FIG. 4B, when the garbage collection process is executed (GC execution), the same number of garbage collection processes as the number of areas are executed. Load balancing is not possible. Furthermore, it is very difficult to provide a user with a garbage collection process execution command unless the user fully understands the inside of the system. For example, if you do not fully understand the inside of the system, garbage collection processing will be executed too much, increasing the load on the CPU and disk I / O, and adding (Key, Value) that should be provided to the user In some cases, the throughput and response time of functions such as update, deletion, and search deteriorate. On the other hand, there are cases where the garbage collection process is executed too little and the disk occupancy on the distributed file system does not decrease, causing a problem such as disk full.

  Further, in the method of executing the garbage collection process at the time of area division (before division), writing to the area of the record is performed at random, and the accumulation state of the record in each area is also equalized. As shown in (C), the timing of division occurring in each area is almost simultaneous (concentrates within a certain period). As a result, in the method of executing the garbage collection process at the time of area division, the load on the disk I / O, the network, and the CPU is high when the areas are divided almost simultaneously. In other words, as the amount of managed data increases, the number of garbage collection processes occurring in a concentrated manner in a short time increases, and the resource load also increases. In FIG. 4C, the time during which the area is divided is T → T2 → T4. Once the division is performed, the number of areas is doubled at the same speed (Key, This is because it takes twice as long to divide the area when (Value) is input.

  As described above, in any of the conventional techniques, the number of garbage collection processes that occur intensively in a short time is large. Therefore, the slave server according to the present application suppresses the resource load by distributing the occurrence of garbage collection. FIG. 5 is a diagram illustrating an example of the configuration of the slave server 100A according to the first embodiment. Since the slave servers included in the distributed file system have the same configuration, FIG. 1 will be described by taking the slave server 100A as an example.

  As illustrated in FIG. 5, the slave server 100A includes a communication control I / F unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150. The communication control I / F unit 110 controls communication related to various types of information exchanged between the user terminal device operated by the user and the control unit 150. For example, the communication control I / F unit 110 controls communication related to search / addition / update / deletion of records. Further, the communication control I / F unit 110 controls communication related to various information exchanged between the master server 200 and the control unit 150. The communication control I / F unit 110 controls the exchange of various information between the input unit 120 and the display unit 130 and the control unit 150.

  The input unit 120 is, for example, a keyboard or a mouse, and accepts various information input processes by the administrator of the distributed key-value store. The display unit 130 is, for example, a display, and displays and outputs the processing result to the administrator of the distributed key-value store.

  As shown in FIG. 5, the storage unit 140 includes a data storage area 141 and a sorted file storage area 142. The storage unit 140 is, for example, a storage device such as a hard disk or an optical disk, or a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, and stores various programs executed by the slave server 100A. To do.

  The data storage area 141 is a storage area for storing records (Key, Value), and is, for example, a memory buffer shown in FIG. The sorted file storage area 142 is a storage area for storing records sorted by key, and is, for example, a sorted key value file shown in FIG. These sorted KeyValue files correspond to the distributed file system, and replicas of these files are managed by a plurality of servers, thereby maintaining reliability.

  As illustrated in FIG. 5, the control unit 150 includes a garbage collection determination unit 151 and a garbage collection execution unit 152. The control unit 150 is, for example, an electronic circuit such as a CPU or MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), and executes overall control of the slave server 100A. To do. Although not shown, the control unit 150 includes a functional unit that executes processing such as search, addition, deletion, and update of records, a functional unit that sorts records, and a functional unit that divides areas.

  The garbage collection determination unit 151 performs a garbage collection process on records including Key and Value distributed and managed by a plurality of areas formed with a predetermined capacity in each of a plurality of servers included in the distributed Key-Value store. The execution time for each area is determined so that each area is distributed in time series. Specifically, the garbage collection determination unit 151 divides an area so that a predetermined capacity changes within a predetermined range when a garbage collection process is executed when the area is divided.

  For example, each time a record is added by the user, the garbage collection determination unit 151 includes an area base size that is a threshold for dividing an area, a variable size that randomly changes the area base size, and the current area. Get the size and. Then, the garbage collection determination unit 151 generates a random value (R) that is equal to or smaller than the acquired variation size and is equal to or greater than zero.

  Here, the garbage collection determination unit 151 determines whether or not the value obtained by adding the generated R to the area base size value exceeds the current area size value. Schedule processing. That is, the garbage collection determination unit 151 determines to divide an area when the current area size value exceeds a value obtained by adding a random value to a threshold value for dividing the area, and performs garbage collection processing. A garbage collection execution unit 152, which will be described later, is notified to execute.

  Thereby, the area size to be divided can be changed at random, and the occurrence of area division can be distributed over time. As a result, the execution of the garbage collection process can also be distributed over time, and the resource load can be distributed. The variable size can be arbitrarily set by the administrator of the distributed file system or the user.

  The garbage collection execution unit 152 executes the garbage collection process for the target area at the execution time determined by the garbage collection determination unit 151. Specifically, the garbage collection execution unit 152 executes a garbage collection process when the area is divided. That is, the garbage collection execution unit 152 executes the garbage collection process when each area whose area size is randomly changed by the garbage collection determination unit 151 is divided.

  FIG. 6 is a diagram for explaining an example of area division and garbage collection processing executed by the slave server 100A according to the first embodiment. Here, in the figure indicated by the upper rectangle and the arrow in FIG. 6, the rectangle indicates the area, the rectangle length indicates the division size, and the area is divided as the record is added. In the lower diagram of FIG. 6, the horizontal axis indicates time (t), and the vertical axis indicates the number of occurrences of garbage collection (GC).

  For example, in the area division executed by the slave server 100A, as shown in the upper part of FIG. As described above, since the record accumulation status in each area is leveled, the timing at which the area is divided differs for each area. Therefore, when the garbage collection process is executed at the time of area division, the garbage collection process is distributed and executed in time series as shown in the lower part of FIG. In other words, the slave server 100A can equalize the number of garbage collection processes that occur at regular time intervals (suppress the occurrence of intensive garbage collection processes within a specific time interval).

  Next, a processing procedure performed by the slave server 100A according to the first embodiment will be described with reference to FIGS. FIG. 7 is a flowchart illustrating a processing procedure performed by the garbage collection determination unit 151 according to the first embodiment. When a record is added by the user, the garbage collection determination unit 151 according to the first embodiment acquires an area base size, a variable size, and a current area size as illustrated in FIG. 7 (step S101).

  Then, the garbage collection determination unit 151 generates a random value (R) that is not less than 0 and not more than the variation size (step S102), and determines whether or not the area base size + R is smaller than the current area size (step S102). S103). If the area base size + R is smaller than the current area size (Yes at Step S103), the garbage collection determination unit 151 schedules the garbage collection process (Step S104) and ends the process.

  On the other hand, if the area base size + R is not smaller than the current area size (No at Step S103), the garbage collection determination unit 151 ends the process without scheduling the garbage collection process.

  FIG. 8 is a flowchart illustrating a processing procedure performed by the garbage collection execution unit 152 according to the first embodiment. FIG. 8 shows a process after the garbage collection determining unit 151 schedules the garbage collection process.

  When the garbage collection determination unit 151 schedules the garbage collection process, the garbage collection execution unit 152 according to the first embodiment acquires a scheduled garbage collection task as illustrated in FIG. 8 (step S201). Specifically, the garbage collection execution unit 152 acquires information on an area to be subjected to the garbage collection process.

  Then, the garbage collection execution unit 152 reflects the update / deletion from the existing sorted KeyValue file group in the acquired area, generates a new sorted KeyValue file (step S202), and ends the process.

[Effect of the first embodiment]
As described above, according to the first embodiment, the garbage collection determination unit 151 is distributed and managed by a plurality of areas formed with a predetermined capacity in each of a plurality of servers included in the distributed key-value store. Further, the execution timing for each area is determined so that the garbage collection processing for the records including Key and Value is executed in a time-series manner in each server. Then, the garbage collection execution unit 152 executes a garbage collection process for the target area at the execution time determined by the garbage collection determination unit 151. Therefore, the slave server 100A according to the first embodiment can distribute the execution of the garbage collection process in time series, and can distribute the resource load.

  In addition, according to the first embodiment, the garbage collection determination unit 151 divides an area so that a predetermined capacity changes within a predetermined range when the garbage collection process is executed when the area is divided. And the garbage collection execution part 152 performs a garbage collection process at the time of area division. Therefore, the slave server 100A according to the first embodiment can easily perform garbage collection processing in time series using existing technology, and can easily execute resource load distribution. To do.

(Second Embodiment)
In the first embodiment described above, the case has been described in which the area division timing is distributed by randomly changing the area size, and the garbage collection is executed when the area is divided. In the second embodiment, a case will be described in which the capacity to be reduced by the garbage collection process is predicted, and the garbage collection process is executed according to the prediction result. Note that the slave server according to the second embodiment differs from the slave server 100A according to the first embodiment only in the processing contents by the garbage collection determination unit 151 and the garbage collection execution unit 152. Hereinafter, these will be mainly described.

  The garbage collection determination unit 151 according to the second embodiment predicts a reduction capacity when the garbage collection process is executed for each area at a predetermined time interval, and determines an area where the predicted reduction capacity is the maximum for the garbage collection process. Decide on the target. Specifically, the garbage collection determination unit 151 determines that the record is a deletion target record that can be deleted by the garbage collection process when a predetermined period has elapsed since the record was registered in the area, and is included in the area. The total capacity of records to be deleted is predicted as the deletion capacity.

  Here, the slave server 100A according to the second embodiment uses a record obtained by adding Timestamp indicating the time when (Key, Value) is registered to (Key, Value). That is, the slave server 100A according to the second embodiment stores a (Key, Timestamp, Value) tuple in the storage unit 140. Note that Timestamp may be set by the user when (Key, Value) is added, or may be registered by the control unit 150. When registered by the control unit 150, the control unit 150 further includes a function unit that registers the time when (Key, Value) is added.

  For example, the garbage collection determination unit 151 predicts the deletion capacity by the garbage collection process for each area by the following equation (1) every time a predetermined time elapses. Note that “tpds: Total Prediction of Deleting Size” in Equation (1) is a function indicating the total capacity that is predicted to be deleted. In addition, “Area [i]” in Expression (1) indicates each area in the slave server, and is (1 ≦ i ≦ N: a positive integer). In addition, “Area [i] .SortedKVF [j]” in Expression (1) indicates the sorted KeyValue file stored in each area, and is (1 ≦ j ≦ M: positive integer). In addition, “pds” in Expression (1) is a function indicating the capacity that is predicted to be deleted.

  That is, the garbage collection determination unit 151 predicts the deletion capacity by calculating the total value of the sorted KeyValue files that can be deleted for each area, as shown in Expression (1).

  Here, “pds” included in the equation (1) is defined by the following equation (2). Note that “SortedKVF [j] .size” in Expression (2) indicates the record size. Further, “SortedKVF [j] .maxts” in Expression (2) indicates the latest record in the area, and “SortedKVF [j] .mints” indicates the oldest record in the area. This is because, in principle, each registered record is a set having (Key, Timestamp, Value) as an element, and in principle, the record having the maximum Timestamp (latest) and the one having the minimum Timestamp (oldest) Can be used. In addition, “CT: Current Time” in Equation (2) indicates the current time, and “PV: Period of Validity” indicates the validity period of the record.

  That is, the garbage collection determining unit 151 uses the record size as the deletion capacity when the record has expired (SortedKVF [j] .maxts ≦ CT-PV), as shown in Expression (2). If the record has not expired (CT-PV ≦ SortedKVF [j] .mints), the deletion capacity is set to “0”. Here, the garbage collection determination unit 151 linearly predicts the capacity that has expired. That is, the garbage collection determination unit 151 predicts the deletion capacity according to the assumption that the amount that can be deleted increases linearly with the passage of time.

  FIG. 9 is a diagram for explaining prediction of deletion capacity by the garbage collection determination unit 151 according to the second embodiment. Here, in FIG. 9A, the horizontal axis indicates time (t). In FIG. 9A, line segments respectively indicating all “SortedKVF [j] (1 ≦ j ≦ M)” included in one area are shown on the horizontal axis. That is, in the line segment shown in FIG. 9A, the left end indicates “SortedKVF [j] .mints” and the right end indicates “SortedKVF [j] .maxts”, as shown in FIG. 9B. The length indicates “SortedKVF [j] .size”.

  For example, the garbage collection determining unit 151 does not delete the sorted KeyValue file in the effective period (between “CT” and “CT-PV”) in FIG. Is “0”. On the other hand, for the sorted KeyValue file in the invalid period, the deletion capacity is set to the size of the sorted KeyValue file. Then, the garbage collection determination unit 151 calculates the deletion capacity for the sorted KeyValue file for both the invalid period and the valid period using the second expression of Expression (2). That is, as shown in equation (2), in the case of (SortedKVF [j] .mints ≦ CT-PV ≦ SortedKVF [j] .maxts), the garbage collection determining unit 151 calculates the ratio of expired. Thus, the size of the ratio in the entire record size is calculated as the deletion capacity.

  The garbage collection determination unit 151 calculates the deletion capacity of each area using Expression (1) and Expression (2), and extracts an area that is a target of garbage collection processing according to Expression (3) below. Note that “gct: garbage collection target” in Expression (3) is a function that returns the area with the largest deletion amount when the garbage collection process is executed at the present time. In addition, “Areas” in Expression (3) represents a set represented by {Area [j] | 1 ≦ j ≦ M}.

  That is, the garbage collection determination unit 151 returns the area with the largest deletion amount when the garbage collection is currently executed by the function “gct” having “Area” as an argument. Here, the garbage collection determination unit 151 performs control so as not to execute the garbage collection process when the deletion capacity does not exceed a certain amount, according to the following equation (4). Note that “gtc_over_mds” in Expression (4) is a function indicating “0” indicating that the garbage collection process is not performed when the deletion amount for the returned area does not satisfy a certain amount. Further, “MDS: Minimum Deleting Size” in Expression (4) indicates a threshold value for determining whether or not to execute the garbage collection process.

  That is, the garbage collection determination unit 151 notifies the garbage collection execution unit 152 of information related to the area when the deletion capacity of the area when the garbage collection process is currently executed exceeds MDS. On the other hand, if the deletion capacity of the area when the garbage collection process is currently executed is equal to or less than MDS, the garbage collection determination unit 151 gives “0” indicating that the garbage collection process is not executed to the garbage collection execution unit 152. Notice.

  As described above, the garbage collection determination unit 151 calculates the deletion capacity based on whether or not the expiration date has expired for each area, and determines the area where the calculated deletion capacity is the maximum as the target of the garbage collection process. Then, the garbage collection determination unit 151 executes the garbage collection process on the condition that the deletion capacity of the target area exceeds a certain amount (MDS). The expiration date and MDS described above can be arbitrarily set by the administrator of the distributed file system or the user. For example, “30 days” may be set as the expiration date for the Web information or the like.

  The garbage collection execution unit 152 according to the second embodiment executes a garbage collection process for the area determined by the garbage collection determination unit 151. Specifically, the garbage collection execution unit 152 executes the garbage collection process for the area when the deletion capacity in the area determined as the object of the garbage collection process by the garbage collection determination unit 151 exceeds MDS. That is, when the garbage collection execution unit 152 is notified of area information from the garbage collection determination unit 151, the garbage collection execution unit 152 executes a garbage collection process for the notified area.

  FIG. 10 is a diagram for explaining an example of the garbage collection process executed by the slave server according to the second embodiment. Here, in the figure indicated by the upper rectangle and the arrow in FIG. 10, the rectangle indicates the area, the rectangle length indicates the division size, and the area is divided as the record is added. In the lower diagram of FIG. 10, the horizontal axis indicates time (t), and the vertical axis indicates the number of occurrences of garbage collection (GC). Further, the black triangle shown in the lower part of FIG. 10 indicates that the garbage collection process has been executed, and the dotted triangle indicates that the garbage collection process has not been executed.

  For example, in the garbage collection process executed by the slave server 100A according to the second embodiment, as shown in the lower part of FIG. 10, the garbage collection process for one area is executed in a time-series distributed manner. That is, the slave server 100A can suppress the occurrence of intensive garbage collection processing within a specific time interval.

  Next, a processing procedure by the slave server 100A according to the second embodiment will be described with reference to FIG. The garbage collection process by the garbage collection execution unit 152 according to the second embodiment is the same as that of the first embodiment (see FIG. 8). FIG. 11 is a flowchart illustrating a procedure of processing performed by the garbage collection determination unit 151 according to the second embodiment. As shown in FIG. 11, the garbage collection determination unit 151 according to the second embodiment calculates an estimated deletion data amount for each area when a predetermined time has passed (Yes in Step S <b> 301) (Step S <b> 302).

  Then, the garbage collection determination unit 151 extracts an area with the maximum deletion capacity (step S303), and determines whether or not the deletion capacity of the extracted area exceeds the MDS (step S304). Here, when the deletion capacity of the extracted area exceeds MDS (Yes at Step S304), the garbage collection determination unit 151 schedules garbage collection processing for the corresponding area (Step S305), and ends the processing.

  On the other hand, if the deleted capacity of the extracted area does not exceed MDS (No at step S304), the garbage collection determination unit 151 ends the process without scheduling the garbage collection process. Note that the garbage collection determination unit 151 is in a standby state until a predetermined time has elapsed (No in step S301).

[Effects of Second Embodiment]
As described above, according to the second embodiment, the garbage collection determination unit 151 predicts the reduction capacity when the garbage collection processing is executed for each area at a predetermined time interval, and the area where the reduction capacity is maximized. Is determined as a target of garbage collection processing. Then, the garbage collection execution unit 152 executes a garbage collection process for the area determined by the garbage collection determination unit 151. Therefore, the slave server 100A according to the second embodiment can control the trade-off between the disk occupation amount and the resource load, and can distribute the resource load.

  Further, according to the second embodiment, the garbage collection determination unit 151 includes a deletion target record that can be deleted by the garbage collection process when a predetermined period has elapsed since the record was registered in the area. Judgment is made, and the total capacity of the records to be deleted included in the area is predicted as the deletion capacity. Therefore, the slave server 100A according to the second embodiment makes it possible to execute the garbage collection process so as to reliably delete unnecessary records.

  In addition, according to the second embodiment, the garbage collection execution unit 152, when the deletion capacity in the area determined as the garbage collection processing target by the garbage collection determination unit 151 exceeds a predetermined threshold, Execute garbage collection processing. Therefore, the slave server 100A according to the second embodiment makes it possible to execute an effective garbage collection process.

(Third embodiment)
Although the first embodiment and the second embodiment have been described so far, the embodiment according to the present application is not limited to the first embodiment and the second embodiment. That is, these embodiments can be executed in various other forms, and various omissions, replacements, and changes can be made.

  For example, the specific form of distribution / integration of each device (for example, the form shown in FIG. 5) is not limited to the one shown in the figure, and all or a part thereof can be changed in arbitrary units according to various loads and usage conditions. Functionally or physically distributed and integrated. For example, the garbage collection determination unit 151 and the garbage collection execution unit 152 may be integrated as one processing unit, while the garbage collection determination unit 151 includes a prediction unit that predicts a deletion capacity, and a deletion target area. It may be distributed to the extraction unit that extracts.

  The control unit 150 may be connected as an external device of the slave server 100A via a network, or another device has a garbage collection determination unit 151 and a garbage collection execution unit 152, and is connected to the network. Thus, the functions of the slave server 100A described above may be realized by cooperating.

  The slave server 100A described in the above-described embodiment can also be realized by executing a program prepared in advance by a computer. Therefore, an example of a computer that executes a garbage collection execution program that realizes the same function as that of the slave server 100A illustrated in FIG. 5 will be described below.

  FIG. 12 is a diagram illustrating a computer 1000 that executes a garbage collection execution program according to the third embodiment. As shown in FIG. 12, the computer 1000 includes, for example, a memory 1010, a CPU (Central Processing Unit) 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network. Interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100, for example. For example, a mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050. For example, a display 1130 is connected to the video adapter 1060.

  Here, as shown in FIG. 12, the hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. The garbage collection execution program according to the third embodiment is stored in, for example, the hard disk drive 1090 as a program module in which a command executed by the computer 1000 is described. Specifically, a garbage collection determination step for executing information processing similar to the garbage collection determination unit 151 described in the above embodiment and a garbage collection execution step for executing information processing similar to the garbage collection execution unit 152 are described. The programmed program module is stored in the hard disk drive 1090.

  Further, like the data stored in the database described in the above embodiment, data used for information processing by the performance analysis program is stored as program data, for example, in the hard disk drive 1090. Then, the CPU 1020 reads program modules and program data stored in the hard disk drive 1090 to the RAM 1012 as necessary, and executes a garbage collection determination step and a garbage collection execution step.

  Note that the program module and program data related to the performance analysis program are not limited to being stored in the hard disk drive 1090, but are stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. May be. Alternatively, a program module and program data related to the information transmission / reception program are stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and the CPU 1020 via the network interface 1070. May be read.

  These embodiments and modifications thereof are included in the invention disclosed in the claims and equivalents thereof as well as included in the technology disclosed in the present application.

100A, 100B, 100C Slave server 110 Communication control I / F unit 120 Input unit 130 Display unit 140 Storage unit 141 Data storage area 142 Sorted file storage area 150 Control unit 151 Garbage collection determination unit 152 Garbage collection execution unit 200 Master server 300 Network switch

Claims (7)

Garbage collection process for a plurality of server record containing their respective on thus distributed key information and value information is managed included in the distributed Key-Value store has each server respectively, changing the capacitance randomly for execution upon division of each of the plurality of parts tables, a determination unit that determines an execution timing of the garbage collection process for each of the partial table,
An execution unit that executes a garbage collection process on a target partial table at an execution time determined by the determination unit;
A garbage collection execution device characterized by comprising: The determining unit divides the partial table so that the capacity after the division changes within a predetermined range when the garbage collection process is executed when the partial table is divided.
The garbage collection execution apparatus according to claim 1, wherein the execution unit executes a garbage collection process when the partial table is divided. The determination unit predicts a reduction capacity when the garbage collection process is executed for each partial table at a predetermined time interval, determines a partial table having the maximum reduction capacity as a target of the garbage collection process,
The garbage collection execution apparatus according to claim 1, wherein the execution unit executes a garbage collection process on the partial table determined by the determination unit.   The determination unit determines that the record is a deletion target record that can be deleted by the garbage collection process when a predetermined period has elapsed from the time when the record is registered in the partial table, and is included in the partial table. The garbage collection execution apparatus according to claim 3, wherein a total capacity of the deletion target records is predicted as the deletion capacity.   The execution unit executes a garbage collection process on the partial table when the deletion capacity in the partial table determined as the garbage collection process target by the determination unit exceeds a predetermined threshold. The garbage collection execution apparatus according to claim 3 or 4. A garbage collection execution method executed by a device included in a distributed Key-Value store,
Garbage collection processing for the record including a plurality of server key information and value information managed in their respective Accordingly dispersed contained in the dispersion Key-Value store has each server respectively, to change the capacity at random A determination step of determining the execution time of the garbage collection process for each partial table , so that it is executed at the time of division for each of the plurality of partial tables;
An execution step of performing a garbage collection process on the target partial table at the execution time determined by the determination step;
The garbage collection execution method characterized by including. A garbage collection execution program executed by a device included in a distributed Key-Value store,
Garbage collection processing for the record including a plurality of server key information and value information managed in their respective Accordingly dispersed contained in the dispersion Key-Value store has each server respectively, to change the capacity at random a plurality of as may be performed in time division for each partial table, determining step of determining the execution timing of the garbage collection process for each of the parts tables,
An execution step of executing a garbage collection process on the target partial table at the execution time determined by the determination step;
Is executed by the apparatus. A garbage collection execution program characterized in that

Images