KR101938501B1 - Dynamic load-based merging - Google Patents

Abstract

One embodiment of the present invention provides a system that facilitates dynamic load-based merging. During operation, the system identifies incremental data in the memory of the first computing device. The incremental data includes an update to the reference data at the storage device of the second computing device. When the size of the incremental data exceeds the threshold value, the system selects a first server group from a plurality of server groups. The second computing device belongs to this first server group. The system then migrates data access requests from the first server group to the other server groups in the plurality of server groups and merges the incremental data from the memory of the first computing device into the storage device at the second computing device. The merge includes batch sequential disk writes to the storage device of the second computing device.

Description

{DYNAMIC LOAD-BASED MERGING}

Related application

35 U.S.C. 119, the present application claims the benefit of and priority of Chinese patent application No. 201410429616.0, filed on August 27, 2014.

Field

This application relates to the field of computer technologies and, more particularly, to methods and systems for data management.

With advances in computer and network technologies, various operations performed by users from different applications lead to widespread storage allocation. For example, multiple users can simultaneously shop on an e-commerce website through different applications, such as mobile applications running on different platforms as well as web-interfaces running on different browsers in different operating systems have. Corresponding shopping records are stored (e. G., In a database) in the storage devices of the system facilitating e-commerce services.

However, different applications that access the system at the same time cause an increasing number of user activities, resulting in an increasing amount of user records being stored in the system. A large number of these records can be accessed by users while they are being updated. Thus, the stored data in the system needs to be continuously managed based on the users' access and update frequencies. Management of such stored data includes allocation of write operations to the data store in such a manner as to reduce the impact on the user ' s data access.

Although a number of methods are available for data management in a system, some problems still remain in efficient storage management for concurrent applications.

One embodiment of the present invention provides a system that facilitates dynamic load-based merging. During operation, the system identifies incremental data in the memory of the first computing device. The incremental data includes an update to the reference data at the storage device of the second computing device. When the size of the incremental data exceeds the threshold value, the system selects a first server group from a plurality of server groups. The second computing device belongs to this first server group. The system then migrates data access requests from the first server group to the other server groups in the plurality of server groups and merges the incremental data from the memory of the first computing device into the storage device at the second computing device. The merge includes batch sequential disk writes to the storage device of the second computing device.

In a variation of this embodiment, the reference data is divided into a plurality of tablets, and the tablet is a non-overlapping segment of the reference data.

In a further variation, each server group of the plurality of server groups includes a copy of each tablet of reference data.

In a variation of this embodiment, selecting a first server group from a plurality of server groups comprises identifying a first server group from a sequence of server groups.

In a further variation, the system determines if a merge of incremental data has been performed for each server group and selects a group of servers for which no merge of incremental data has been performed.

In a variation of this embodiment, the system maintains a traffic control table containing the data access request traffic ratios of each of the plurality of server groups, and the traffic ratio for the first server group is greater than the threshold Value has been reached. If so, the system determines if the first server group is ready for merging.

In a variation of this embodiment, the system determines if the merge is complete for the first server group. If so, the system restores data access requests from the other server groups back to the first server group.

In a variation of this embodiment, the system determines whether the first server group is needed to serve data access requests during the merge of the incremental data, and whether the resource consumption by the computing devices in the first server group satisfies the condition .

In a further variation, the system determines a performance level associated with data access requests served from the first server group. If the performance level is below the threshold, the system pauses the merge of the incremental data.

In a further variation, the system releases the resources occupied by the merge of the incremental data to serve the data access requests.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate several exemplary embodiments of the present application and serve to illustrate the present application, along with the specification not being construed as a limitation on the present application . Among the drawings:
FIG. 1A illustrates exemplary data management based on incremental and reference data, in accordance with an embodiment of the present application.
1B illustrates an exemplary data management system based on update and storage servers, in accordance with one embodiment of the present application.
Figure 2 illustrates exemplary tablets of reference data, according to one embodiment of the present application.
FIGURE 3A illustrates a flow diagram illustrating a data management system for initiating a merge process, in accordance with one embodiment of the present application.
Figure 3B shows a flow diagram illustrating an off-peak merging process of a data management system, in accordance with one embodiment of the present application.
3C depicts a flow diagram illustrating a dynamic-controlled merging process of a data management system, in accordance with one embodiment of the present application.
4 is a schematic structural diagram of an exemplary data management system, in accordance with one embodiment of the present application.
In the drawings, like reference numerals refer to the same drawing elements.

Embodiments of the present invention solve the problem of efficiently managing data access by diverting data updates from the storage servers being accessed. If data access is unavoidable on their storage servers, the data from the storage server is updated when resources on the storage servers are available.

With existing technologies, data is typically stored in a Log-Structured Merge-Tree. In the LSM tree, data is typically partitioned into multiple levels at the system ' s memory and storage disk. If the data is updated (e.g., due to user behavior), the data is written into the memory of the system. However, the data in the memory is volatile. Thus, the system writes the updated data in memory to the disk file through the merge process. This merging process is usually performed in parallel with data access (i.e., data reading). The merge process is batch sequential disk writes. However, this merging process typically consumes a large amount of input / output (I / O) resources and directly affects the throughput and response time of normal operations of the storage system (e.g., online services). Online service is a service that the system maintains operation. For example, the availability and accessibility of the user's purchase records.

To solve this problem, embodiments of the present invention provide a system for a dynamic load-based merge process by separating data access and update operations and automatically bypassing data access traffic from currently being updated storage servers . The system divides the data into two sets of data, i.e., reference data for each data update (e.g., write operations) and data access (e.g., read operations) and incremental data. The incremental data resides in the memory of the update server, and the update operations are performed on the incremental data. On the other hand, the reference data is stored in the disks of the storage servers. The reference data typically has multiple copies at multiple storage servers to facilitate high availability. When the amount of data written into the memory of the update server reaches the threshold, the merge process is triggered. The incremental data in the memory is merged into the reference data on the disk of the corresponding storage server.

Moreover, to ensure separation of data access from the merge process, the system maintains at least two groups of servers of storage servers, each set holding the entire reference data. When the merge process is performed for the first group, the data access requests are served in the second group. Upon completion of the merge process for the first group, the data access requests are bypassed to the first group and the merge process is performed for the second group. As a result, during the merge process, the performance of data access from storage servers is not impaired. In some circumstances, due to the high volume of data access traffic or timeout events, some data access requests may be served in a group that is currently being merged. Under these circumstances, to reduce the impact of the merge process, the system monitors the load of the storage servers and allows the merge process to continue only when the load is low. Otherwise, the system initiates an overload protection mode and pauses the merge process to restore the resources (e.g., network, processing, and I / O resources) allocated to the merge process.

FIG. 1A illustrates exemplary data management based on incremental and reference data, in accordance with an embodiment of the present application. In this example, the data management system 100 for dynamic load-based data merge is deployed to facilitate separation of data access and update operations. It is assumed that the data includes incremental data 122 residing in memory 112 of update server 102. In some embodiments, the incremental data 122 may be organized into a B-tree. On the other hand, the reference data 124 resides on the disks 114 and 116 of the storage servers 104 and 106, respectively. The reference data is persistent data stored on the disks and the incremental data is an update (e.g., modification, addition, deletion, etc.) to the reference data. Storage servers 104 and 106 may host non-redundant reference data blocks 126 and 128 of reference data 124, respectively. The data blocks 126 and 128 together represent the entire reference data 124.

The system 100 performs updates (e.g., write operations) to the reference data 124 on the memory 112 to generate the incremental data 122. As a result, during the update process, write operations are not applied directly to the storage servers 104 and 106. Thus, storage servers 104 and 106 can reduce the impact of write operations associated with data updates while facilitating data access to client devices. When the amount of data written into the memory 112 of the update server 102 reaches a threshold value, the system 100 triggers the merge process. The merge process is batch sequential disk writes to disks 114 and / or 116. Thus, the merging process may be performed on the reference data blocks 126 and / or 128 in the disks 114 and / or 116 of the storage servers 104 and / or 106, respectively, And merges. After the merge process, updates in incremental data 122 become part of reference data 124. The system 100 performs any subsequent updates on the memory 122.

In some embodiments, to ensure separation of data access from the merge process, the system 100 maintains at least two groups of storage servers, each group holding all of the reference data 124. 1B illustrates an exemplary data management system based on update and storage servers, in accordance with one embodiment of the present application. The system 100 may group the storage servers 104 and 106 into a server group 142. The system 100 may maintain another server group 144 that includes storage servers 134 and 136. [ Both groups 142 and 144 host all of the reference data 124. This allows for load balancing between groups and separates data access from the merge process. In some embodiments, groups 142 and 144 may be in separate locations separated by network 150 (e.g., local or wide area networks). The update server 102 may be coupled to the groups 142 and 144 via the network 150.

System 100 ensures separation and merging processes of data access between groups 142 and 144. The system 100 uses the groups 142 and 144 as control units for the merge process and performs the merge process at different times. In the preparation stage of the merge process, the system 100 selects a group for the merge process and migrates data access traffic (e.g., data access requests and / or corresponding data) to other groups. In this way, the merge process and online services are separate from each other. When the system 100 performs the merge process for the group 142, the system 100 diverts the data access traffic to the group 144. For example, if client device 160 sends a data access request (e.g., a request for a customer record), the data access request is served in group 144 (e.g., to provide). After the system 100 completes the merge process for the group 142, the system 100 migrates the data access traffic to the group 142 and initiates the merge process for the group 144. Subsequent requests for data access from client device 160 are served in group 142. This merging process may be referred to as an off-peak merging process. The off-peak merge process reduces the impact of the merge of incremental data on the performance of online services (e. G., The time it took the serving server to serve the data access request).

In some embodiments, system 100 switches between groups 142 and 144 based on a traffic control table that includes data access traffic ratios between groups 142 and 144. Requests for data access from client devices (e.g., client device 160) are distributed to groups 142 and 144 based on a ratio. System 100 migrates data access traffic from one group to another group gradually. In this way, traffic ratios continue to increase for one group and decrease for the other group. For example, at a given time, if the ratio is 50: 50, the system 100 forwards 50% of the data access requests to the group 142 and another 50% to the group 144. [ If a merge process is required, the system 100 begins to migrate data access traffic from one group to another based on the sequence (e.g., group 142 first, group 144 second) . As a result, the traffic ratio between groups 142 and 144 continues to change. When the traffic rate for a group becomes zero, the group is ready for the merge process. For example, when the traffic rate for the group 142 is zero, the system 100 is forwarding each data access request to the group 144. Thus, the group 142 is ready for the merge process.

In the same manner, when the system 100 completes the merge process for the group 142, the system 100 gradually migrates the data access traffic from the group 144 to the group 144. As a result, the traffic ratio between the groups 142 and 144 begins to change in the opposite direction. When the traffic rate for the group 144 is zero, the system 100 is forwarding each data access request to the group 142. Thus, the group 144 is ready for the merge process. In some embodiments, update server 102 determines whether incremental data 122 has been merged into reference data 124 in each group (groups 142 and 144 in this example). If the incremental data 122 is not merged into the reference data 124 in one or more of the groups 142 and 144, the system 100 selects the group that currently needs the merge. It is assumed that the merge process has been completed for group 142 and not for group 144. [ The system 100 then selects the group 144 for merging. The system 100 removes the incremental data 122 from the memory 112 of the update server 102 when the incremental data 122 is merged into the reference data 124 at both of the groups 142 and 144 .

In some scenarios, the off-peak merge process may not be applicable. In one such scenario, a timeout event may occur for the merge process for the group. For example, the time allocated for the merge process for group 142 may pass without completion of the merge process. As a result, the system 100 may begin to bypass the traffic group 142 from the group 144. In other such scenarios, during the merge process for group 142, group 144 may not be able to serve all data access requests. Thus, the system 100 may forward some data access requests to the group 142, even during the merge process, to ensure maximum availability of online services (e.g., data accessibility).

In these two scenarios, the data access and merging process may not be separate between the groups 142 and 144. In some embodiments, the system 100 monitors the load of the storage servers in each group to effectively reduce the impact of the merge process on data access. When the load reaches the threshold, the system initiates an overload protection mode. In this mode, the system 100 suspends the merge process and releases the resources allocated for the merge process. For example, if the merge process is executed by a merge thread, the system 100 suspends the operations of the thread and releases the resources allocated to that thread, such as network, processing, and I / O resources. When the system 100 detects that the load is low, the system 100 may resume the merge process. In this way, the system 100 ensures quality of service (QoS) for data access requests of client devices by providing online services at a higher priority than the merge operations.

Figure 2 illustrates exemplary tablets of reference data, according to one embodiment of the present application. In this example, the reference data 124 is divided into a plurality of non-overlapping data segments. Such a data segment may be referred to as a tablet. A block of data at the storage server may include one or more tablets. For example, if the tablets 202, 206, and 208 of the reference data 124 are in a data block 126 hosted on the storage server 104 and the tablets 204, 210, And 212 are in the data block 128 hosted on the storage server 106.

To ensure high availability and isolation of the merge process, these tablets are also hosted in group 144. Tablets 202 and 210 are in data block 222 hosted at storage server 134 and tablets 204,206, 208 and 212 are in data block 224 hosted at storage server 136. Tablets 202, . In other words, the reference data 124 is entirely divided into non-redundant tablets 202, 204, 206, 208, 210, and 212, each of which has a plurality of storage servers / RTI > It should be noted that although each group includes each tablet of reference data 124, the group may host one copy of the tablet. As a result, the group may contain a single subset of the set of all copies of all the tablets.

In some embodiments, the system 100 uses compartmentalization of updated data (i.e., write operations) to avoid a race condition. When the amount of data written to the memory 112 of the update server 102 reaches a threshold value, the system 100 separates the amount of data for the merge process from any subsequent updates. In this way, during the merge process, the system 100 ensures that the same set of updates is merged into the reference data 124 in each group. The system 100 may also use distributed commits to avoid contention between groups. When the merge process is completed, the system 100 receives confirmation from the corresponding storage server that the merge process for that server has been completed. Upon receiving acknowledgments from each storage server in both groups 142 and 144, system 100 commits updates to each storage server. Otherwise, the system 100 may roll back to the previously comitted reference data at each storage server. This prevents data inconsistency between storage servers and groups due to the unsuccessful merging process.

FIGURE 3A illustrates a flow diagram illustrating a data management system for initiating a merge process, in accordance with one embodiment of the present application. During operation, the system monitors the incremental data at the update server (operation S301). The system checks whether the incremental data written to the memory of the update server is greater than or equal to the threshold value (operation S302). The amount of incremental data may be based on the number of bytes written to the memory. If the incremental data is not greater than or equal to the threshold, system increment data is continuously monitored (operation S301). If the incremental data is greater than or equal to the threshold, the incremental data is ready for the merge process. The system then checks whether the off-peak merge process is allowed (operation S303).

The system checks the timeout events of the previous merge operations and the load on the storage servers of each group to determine whether the off-peak merge process is allowed. If the off-peak merge process is allowed, the system initiates an off-peak merge process from the update server to the corresponding storage servers (act S304). Otherwise, the system initiates a dynamic-controlled merge process from the update server to the corresponding storage servers (act S305). It should be noted that although Fig. 3a only illustrates only one iteration of the merge process, the system continues to monitor incremental data at the update server after completion of the merge process (act S301). This ensures that whenever the amount of data written to the memory reaches a threshold, the system initiates the merge process.

Figure 3B shows a flow diagram illustrating an off-peak merging process of a data management system, in accordance with one embodiment of the present application. During operation, the system determines a merge sequence (or merge order) for the storage server groups (operation S331). In the example of FIG. 1B, the merge sequence may be that the group 142 is the first and the group 144 is the second, or the group 144 may be the first and the group 142 may be the second. The system then selects the group sequentially from that sequence (act S332). For example, if the sequence indicates that the group 142 is the first, the system 100 selects the group 142. The system gradually migrates the data access traffic to another group (s) (e.g., to group 144) (act S333).

The system checks if the selected group is ready for merging (operation S334). In some embodiments, the system maintains a traffic control table that includes the data access traffic ratios of each group. If the traffic rate for the selected group becomes zero, the system considers the selected group to be ready for the merge process. If the selected group is not ready for merging, the system will continue to migrate the data access traffic to another group (s) gradually (act S333). If the selected group is ready for merging, the system identifies storage servers in the selected group (operation S335). The system merges into corresponding tablets at the incremental data identified storage servers (act S336). For example, if a portion of the incremental data 122 includes updates to the reference data segment at the tablet 202, then the system 100 may determine that the corresponding portion of the incremental data 122, (202).

The system checks whether the merge process has been completed for the selected group (operation S337). For example, if a portion of the incremental data 122 includes updates to the reference data segments at the tablets 202, 204, 210, and 212, The merge process is completed for the group 142 when it finishes merging the tablets 202, 204, 210, and 212 hosted in the group 142 (or group 144). If the merge process is not complete, the system continues to merge the current incremental data set into the corresponding tablets at the identified storage servers (act S336). This allows compartmentalization of incremental data during the merge process. Any subsequent incremental data is merged into the next merge process. When the merge process is complete, the system gradually restores data access traffic for the selected group (operation S338). The system then checks whether all the groups have been merged (operation S339). If not, the system sequentially selects the next group from the sequence (operation S332).

3C depicts a flow diagram illustrating a dynamic-controlled merging process of a data management system, in accordance with one embodiment of the present application. During operation, the system identifies the storage servers of the selected group (operation S351). The system may select a group based on the current traffic load or available resources. The system checks the resource consumption of the storage servers (operation S352) and checks whether the resource consumption satisfies the start condition of the merge process (operation S353). The start condition may be whether or not resource consumption meets a threshold value. Examples of such resources include, but are not limited to, network resources, processing resources (e.g., central processing unit (CPU) cycles), and I / O resources.

If the resource consumption does not satisfy the start condition, the system continues to check the resource consumption of the storage servers (act S352). If resource consumption meets the start condition, the system identifies the current incremental data set to be merged (operation S354). Since the merge process is currently done only for the incremental data set, the resource allocation for the merge process is done for that current set. For example, if the merge process is run using a merge thread, the merge thread is only issued for the current incremental data set, and the system allocates resources only for that set. The system issues the next merge thread for the next incremental data set and thus allocates resources for the next thread. This " one-by-one " approach ensures that the next merge thread issues only when the load on the storage servers is low.

Upon identifying the current incremental data set to be merged, the system then initiates the merging process of the current incremental data set for the identified storage servers (act S355). The system collects data regarding performance of online services (operation S356). Examples of such data include, but are not limited to, request throughput, response latency, and request queuing. The system checks whether the performance level of the online services is less than the corresponding threshold level (operation S357). For example, the system may check whether the response latency exceeds the threshold time and / or the length of the request queue exceeds the critical length.

If the performance level is below the threshold level, the system initiates an overload protection mode, suspends the merge process (operation S358), and continues to collect data on the performance of online services (operation S356). In some embodiments, the system terminates the merge process in the overload protection mode. If the performance level is not below the threshold level, the merge process does not seriously hinder the performance level of the online services. The system then completes (e. G., Resumes) the merge process for the current incremental data set (operation S359). However, if the system has completed the merge process in the overload protection mode, the system can restart the merge process when the performance level of the online services is no longer below the threshold level.

4 is a schematic structural diagram of an exemplary data management system, in accordance with one embodiment of the present application. The data management system 400 may include a processor 402, a memory 404, and a storage device 420. Storage device 420 typically stores instructions that may be loaded into memory 404 and executed by processor 402 to perform the methods described above. In one embodiment, the instructions in the repository 420 may implement a selection module 422, a migration module 424, a merge module 426, and a control module 428, all of which may be implemented by various means Can communicate with each other.

In some embodiments, modules 422, 424, 426, and 428 may be implemented partially or entirely in hardware and may be part of processor 402. In addition, in some embodiments, the system may not include a separate processor and memory. Instead, in addition to performing their specific tasks, modules 422, 424, 426, and 428, either separately or in cooperation, may be part of special purpose computation engines.

The storage 420 stores programs to be executed by the processor 402. Specifically, the repository 420 stores a program that implements a system (application) for efficient data management using dynamic load-based merging. During operation, an application program may be loaded into memory 404 from storage 420 and executed by processor 402. [ As a result, the system 400 may perform the functions described above. The system 400 may be further coupled to the optional display 412, the keyboard 414, and the pointing device 416 and coupled to the network 410 via one or more network interfaces.

During operation, selection module 422 maintains a sequence of groups and selects a group from the sequence. Selection module 422 selects a storage server that requires a merge process. Selection module 422 also determines if the merge process was performed for the group. Migration module 424 migrates data access traffic from selected groups to other groups. In some embodiments, the migration module 424 maintains a traffic control table for the groups. When the traffic rate for the group becomes zero, the migration module 424 determines that the group is ready for the migration process. Upon completion of the merge process, migration module 424 recovers data access traffic back to the group.

Merge module 426 merges the incremental data from the updating server into one or more tablets of reference data at the storage server after the data access traffic is migrated from the selected group. The control module 428 determines, during the merge process, whether any data access traffic should be processed by the group that is currently merged. Control module 428 may initiate a dynamic-controlled merge process for the groups, as described in conjunction with Figure 3c, if the resource consumption state satisfies the start condition. For example, control module 424 may suspend (or terminate) the merge process and, if necessary, resume (or restart).

The data structures and computer instructions described in the present description are typically stored on a computer readable storage medium, which may be any device or medium capable of storing code and / or data for use by a computer system. Computer readable storage media include magnetic and optical storage devices such as volatile memory, non-volatile memory, disk drives, magnetic tape, compact discs, digital versatile discs or digital video discs, And other media that can store computer readable media now known or later developed.

The methods and processes described in the Detailed Description section may be embodied as code and / or data that may be stored in a computer-readable storage medium as described above. When a computer system reads and executes code and / or data stored on a computer-readable storage medium, the computer system performs the methods and processes embodied as data structures and code and stored in a computer-readable storage medium.

Moreover, the methods and processes described herein may be embodied in hardware modules or devices. These modules or devices may be implemented as application specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), specific software modules, , And / or other programmable logic devices that are now known or later developed. When hardware modules or devices are activated, they perform the methods and processes contained within them.

The foregoing description is presented to enable any person skilled in the art to make and use embodiments and is provided in the context of a particular application and its requirements. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein may be resorted to by those skilled in the art without departing from the spirit or scope of the disclosure, Lt; / RTI > Accordingly, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

Claims (21)

A computer implemented method that facilitates dynamic load-based merging,
Storing reference data in each storage device in a plurality of server groups;
Storing the incremental data in a memory device of a first computing device until the amount of incremental data reaches a threshold value, the incremental data being stored in a second computing device of a first one of the plurality of server groups The updated data associated with the reference data in the storage device of the storage device;
Determining if the amount of incremental data exceeds the threshold;
Selecting the first server group to merge the incremental data from the plurality of server groups in response to the amount of incremental data exceeding the threshold;
Migrating data access requests for the reference data from the first server group to a second one of the plurality of server groups; And
Merging the incremental data from the memory device of the first computing device with the storage device of the second computing device
/ RTI > 2. The method of claim 1, wherein the reference data is divided into a plurality of tablets, wherein the tablet is a non-overlapping segment of the reference data. 3. The method of claim 2, wherein each server group of the plurality of server groups includes a copy of each tablet of the reference data. 2. The method of claim 1, wherein selecting the first server group from the plurality of server groups comprises identifying the first server group from a sequence of server groups. 5. The method of claim 4,
Determining whether merging of the incremental data has been performed for each server group; And
Further comprising selecting a server group in which the merging of the incremental data has not been performed. The method according to claim 1,
Maintaining a traffic control table including a ratio of data access request traffic of each server group among the plurality of server groups;
Determining if the traffic rate for the first server group has reached a threshold in the traffic control table; And
And in response to the determination, determining whether the first server group is ready for the merge. The method according to claim 1,
Determining if the merge has been completed for the first server group; And
And in response to the determination, restoring data access requests from the second server groups back to the first server group. The method according to claim 1,
Determining whether the first server group needs to serve data access requests during the merge of the incremental data; And
Further comprising determining whether resource consumption by computing devices in the first server group meets a condition. 9. The method of claim 8,
Determining a performance level associated with the data access requests served from the first server group; And
Responsive to the performance level being below a threshold, suspending the merge of the incremental data. 10. The method of claim 9, further comprising: releasing resources occupied by the merge of the incremental data to serve the data access requests. 18. A non-volatile storage medium storing instructions that, when executed by a processor, cause the processor to perform a method that facilitates dynamic load-based merge,
Storing reference data in each storage device in a plurality of server groups;
Storing the incremental data in a memory device of a first computing device until the amount of incremental data reaches a threshold value, the incremental data being stored in a second computing device of a first one of the plurality of server groups The updated data associated with the reference data in the storage device of the storage device;
Determining if the amount of incremental data exceeds the threshold;
Selecting the first server group to merge the incremental data from the plurality of server groups in response to the amount of incremental data exceeding the threshold;
Migrating data access requests for the reference data from the first server group to a second one of the plurality of server groups; And
Merging the incremental data from the memory device of the first computing device with the storage device of the second computing device
And a non-volatile storage medium. 12. The non-transitory storage medium of claim 11, wherein the reference data is partitioned into a plurality of tablets, wherein the tablet is a non-overlapping segment of the reference data. 13. The non-transitory storage medium of claim 12, wherein each server group of the plurality of server groups includes a copy of each tablet of the reference data. 12. The non-transitory storage medium of claim 11, wherein selecting the first server group from the plurality of server groups comprises identifying the first server group from a sequence of server groups. 15. The method of claim 14,
Determining whether merging of the incremental data has been performed for each server group; And
Further comprising selecting a server group in which the merge of the incremental data has not been performed. 12. The method of claim 11,
Maintaining a traffic control table including a ratio of data access request traffic of each server group among the plurality of server groups;
Determining if the traffic rate for the first server group has reached a threshold in the traffic control table; And
And in response to the determination, determining whether the first group of servers is ready for the merge. 12. The method of claim 11,
Determining if the merge has been completed for the first server group; And
And in response to the determination, restoring data access requests from the second server groups back to the first server group. 12. The method of claim 11,
Determining whether the first server group needs to serve data access requests during the merge of the incremental data; And
Further comprising determining whether resource consumption by computing devices in the first server group meets a condition. 19. The method of claim 18,
Determining a performance level associated with the data access requests served from the first server group; And
Responsive to the performance level being below a threshold, suspending the merge of the incremental data. 20. The non-transitory storage medium of claim 19, wherein the method further comprises: releasing resources occupied by the merge of the incremental data to serve the data access requests. 1. A computing system that facilitates dynamic load-based merge,
A processor; And
A memory coupled to the processor for storing instructions that when executed by the processor cause the processor to perform a method,
The method comprising:
Storing reference data in each storage device in a plurality of server groups;
Storing the incremental data in a memory device of a first computing device until the amount of incremental data reaches a threshold value, the incremental data being stored in a second computing device of a first one of the plurality of server groups The updated data associated with the reference data in the storage device of the storage device;
Determining if the amount of incremental data exceeds the threshold;
Selecting the first server group to merge the incremental data from the plurality of server groups in response to the amount of incremental data exceeding the threshold;
Migrating data access requests for the reference data from the first server group to a second one of the plurality of server groups; And
Merging the incremental data from the memory device of the first computing device with the storage device of the second computing device
The computing system.

Images