TWI505111B - Assist systems and methods for decentralized database to achieve automatic load balancing - Google Patents

Description

System and method for assisting distributed database to achieve automatic load balancing

The invention relates to the field of data processing, and can assist the distributed database to find an efficient data relocation or copying scheme when the load is uneven, and relocate or copy the data from the busy database host to the new startup or not busy. On the host of the database, the purpose of balancing the load is achieved, which effectively reduces the amount of data that needs to be relocated or copied, and reduces the time when the database is oscillated due to data relocation or replication.

Press, load balancing is a core issue of distributed information systems. When a service system's service request is shared by multiple hosts, a load balancing mechanism is required to balance the load of each host as much as possible. The load balancing of the distributed database is more complicated than the load balancing of the distributed web server. Because the load balancing of the distributed database requires additional data relocation or replication, a large amount of data is loaded from a large load. Host, relocated or copied to a host or new host with a small load. However, in the process of data relocation or replication, the performance of the database will be greatly affected, resulting in fluctuations in system performance.

The prior art is as disclosed in U.S. Patent No. 20,120,254,175. It is pointed out that "the exchange can be selectively performed in a low load time slot (such as midnight or during maintenance) because the exchange of the copy in terms of execution cost is higher than the connection allocation. However, if the load balancing requirement comes from an unexpected temporary high load condition, the load balancing method proposed by the prior art is impractical.

Prior art, such as U.S. Patent No. 2,120,254,175, issued to the inventor of MongoDB, discloses that MongoDB performs a distributed database optimization through data cutting and moving. MongoDB is a decentralized database, which can be composed of multiple database hosts. Each database host has a primary key range that it is assigned to govern, so it can be integrated through horizontal partitioning. The data is split and stored in multiple hosts according to the primary key to improve the overall read and write performance of the database. However, when the amount of data in the database increases, it may be that a small number of hosts are overburdened due to uneven distribution of data. Therefore, the prior art proposes two methods of optimizing the database: monitoring whether the size of the data chunk exceeds a threshold in the database, cutting if necessary, and distributing the cut data chunk to multiple hosts; and monitoring data Whether the number of data chunks between the libraries is seriously unbalanced, and if so, the data chunks are relocated from hosts with a large number of data chunks to hosts with a small number of data chunks. Both of these approaches focus on the balance of "data volume" and hope to achieve load balancing by averaging the amount of data between hosts.

However, this prior art practice ignores the fact that there is a big gap in the popularity of the data: most requests may be concentrated on a small amount of data. The balanced distribution of data has its needs for long-term maintenance (hardware Proper use of the source), but if the load balancing needs for short-term and immediate, if you can consider the popularity of the data, or you can design a more efficient optimization program, in a shorter time and less impact on performance, Achieve load balancing.

In addition, the prior art also ignores the important fact that there is a specific access pattern between data, that is, a certain material may often be accessed simultaneously with another material by an external request. If two materials with a common access pattern are placed on a single host, the external request can be satisfied once, without having to make multiple requests to multiple hosts. In the prior art, when data is copied or moved, only the balance of the total amount of data between the hosts is considered, and the influence of the access pattern association of the individual data on the number of requests is not considered, and some data relocation or copying schemes may be Achieving a balanced amount of data, but derived more requests.

The object of the present invention is to provide a system and method for assisting a distributed database to achieve automatic load balancing, considering the popularity degree (H) of the data block and the similar weight (S) of the access mode, and giving priority to the most effective performance improvement. Data relocation or copying.

A second object of the present invention is to provide a system and method for assisting a distributed database to achieve automatic load balancing. The data is processed in units of data blocks, that is, one time data migration and copying are involved. Into an integer number of data blocks, in order to comply with most of today's operating systems and database systems with similar ideas for data relocation and replication.

The system and method for assisting the distributed database to achieve automatic load balancing, including the following steps: Step 1: Calculate the numerical data from the data inventory taking record and the data block configuration data, and establish an empty data. The relocation or replication scheme combination Z=(), and an optimal data relocation or replication scheme combination Z ^* that has not been specified; Step 2: define a performance conversion function and a performance target value, and the performance conversion function is a function f, A data block set can be converted into a value, which is positively correlated with the total number of accesses of the data block set, and the performance target value P ^* is a real number; Step 3: Estimating the number of database hosts that need to be additionally activated Estimating the performance conversion function and the performance target value; Step 4: Determining a data block b, and a host m currently storing the data block b replica, if the data block b on the host m is removed, Maximum performance improvement; Step 5: Decide on a newly started or not busy database host m', and move the data block b on the host m into m' to minimize the performance loss; Step 6 Decide on the relocation or copying plan to be added to Z. According to the net performance improvement amount, the net performance improvement amount is determined by the value of step 4 and the value of step 5; Step 7: Check whether the performance target is achieved, and calculate the data by using the performance conversion function. The performance estimate of the library host P _i , if any of the performance estimates do not achieve the performance goal ( P _i P ^* ), repeat step 4, if the performance goal is reached, then consider Z as the current best solution Z ^* , ie Z ^* = Z; Step 8: Check if the additionally launched database host is not fully utilized, If yes, reduce the number of hosts that need to be newly started in step (2), empty Z, recalculate C, and repeat step 4; Step 9: output the best data relocation or replication scheme, output Z ^* is the best Data relocation or copying plan. If Z ^{* is} still not specified, no data will be relocated or copied.

The present invention further provides a system for assisting a distributed database to achieve automatic load balancing, comprising: a decentralized database, at least a single database consisting of a set of database hosts storing data areas through data partitions; Block, each host only stores a part of the data block; a data store takes the record capture module, and the access record is retrieved from the distributed database; a database block configuration data capture module is distributed The block configuration data is retrieved from the database; a load balancing planning module is used to find the best data relocation or copying plan; a cloud resource management module provides a new database host; and a data block migration module provides Relocate or copy data blocks between hosts of each database.

101~109‧‧‧Step 1~Step 9

210‧‧‧Assisting distributed databases to achieve load balancing systems

211‧‧‧Data Inventory Technology Capture Module

212‧‧‧Database Block Configuration Data Capture Module

213‧‧‧Load Balancing Planning Module

214‧‧‧Cloud Resource Management Module

215‧‧‧Data Block Migration Module

220‧‧‧ Cluster of original distributed database hosts

221‧‧‧ original decentralized database

222‧‧‧data block

230‧‧‧Virtual Hosting Group

231‧‧‧New Decentralized Database

232‧‧‧New data block

300‧‧‧Distributed database

1 is a component flow diagram of a system and method for assisting a distributed database to achieve automatic load balancing according to the present invention; Figure 2 is a schematic diagram of the actual operation of the system for assisting the distributed database to achieve automatic load balancing; Figure 3 is a schematic diagram of the distributed database.

Please refer to FIG. 1 . The method provided by the present invention for assisting a distributed database to achieve data relocation or replication of an automatic load balancing system includes the following steps:

Step 1 (101): Calculate the following numerical data for subsequent use: calculate the popularity degree vector H of the data block: from the data inventory take record 350, summarize the number of times each data block is accessed. Users can decide according to their needs to collect data inventory records for a long period of time. After this calculation is completed, a popularity degree vector H is obtained, and H _i is the number of times the i-th data block is accessed, and the value is a non-negative integer.

Calculating the configuration matrix of the data block C: Calculating the configuration matrix of the data block from the data block configuration data 340 of the database. The configuration matrix C is a two-dimensional matrix of h * n, h is the number of data blocks, and n is the number of database hosts. C _ij is 1, indicating that there is a copy of the i-th data block on the host j, and C _ij is 0, indicating that the i-th data block does not have a copy on the host j.

The access association matrix S of the data block is calculated: from the data inventory take record 350, an access association matrix is calculated. The user can decide on how long to take the data inventory to take the calculation according to the needs. The access association matrix S is a two-dimensional matrix of h*h (h is the number of data blocks), S _ij is the access association between the data block i and the block j, and S _ij is 0 indicating data. Whether block i and data block j exist on the same machine has no effect. A negative S _ij indicates that having the data block i placed on a different machine than the data block j has a negative effect. S _ij is positive means that having the data block i placed on a different machine than the data block j will have a positive effect. In a preferred embodiment, the access mode similar weight calculation method is: If i and j are not equal S _ij =1, if i is equal to j, and an empty data relocation or copying scheme step Z=() is required, and an optimal data relocation or copying scheme step has not been specified. Z ^* .

Step 2 (102): The performance conversion function and the performance target value need to be defined. The performance conversion function is a function f, which can convert a data block set D into a numerical value, which is a performance estimation value of the data library host storing the data block set D, and the value needs to be combined with the data block set D. The total number of accesses is positively correlated. Assuming that the performance estimates of the current n database hosts (M ₁ , M ₂ , M ₃ , . . . , M _n ) are (P ₁ , P ₂ , P ₃ , . . . , P _n ), a performance conversion function f can be used. Measuring Mi's performance estimate P _i

Where B _j represents the jth data block, and Access ({ B }) represents the total number of accesses of the data block set B.

The performance target value is a value P ^* , and the performance goal is that all P _i are expected to be less than the value P ^* , that is:

In a preferred embodiment, the performance function f is a function of converting the "data block set" into "the number of external requests carried by the database host storing the data block set for a certain period of time", that is, The preferred embodiment hopes that the number of external requests carried by each host within a certain period of time does not exceed a threshold P ^* to maintain the service quality of the database on the host.

Step 3 (103): Estimate the number of database hosts that need additional startup. Assume that the current performance estimates of the _n database hosts (M ₁ , M ₂ , M ₃ , ..., M _n ) are (P ₁ , P ₂ , P ₃ , ..., P _n ), and the following formula is used to estimate the need for a new start. The number of database hosts △ n is:

Extend C to a two-dimensional matrix of h *( n +Δ n ) and make it expand to a value of zero.

Step 4 (104): Use H, C, S to determine a data block b, and a host m that currently has a copy of the data block b. If the data block b on the host m is removed, the maximum performance can be achieved. Upgrade. This data block b and host m are determined by the following formula:

Restricted P _m P ^* indicates that the performance estimate of this host m does not achieve a performance goal. If you cannot find a b and m combination that meets the restrictions, skip to step 7.

Step 5 (105): Using C, S to determine a newly started or not busy database host m', and moving the data block b on the host m into m' can bring about minimal performance impairment. This host m' can be determined by the following formula:

Where D _{m '} represents a collection of data blocks on the database host m'.

Step 6 (106): Calculate the net performance improvement amount and decide the relocation or replication plan to be added to Z. The decision method is as follows:

(1) If the performance improvement amount of step 4 is positive, the step of "Relocating data block b on host m to m'" is added to Z, and C _{bm ' is} updated to 1. The value of P _{m '} may also vary due to changes in C _{bm '} values.

(2) However, if the performance improvement amount of step 4 is negative, add the step of "copy data block b on host m to m'" to Z, update C _bm to 0, and update C _{bm '} 1. Since the values of C _bm and C _{bm '} vary, the values of P _m and P _{m '} may also vary.

Step 7 (107): If there is P _i P ^* (ie, the performance goal is not achieved), then repeat step 4. If not, Z is considered to be the current best solution Z ^* , ie Z ^* =Z.

Step 8 (108): If Δ n >0, apply the performance conversion function to check whether the average value of the predicted performance indicators ( P _{n +1} , . . . , P _{n +Δ n} ) of the newly launched database host is less than a preset. The threshold ( r * P ^* ), that is, whether the average performance usage of the newly activated host reaches the r ratio of the performance target value P ^* . If yes, reduce the number of hosts that need to be newly started in the second step by one (△ n ←△ n -1), empty Z, and re-extend C to a two-dimensional structure of h *( n +Δ n ) Matrix, and make its expanded value 0, repeat step 4.

Step 9: Output Z ^* as the final data relocation or copying plan.

For practical application of the present invention, please refer to FIG. 2, a system 210 for assisting a distributed database to achieve load balancing, comprising five main modules. The load balancing planning module 213 implements the technical implementation method disclosed in the patent (such as FIG. 1). The module needs the data inventory acquisition technology module 211 to assist in capturing the data inventory acquisition technology and the database configuration data. The capture module 212 retrieves the repository block configuration data. If the load balancing scheme generated by the 213 module indicates that an additional database host is required (ie, Δ n >0), the cloud resource management module 214 dispatches the Δ n virtual host group 230 in the cloud IaaS environment, and performs Appropriate hardware and software settings, with the functions required for the distributed database, and the virtual host group 230 is added to the cluster 220 of the original distributed database host. The data block migration module 215 then performs the relocation and/or copying of the data block according to the load balancing scheme generated by the 213 module, for example, copying a data block 222 on the original distributed database 221 to the new data block 222. The distributed database 231 becomes a new data block 232.

Please refer to FIG. 3, which is a schematic structural diagram of a distributed database 300 of the present invention, wherein: The distributed database 300 stores at least a group of database hosts 310, 320, 330 a single data repository through a data partition, and each host stores only a portion of the data blocks 311, 312; wherein: the distributed database is a single database composed of more than one database host, the distributed database stores the data blocks through the data partition, and each host only stores a part of the data blocks. There will be no host that stores all the data blocks. Please continue to refer to FIG. 3 to further illustrate the features of the distributed database 300. The distributed database 300 includes three database hosts 310, 320, and 330. The data library host 310 includes two data blocks 311, 312, respectively storing the data block A and the data block B; the database host 320 includes two data blocks 321, 322, respectively storing the data block B and the data block C; The database host 330 includes two data blocks 331, 332 for storing the data block A and the data block C, respectively. The data block A has two duplicates 311 and 331, which are respectively stored in the database hosts 310 and 330; the data block B has two duplicates 312 and 321 which are respectively stored in the database hosts 310 and 320; the data block C has two The replicas 322 and 332 are stored in the database hosts 320 and 330, respectively. Due to the data cutting relationship, the decentralized database needs to maintain an additional mechanism for recording or determining which host the data exists on. A common mechanism is to maintain a consistent hash function that determines which host the data resides on. Another common mechanism is to maintain a look-up table to record which host the data resides on. The database also needs to record the correspondence between the data and the data block to target the data. Request access to the corresponding data block. This patent refers to the above mechanism and data as "data block configuration data" and is indicated in the data area configuration data 340 in FIG. In order to maintain the durability and disaster recovery requirements of the database, the database stores past data access records on a persistent storage medium. The data access records are recorded in the data inventory record in Figure 3. 350.

The data inventory is taken from the record capture module 211, and the access record is retrieved from the distributed database 300; the database block configuration data capture module 212 is used to retrieve the block configuration data from the distributed database 300. 340; the load balancing planning module 213 is configured to find an optimal data relocation or replication solution; the cloud resource management module 214 provides a new database host 310-330; and the data block migration module 215 provides each station. The data bases 310 to 330 relocate or copy the data block 220.

The following is the actual operation of the present invention: assuming that a distributed database consists of three database hosts (Host A, Host B, Host C), and stores seven data blocks (data block A, data block B, Data block C, data block D, data block E, data block F, data block G). The data block configuration is as follows: Table 2: Data block configuration

And from the data inventory record, the number of queries of different queries and the data blocks to be accessed are counted.

Step 1 (101): Calculate the following numerical data for subsequent use

(1) Calculate the popularity degree vector H of the data block: (115000, 100500, 20000, 78000, 13000, 41000, 1000, 75000), representing the data blocks A, B, C, D, E, F, G, respectively. The number of times H is accessed.

(2) Calculate the configuration matrix of the data block For example, the first (column) of (1,0,0,1,0,0,0,0) indicates that the data block A and the data block D are on the host A.

(3) Calculate the access association matrix S of the data block:

Here, S _1,2 (-0.87) is taken as an example to explain the value. First, the data block A is accessed 115,000 times in this data inventory record. This number is the denominator; when the data block A is accessed, 100000 times will access the data block B at the same time. This number is a numerator. Dividing the two numbers and taking a negative number (-100000/115000) is its value.

Step 2 (102): defining a performance conversion function and a performance target value, Here, the performance function f is defined as "the number of external requests hosted by the database host of this data block set for a certain period of time". Taking host A as an example, referring to Table 2, it can be known that the data block set stored by host A is {A, D}. Referring to Table 3, the query involving {A, D} has Query 1, 4, 9, 10, 11, 12, 13, and the total number of queries of these Query is: 188000 (10000+70000+90000+10000+) 5000+2000+1000), which is the estimated performance of the host A. The same algorithm can calculate the performance estimate of host B is 114,500, and the estimated performance of host C is 136000. The performance target value is set at 140,000, that is, all database hosts are expected to have less than 140,000 external requests carried in a certain period of time to maintain the service quality of the database on the host.

Step 3 (103): Estimate the number of database hosts that need additional startup

According to this formula , can be calculated .

Name the database host that needs additional startup as the host and update the configuration matrix C of the data block to: .

Step 4: Determine a data block b, and a host m that currently has a copy of the data block b. If the data block b on the host m is removed, the maximum performance improvement can be achieved. This data block b and host m are determined by the following formula:

This formula can be calculated: when (b = data block A, m = host A), its value is 115000 * (1-0.043) = 110000, does not violate the restriction formula (188000> = 140000, C11 = 1), And is the maximum.

Step 5 (105): Determining a newly started or not busy database host m', and moving the data block b on the host m into m' can bring about minimal performance impairment. This host m' can be determined by the following formula:

This formula can be calculated: when (m'=Host B), its value is -0.87, does not violate the restriction formula (129500<140000, C ₁₂ =0), and is the minimum value.

Step 6 (106): The performance improvement amount (110000) of step 4 is positive, so the step of "relocating data block A on host A to host B" is added to Z, and C is updated to: .

Step 7 (107): Check if there is a performance estimate P _{i of the} host. P ^* (ie, no performance goal is achieved). All achieve the performance goal, so Z ^* = {relocate the data block A on the host A to the host B}, which is the best solution at present.

Step 8 (108): r is set to 0, 8, ie the threshold is (0.8* P ^* ) = 112000. The estimated performance of the host D is 0, which is less than this threshold, indicating that the newly added host is not fully utilized, which can reduce the number of newly started database hosts. After reducing the number of database hosts by one (△ n ← △ n -1), the result is the same as Z ^* .

Step 9 (109): Therefore, Z ^* = (relocating the data block A on the host A to the host B) is the final data relocation or copying scheme.

The detailed description of the preferred embodiments of the present invention is intended to be limited to the scope of the invention, and is not intended to limit the scope of the invention. The patent scope of this case.

In summary, this case is not only innovative in terms of space type, but also can enhance the above-mentioned multiple functions compared with conventional articles. It should fully comply with the statutory invention patent requirements of novelty and progressiveness, and apply for it according to law. Approved this invention patent application, in order to invent invention, to the sense of virtue.

101~109‧‧‧Step 1~Step 9

Claims (7)

A method for relocating or copying data of a system that assists a distributed database to achieve automatic load balancing includes: Step 1: Calculate the numerical data from the data inventory taking record and the data block configuration data, and establish an empty Data relocation or replication scheme combination Z=(), and an optimal data relocation or replication scheme combination Z ^* that has not been specified; Step 2: Define a performance conversion function and a performance target value, and the performance conversion function is a function f , a data block set can be converted into a value, the value needs to be positively correlated with the total number of accesses of the data block set, and the performance target value P ^* is a real number; Step 3: estimating the database host that needs additional startup Quantity, using the performance conversion function and the performance target value for estimation; Step 4: Determining a data block b, and a host m that currently has a copy of the data block b, if the data block b on the host m is removed To maximize performance improvement; Step 5: Decide on a newly started or not busy database host m', and move the data block b on the host m into m' to minimize performance loss; Step 6: Decide on the relocation or copying plan to be added to Z. According to the net performance improvement amount, the net performance improvement amount is determined by the value of step 4 and the value of step 5; step 7: check whether the performance target is achieved, and calculate the performance conversion function. The performance estimate P _{i of} each database host, if any one of the performance estimates does not achieve the performance goal ( P _i P ^* ), repeat step 4, if the performance goal is reached, then consider Z as the current best solution Z ^* , ie Z ^* = Z; Step 8: Check if the additionally launched database host is not fully utilized, If yes, reduce the number of hosts that need to be newly started in step (2), empty Z, recalculate C, and repeat step 4; Step 9: output the best data relocation or replication scheme, output Z ^* is the best Data relocation or copying plan. If Z ^{* is} still not specified, no data will be relocated or copied. For example, the method for assisting the distributed database to achieve the data relocation or replication of the automatic load balancing system as described in the first application of the patent scope, wherein the numerical data needs to calculate the following three types of data: (1) the hotspot of the data block The degree vector H, (2) the configuration matrix C of the data block, and (3) the access association matrix S of the data block. For example, the application for assisting the distributed database to achieve the data relocation or replication of the automatic load balancing system as described in the first application of the patent scope, wherein the estimation requires an additional database host number formula: assume that the current n database The performance of the host (M ₁ , M ₂ , M ₃ , ..., M _n ) is (P ₁ , P ₂ , P ₃ , ..., P _n ), and the number of database hosts to be newly started is estimated by the following formula Δ n for: . For example, the method for assisting a distributed database to achieve data relocation or replication of an automatic load balancing system as described in claim 1 of the patent application scope, wherein the data block b and the host m are determined by the following formula: Where P _m is the estimated performance of the mth database host, which can be obtained by: P _i = f ({ B _j }), st ( C _ji =1) ^ ( f ({ B }) Access ({ B })) where B _j represents the jth data block and Access ({ B }) represents the total number of accesses of the data block set B. The method for assisting a distributed database to achieve data relocation or replication of an automatic load balancing system as described in claim 1 of the patent application scope, wherein the host m' can be determined by the following formula: Where D _{m '} represents a collection of data blocks on the database host m'. The method for relocating or copying data of a system that assists a distributed database to achieve automatic load balancing, as described in claim 1, wherein the calculated net performance increase And decide on the relocation or copying plan to be added to Z. The decision method is as follows: (a) If the net performance improvement is positive and the performance improvement of step 4 is positive, then the data block b on the host m will be relocated. Add Z to m' and update C; (b) If the net performance improvement is positive, but the performance improvement of step 4 is negative, then "copy data block b on host m to m'" Z, and update C; (c) Otherwise, skip to step (9) and output Z ^* as the best data relocation or copying scheme combination. A system for assisting a distributed database to achieve automatic load balancing, comprising: a decentralized database, at least a single database consisting of a set of database hosts, through a data partition to store data blocks, each host Only a part of the data block is stored; a data inventory is taken from the record capture module, and the access record is retrieved from the distributed database; a database block configuration data capture module is captured by the distributed database Block configuration data; a load balancing planning module to find the best data relocation or replication solution; a cloud resource management module to provide a new database host; a data block migration module to provide each database host Relocate or copy data blocks.