WO2021217864A1 - Load balancing method and apparatus for storage cluster, computer device, and storage medium - Google Patents

Abstract

A load balancing method for a storage cluster, relating to the field of cloud technology. The method comprises: upon receiving a load balancing instruction, calculating a group-count mean value for basic data of all physical nodes in a storage cluster (S10); determining, according to the group-count mean value and a group count of basic data in each physical node, a migration group count of basic data in each physical node (S20); determining, for each physical node, migration data from the basic data according to the migration group count (S30); acquiring, from all the physical nodes in the storage cluster, a physical node having the smallest group count of basic data to serve as an initial migration-reception node (S40); and if the initial migration-reception node has different basic data, migrating the migration data to the initial migration-reception node so as to perform load balancing (S60). The invention thus improves the performance of storage clusters.

Description

Load balancing method, device, computer equipment and storage medium of storage cluster

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on April 29, 2020, the application number is 202010355344.X, and the invention title is "Load balancing methods, devices, computer equipment and storage media for storage clusters". The entire content is incorporated into this application by reference.

Technical field

This application relates to the field of cloud technology, and in particular to a load balancing method, device, computer equipment, and storage medium of a storage cluster.

Background technique

Because of its convenient and flexible scalability, software-defined storage allows the storage system to scale arbitrarily within the scope of the hardware. It is being adopted by more and more companies. With the popularity of cloud storage, cloud storage management has become an indispensable technology in the storage field. Cloud storage management is a type of application that monitors and manages storage resources on the physical and logical layers. However, after the cluster continues to expand and upgrade, it will face the problem of high data distribution and uncontrollable data distribution.

As the scale of the cluster expands and new machines continue to join, the problem of uneven distribution of data in the cluster will inevitably occur, causing some machines to be idle and other machines to be heavily loaded; at the same time, due to the different cold and hot data, it will cause the IO of the cluster. The distribution is unbalanced, thereby reducing the effective capacity utilization rate and overall performance of the cluster. However, the inventor realizes that for the current multi-dimensional balance problems such as data balance and IO balance, the usual approach is to comprehensively consider multiple factors, assign weights to each factor according to the degree of importance, and sort them according to the addition of the weights to generate the final migration. Strategy. This multi-dimensional balancing algorithm makes the cluster unable to achieve the optimal in each dimension, and it is difficult to ensure the synchronization of data reading and writing, and it is difficult to achieve load balancing, which reduces the performance of the cluster.

technical problem

The embodiments of the present application provide a load balancing method, device, computer equipment, and storage medium for a storage cluster to solve the problem of low performance of the storage cluster.

Technical solutions

A load balancing method for a storage cluster. The storage cluster includes multiple physical nodes, and each physical node contains multiple sets of basic data. The load balancing method for the storage cluster includes: when a load balancing instruction is received, calculating the storage The average number of groups of the basic data of all physical nodes in the cluster; according to the average number of groups and the number of groups of the basic data in each physical node, determine the migration of the basic data in each physical node The number of outgoing groups; for each physical node, the outgoing data is determined in the basic data according to the number of outgoing groups; the number of groups of basic data corresponding to all physical nodes of the storage cluster is compared, and the basic data The physical node with the smallest number of groups is used as the initial migration node; the initial migration node is detected to determine whether the basic data in the initial migration node is the same, and the basic data in the initial migration node When the same, the initial migration node is excluded, the updated initial migration node is obtained, and the comparison of the number of groups of basic data corresponding to all physical nodes of the storage cluster is performed, and the number of groups of basic data is the smallest The physical node is used as an initial migration step; when the basic data in the initial migration node is different, the migration data is migrated to the initial migration node to achieve load balancing.

A load balancing device for a storage cluster. The storage cluster includes a plurality of physical nodes, and each physical node contains multiple sets of basic data. The load balancing device for the storage cluster includes: an average value calculation module for when a load balancing instruction is received Calculate the average number of groups of the basic data of all physical nodes in the storage cluster; the migration group number determination module is used to calculate the average number of groups and the basic data in each physical node The number of groups determines the number of outgoing groups of the basic data in each physical node; the outgoing data determining module is used to determine the outgoing data in the basic data for each physical node according to the number of outgoing groups Migration node acquisition module, used to compare the number of groups of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest number of groups of basic data as the initial migration node; Migration node detection A module for detecting the initial migration node, judging whether the basic data in the initial migration node is the same, and when the basic data in the initial migration node is the same, excluding the initial migration node, Obtain the updated initial migration node, and perform the comparison of the group numbers of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest group number of basic data as the initial migration node Step; a load balancing module for migrating the migration data into the initial migration node when the basic data in the initial migration node is different.

A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the load balancing method for a storage cluster when the processor executes the computer program For example, the following steps are implemented: when a load balancing instruction is received, the average number of groups of basic data of all physical nodes in the storage cluster is calculated, the storage cluster includes multiple physical nodes, and each physical node contains multiple sets of basic data ; According to the average of the number of groups and the number of groups of the basic data in each physical node, determine the number of migration groups of the basic data in each physical node; for each physical node, according to the migration The number of outgoing groups determines the outgoing data in the basic data; the group numbers of basic data corresponding to all physical nodes of the storage cluster are compared, and the physical node with the smallest group number of basic data is used as the initial incoming node Detect the initial migration node to determine whether the basic data in the initial migration node is the same, and when the basic data in the initial migration node is the same, exclude the initial migration node to obtain the updated And execute the step of comparing the number of groups of basic data corresponding to all physical nodes of the storage cluster, and using the physical node with the smallest number of groups of basic data as the initial migration node; When the basic data in the initial migration node is different, the migration data is migrated to the initial migration node.

A computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the load balancing method of the storage cluster are realized, for example, the following steps are realized: When balancing instructions, calculate the average number of groups of basic data of all physical nodes in the storage cluster. The storage cluster includes multiple physical nodes, and each physical node contains multiple groups of basic data; The number of groups of the basic data in the physical node determines the number of migration groups of the basic data in each physical node; for each physical node, the number of migration groups is determined in the basic data according to the number of migration groups. Out data; compare the number of groups of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest number of groups of basic data as the initial migration node; detect the initial migration node, Determine whether the basic data in the initial migration node is the same. When the basic data in the initial migration node is the same, exclude the initial migration node to obtain the updated initial migration node, and execute the pairing The number of groups of basic data corresponding to all physical nodes of the storage cluster is compared, and the physical node with the smallest number of groups of basic data is used as the initial migration node; the basic data in the initial migration node is not At the same time, the migration data is migrated into the initial migration node.

Beneficial effect

The foregoing load balancing method, device, computer equipment and storage medium of the storage cluster improve the performance of the storage cluster.

Description of the drawings

Fig. 1 is a flowchart of a load balancing method for a storage cluster provided by an embodiment of the present application.

Fig. 2 is another flowchart of a load balancing method for a storage cluster provided by an embodiment of the present application.

Fig. 3 is a flowchart of a basic data distribution method provided by an embodiment of the present application.

FIG. 4 is another flowchart of the load balancing method of the storage cluster provided by the embodiment of the present application.

Fig. 5 is a flowchart of a load balancing method provided by an embodiment of the present application.

Fig. 6 is a flowchart of a method for determining a role tag provided by an embodiment of the present application.

FIG. 7 is another flowchart of the load balancing method of the storage cluster provided by the embodiment of the present application.

Fig. 8 is a functional block diagram of a load balancing device for a storage cluster provided by an embodiment of the present application.

Fig. 9 is a schematic diagram of a computer device provided by an embodiment of the present application.

Embodiments of the present invention

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

The technical solution of the present application can be applied to the field of big data and/or blockchain technology, and the related data can be stored in a database such as a database cluster, or can be distributed storage through a blockchain, etc., which is not limited by this application.

As shown in Figure 1, in one embodiment, a load balancing method for a storage cluster is provided. This embodiment is applied to a distributed storage system as an example. The storage cluster of the distributed storage system includes multiple physical nodes. , Each physical node contains multiple sets of basic data, and each physical node is connected to the network, the load balancing method of the storage cluster specifically includes the following steps.

S10: When the load balancing instruction is received, calculate the average group number of the basic data of all physical nodes in the storage cluster.

Among them, the load balancing instruction refers to the basic data sharing strategy on each physical node in the storage system, the physical node refers to the node server in the storage system, and the basic data refers to the load currently carried by the node server, such as the data requested by the user. When it is detected that the load is unbalanced in the storage cluster, for example, a new physical node is added to the cluster. At this time, the storage cluster will be seriously unbalanced (for example, the load of the newly added physical node is empty). ), the distributed storage system automatically triggers load balancing instructions to calculate the average number of groups of the basic data of all physical nodes in the storage cluster, that is, the number of groups is calculated based on the sum of the number of groups of the basic array on each physical node and the number of all physical nodes The average value, that is, the number of basic data groups that each physical node can achieve a better number of load balancing.

S20: Determine the number of migration groups of the basic data in each physical node according to the average number of groups and the number of groups of the basic data in each physical node.

Among them, the number of migration groups refers to the number of groups of basic data that need to be migrated to other physical nodes in order to ensure the balance of physical nodes. Specifically, after receiving the load balancing instruction, the basic data needs to be migrated out according to the distribution of the basic data on the physical nodes, and an appropriate amount of basic data is migrated out to balance the load of the physical nodes. Therefore, you can By comparing the average number of groups with the number of groups of basic data in each physical node, when the number of groups of basic data in a physical node is greater than the average number of groups, it means that the load on the physical node is relatively large. In order to reduce the physical node’s Therefore, a suitable amount of basic data needs to be moved out to ensure the load balance of the physical nodes. At this time, it is necessary to determine the number of outgoing groups. The number of outgoing groups can be the difference between the group number of the basic data and the average value of the group number, or close to the difference, which is not limited here. In a specific embodiment, the number of basic data groups on the physical node is 5 groups, and the average number of groups is 3 groups, and the number of outgoing groups is 2. Understandably, by determining the number of migration groups of the basic data in each physical node, the basic data on each physical node can be further processed subsequently.

S30: For each physical node, determine the migration data in the basic data according to the number of migration groups.

Among them, the migration data is the basic data in one physical node that needs to be migrated to other physical nodes, and the number of groups of the basic data is the number of migration groups. Specifically, the migration data can be directly determined by random selection in the corresponding physical node, or can be determined according to the memory size of each basic data in the physical node, or the basic data in the physical node can be directly designated for determination. As a preference of this embodiment, the basic data of the migration array is selected from the migration array by random selection and determined to be the migration data. Since the basic data has a relatively close impact on the load performance of each physical node, the migration can be improved. The efficiency of data acquisition.

S40: Compare the number of groups of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest number of groups of basic data as the initial migration node.

Among them, the initial migration node refers to the candidate physical node used to determine the migration data. The initial migration node may be the target node for receiving the migration data, or it may be the candidate node for receiving the migration data. The basic data in the migration node is judged. Specifically, since the physical node with the smallest number of groups of basic data is the most idle, the load is the smallest, and can carry more load, the physical node with the smallest number of groups of basic data is used as the initial migration node, which avoids one by one. The redundant operation process of detecting each physical node is conducive to improving the efficiency of load balancing.

S50: Detect the initial migration node to determine whether the basic data in the initial migration node is the same. When the basic data in the initial migration node is the same, exclude the initial migration node to obtain the updated initial migration node, and Perform a step of comparing the number of groups of basic data corresponding to all physical nodes of the storage cluster, and taking the physical node with the smallest number of groups of basic data as the initial migration node.

Specifically, in order to ensure the reliability of the storage cluster, avoid the same basic data in the initial migration node, just as there are two or more copies of the basic data in an initial migration node. Therefore, it is necessary to determine whether the initial migration node is in advance. Set a restriction condition. The preset restriction condition is that the basic data in the initial migration node is different. When the basic data in the initial migration node is the same, the initial migration node is excluded, and the initial migration node that is not excluded is the updated Initially migrate in the node, and then execute step S50. Understandably, in the case that the initial migrated node does not meet the preset restriction conditions, the updated initial migrated node is the physical data group that is several times smaller (the second smallest) of the basic data group. If the next smallest physical node still does not meet the preset restriction conditions, the physical node needs to be excluded, and the initial migration node after continuing to update is the second smallest (third smallest) physical node of the basic data group. The minimum number of groups of basic data on the physical node that meets the preset restriction condition determines the initial migration node to perform cyclic processing until the initial migration node meets the preset restriction condition.

It should be noted that this step S50 will not loop indefinitely. It can be seen from step S10 that the average number of groups is calculated and determined according to the number of groups of the basic data of all physical nodes in the storage cluster. Therefore, each basic data has an initial migration node.

S60: When the basic data in the initial migration node is different, migrate the migration data into the initial migration node to achieve load balancing, wherein the preset restriction condition is that the basic data in the initial migration node is different.

Specifically, the preset restriction condition is that the basic data in the initial migration node is different. If the initial migration node meets the preset restriction condition, the initial migration node is the target node. Therefore, the migration data is migrated in. To the initial migration node, the load balancing of each physical node is realized, thereby ensuring the load balancing of the storage cluster.

It should be noted that, in order to further ensure the load balance of the storage cluster, it is possible to determine whether the difference between the number of basic data groups stored in the initial migration node and the number of basic data groups on the physical node corresponding to the migration data is greater than 1, if it is greater than 1, Then, migration is allowed, and after migration is completed, it should be ensured that the number of basic data groups of the initial migration node is less than or equal to the number of basic data groups on the physical node corresponding to the migration data. Understandably, when the initial migration node meets the preset restriction conditions, the migration data is migrated to the initial migration node, thereby achieving load balancing of the storage cluster and improving the performance of the storage cluster.

In this embodiment, when the load balancing instruction is received, the average number of groups of the basic data of all physical nodes in the storage cluster is calculated; then, according to the average number of groups and the number of groups of basic data in each physical node, each group is determined The number of outgoing groups of basic data in physical nodes, so that the basic data on each physical node can be further processed; then, for each physical node, the outgoing data is determined in the basic data according to the number of outgoing groups; next, From all physical nodes in the storage cluster, the physical node with the smallest group of basic data is used as the initial migration node; when the basic data of the initial migration node is the same, the initial migration node is excluded, and the updated initial migration node is obtained , And execute the step of obtaining the physical node with the smallest number of basic data groups from the physical nodes of the storage cluster as the initial migration node; when the basic data of the initial migration node is different, the migration data will be migrated to the initial migration node In order to achieve load balancing and improve the performance of the storage cluster.

In an embodiment, as shown in FIG. 2, before step S10, the following steps are further included.

S70: When deploying the storage cluster, select a preset number of virtual nodes according to the number of physical nodes included in the storage cluster, and perform N copies of the basic data on each physical node to obtain the corresponding virtual node And generate a mapping relationship table between virtual nodes and physical nodes, where N is a positive integer greater than 1.

S80: Distribute the basic data to the virtual nodes through a hash algorithm, and store the N groups of basic data mapped to the virtual nodes in the corresponding physical nodes through the mapping relationship table.

Among them, replica replication refers to the replication of basic data, specifically by synchronizing the basic data on multiple physical nodes. A virtual node refers to a pre-set virtual space, or it can be a hardware architecture. Specifically, the storage space in the storage cluster is divided into a preset number of virtual nodes according to the number of management nodes, and N copies are replicated in units of virtual nodes, so as to obtain N sets of basic data. The virtual node and the physical node are corresponding, that is, the mapping relationship between the virtual node and the physical node is 1:N. Specifically, the corresponding relationship between the virtual node and the physical node can be reflected by the mapping relationship table between the virtual node and the physical node. It is worth noting that in this embodiment, more basic data is involved, and the performance of the machine to be processed is limited. As a preferred method, N is 3 in this embodiment, that is, three copies. Understandably, for a three-copy storage cluster, The successful application of two copies of the three copies is the application success, so one copy is allowed to be broken; if it is a two copy storage cluster, more than half of the machines in the two copies are the integer two, so the application of both copies is successful. Bad machines are allowed; if it is a four-copy storage cluster, three of the four applications are successful, and only one bad copy is allowed; so, five copies allow two bad copies, six copies allow two bad copies, and seven copies allow bad applications Three copies and so on. Therefore, compared with even-numbered copies, odd-numbered copies save storage space and guarantee the same fault tolerance. Therefore, odd-numbered copies are preferred. Regarding the choice between three copies, five copies, and seven copies, the more copies, the longer the data synchronization time between the copies, and the slower the response to user requests. Therefore, choosing three copies can improve the reliability of basic data and the practicality of storage clusters. Performance, reducing the application cost of storage clusters.

Specifically, the basic data is distributed to the virtual nodes through a hash algorithm. In a specific embodiment, the basic data is divided into pieces, for example, a piece of 64M is used to number each piece of file, and the data storage address and the piece number are combined, and the MurmurHash (consistent hash) algorithm is used. A 64-bit character string is generated as the hash value of the fragment file, and the hash value is subjected to a remainder operation on the number of virtual nodes, that is, the preset number in step S10, so as to determine which virtual node the fragment file falls on. By looking up the mapping table, the fragmented data is stored in N (such as N=3) physical nodes, and by establishing virtual nodes and mapping the basic data to the physical nodes, the consistency of the basic data and the availability of the storage cluster are improved.

In this embodiment, when the storage cluster is deployed, a preset number of virtual nodes are selected according to the number of physical nodes included in the storage cluster, and N copies of the basic data on each physical node are replicated to obtain each N groups of basic data corresponding to the virtual node, and generate a mapping relationship table between virtual nodes and physical nodes; distribute the basic data to the virtual node through a hash algorithm, and map the N groups of basic data to the virtual node through the mapping relationship table Stored in the corresponding physical node, improving the consistency of basic data and the availability of storage clusters.

In an embodiment, as shown in FIG. 3, in step S80, the basic data is distributed to the virtual nodes through a hash algorithm, which specifically includes the following steps.

S81: Based on the preset memory size, perform fragmentation processing on the basic data to obtain fragmented data, and sequentially number each fragmented data.

S82: Perform sequential numbering on the virtual nodes, and for each piece of data, use the sequential number of the piece of data to perform a remainder operation on the number of virtual nodes to obtain the numbering modulus of the piece of data.

S83: Divide the fragment data with the same number modulus into the same fragment set, and save each fragment set on the virtual node corresponding to the sequence number consistent with the number modulus.

In this embodiment, first, a preset memory size, such as 64M, performs fragmentation processing on the basic data to obtain fragmented data, and sequentially number each fragmented data; then, sequentially number the virtual nodes, For each piece of data, use the sequential number of the piece of data to perform the remainder operation on the number of virtual nodes to obtain the number modulus of the piece of data; finally, divide the piece data with the same number mod into the same piece set , And save each shard set to the virtual node corresponding to the sequential number that is consistent with the numbering modulus, and the consistency of the basic data is improved through the consistent hash algorithm.

In one embodiment, as shown in FIG. 4, when the initial migration node meets the preset restriction conditions, the migration data is migrated to the initial migration node to achieve load balancing, and the following steps are further included .

S90: Determine the role labels of the N groups of basic data corresponding to each virtual node, the role labels include the leader role and the follower role.

S100: For each physical node, determine the migration threshold according to the ratio of the number of groups of basic data on the physical node and N.

S110: Perform role label conversion according to the number of groups of basic data whose role label is the leader role, the number of basic data groups whose role label is the follower role, and the migration threshold corresponding to each physical node.

Among them, the role tag is used to identify the IO (input/output) of the physical node in the storage cluster, the leader role is used to receive read and write requests, and the follower role is used to synchronize the basic data corresponding to the leader role to achieve data synchronization , Improve data security. Specifically, when the IO of any physical node in the storage cluster reaches a preset threshold, the IO balancing operation is automatically triggered. IO balance, reflected in the storage cluster, is the role label conversion. When no data is stored in the virtual node, the N groups of basic data of the virtual node compete with each other, and the basic data is elected from the N groups of basic data through the Raft algorithm as the leader role, and the role label of the remaining basic data is the follower Role. For N replicas of basic data, the leader role on each physical node is one-Nth of the number of basic data groups that carry it. This is used as the migration threshold. According to the role label, the number of groups and role label of the basic data of the leader role Perform role label conversion for the number of basic data groups of the follower role and the migration threshold corresponding to each physical node, and perform the migration of the basic data whose role label is the leader role, so that the IO on the physical node is evenly distributed, thereby further realizing the load balanced.

In one embodiment, as shown in FIG. 5, in step S110, according to the number of basic data groups whose role label is the leader role, the number of basic data groups whose role label is the follower role, and the migration threshold corresponding to each physical node , Perform role label conversion, specifically including the following steps.

S111: When the number of groups of basic data whose role label is the leader role is greater than the migration threshold, convert the role label that exceeds the migration threshold into the role label of the basic data of the leader role into a follower role, and set it on the basis of the physical node Select a basic data in the data and set its role label as the leader role.

S112: When the number of groups of basic data with the role label as the leader role is less than the migration threshold, transfer the basic data with the role label as the leader role to the physical node, and label the role in the physical node as the basis of the leader role The data is converted into a follower role until the number of groups of basic data whose role label is the leader role in the physical node meets the migration threshold.

In this embodiment, by comparing the number of groups of basic data with the role label as the leader role and the migration threshold, different role label conversions are performed, and the basic data is migrated according to the converted role labels, which ensures that the physical node IO is evenly distributed to achieve load balancing. Understandably, by migrating the basic data according to the role tag, the separation of the read and write requests of the physical node is avoided, that is, the user writes data to the machine 1, and then reads the data from the machine 1, thereby ensuring data consistency .

In an embodiment, as shown in FIG. 6, in step S90, the role tags of the N sets of basic data corresponding to each virtual node are determined. The role tags include the leader role and the follower role, which specifically includes the following steps.

S91: Send a voting request to the physical node in the storage cluster.

S92: Receive the voting result returned by the physical node corresponding to the voting request.

S93: According to the voting result, from the N groups of basic data of the virtual node corresponding to the physical node, the role label of one group of basic data is selected and determined as the leader role, and the role label of the N-1 group of basic data is determined as the follower role.

In this embodiment, the Raft algorithm is used to determine the role labels of the N sets of basic data corresponding to each virtual node according to the voting results corresponding to the voting request sent by the physical nodes in the storage cluster, and then the leader role is based on itself The data is synchronized to the basic data corresponding to the follower role, which can ensure that after the leader role is down, the other N-1 groups of basic data in the virtual node re-elect a leader role and N-2 follower roles, making the new The role of the leader is consistent with the basic data of the previous leader role, ensuring that data is not lost. And achieving IO balance can maximize the performance of all physical nodes in the storage cluster, and improve the load capacity and response speed of the entire storage cluster.

In one embodiment, as shown in FIG. 7, when the initial migration node meets the preset restriction conditions, the migration data is migrated to the initial migration node to achieve load balancing, and the following steps are further included .

S120: Traverse all physical nodes in the storage cluster.

S130: When there is no migration data in the physical node, it is determined that the load of the storage cluster reaches a balanced state.

In this embodiment, all physical nodes in the storage cluster are traversed. When all the basic data in the data to be migrated out cannot find the node to be initially migrated, it is determined that the load balancing is completed, thereby further ensuring the storage Cluster load balancing.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

In one embodiment, a load balancing device for a storage cluster is provided, and the load balancing device for the storage cluster corresponds to the load balancing method for the storage cluster in the foregoing embodiment one-to-one. As shown in FIG. 8, the load balancing device of the storage cluster includes an average value calculation module 10, a migration group number determination module 20, a migration data determination module 30, a migration node acquisition module 40, a migration node detection module 50, and load balancing. Module 60. The detailed description of each function module is as follows.

The average value calculation module 10 is configured to calculate the average value of the group number of the basic data of all physical nodes in the storage cluster when a load balancing instruction is received.

The migration group number determining module 20 is configured to determine the migration group number of the basic data in each physical node according to the average value of the group number and the group number of the basic data in each physical node.

The migration data determining module 30 is configured to determine the migration data in the basic data according to the number of migration groups for each physical node.

The migration node acquisition module 40 is configured to compare the number of groups of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest number of groups of basic data as the initial migration node.

The migration node detection module 50 is configured to detect the initial migration node and determine whether the basic data in the initial migration node is the same. When the basic data in the initial migration node is the same, exclude the Initial migration node, obtain the updated initial migration node, and perform the comparison of the number of groups of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest number of groups of basic data as Steps for initial migration into the node.

The load balancing module 60 is configured to migrate the migration data into the initial migration node when the basic data in the initial migration node is different, so as to achieve load balancing.

Specifically, the load balancing device of the storage cluster further includes a copy replication module and a data distribution module.

The copy replication module is used to select a preset number of virtual nodes according to the number of physical nodes included in the storage cluster when the storage cluster is deployed, and respectively perform N copies of the basic data on each physical node Copy to obtain N groups of basic data corresponding to each virtual node, and generate a mapping relationship table between virtual nodes and physical nodes, where N is a positive integer greater than 1.

The data distribution module is configured to distribute the basic data to the virtual node through a hash algorithm, and store the N groups of basic data mapped to the virtual node in the corresponding physical node through the mapping relationship table. node.

Specifically, the data distribution module includes a data slicing unit, a number modulus calculation unit, and a data storage unit.

The data fragmentation unit is configured to perform fragmentation processing on the basic data based on a preset memory size to obtain fragmented data, and sequentially number each fragmented data.

The numbering modulus calculation unit is configured to sequentially number the virtual nodes, and for each of the fragmented data, use the serial number of the fragmented data to perform a remainder operation on the number of the virtual nodes to obtain the value of the fragmented data Numbering mode.

The data storage unit is configured to divide the fragment data with the same serial number module into the same fragment set, and save each of the fragment sets on the virtual node corresponding to the sequence number consistent with the serial number modulus.

Specifically, the load balancing device of the storage cluster further includes a role determination module, a threshold determination module, and a role conversion module.

The role determination module is configured to determine the role tags of the N groups of the basic data corresponding to each virtual node, and the role tags include a leader role and a follower role.

The threshold determination module is configured to determine the migration threshold according to the ratio of the number of groups of basic data on the physical node and N for each physical node.

The role conversion module is configured to perform according to the number of groups of the basic data whose role label is a leader role, the number of groups of basic data whose role label is a follower role, and the migration threshold corresponding to each physical node Role label conversion.

Specifically, the role conversion module includes a first conversion unit and a second conversion unit.

The first conversion unit is configured to convert the role label that exceeds the migration threshold into the role label of the basic data of the leader role when the number of groups of the basic data whose role label is the leader role is greater than the migration threshold It is a follower role, and one basic data is selected from the basic data of the physical node and its role label is set as the leader role.

The second conversion unit is configured to transfer the basic data whose role is labeled as the leader role to the physical node when the number of groups of the basic data whose role is labeled as the leader role is less than the migration threshold. The basic data whose role label is the leader role in the physical node is converted into the follower role until the number of groups of the basic data whose role label is the leader role in the physical node meets the migration threshold.

Specifically, the role determination module includes a request sending unit, a result receiving unit, and a role determination unit.

The request sending unit is configured to send a voting request to the physical nodes in the storage cluster.

The result receiving unit is configured to receive the voting result returned by the physical node corresponding to the voting request.

The role determination unit is configured to select, from the N sets of basic data of the virtual node corresponding to the physical node, the role label of one set of basic data to determine the role of the leader according to the voting result, and the N-1 sets of basic data The role tag is determined as the follower role.

Specifically, the load balancing device of the storage cluster further includes a node traversal module and a load balancing module.

The node traversal module is used to traverse all physical nodes in the storage cluster.

The load balancing module is configured to determine that the load of the storage cluster reaches a balanced state when there is no migration data in the physical node.

For the specific limitation of the load balancing device of the storage cluster, please refer to the above limitation of the load balancing method of the storage cluster, which will not be repeated here. The various modules in the load balancing device of the storage cluster can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 9. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The computer equipment database is used to store physical nodes and basic data. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to realize a load balancing method of the storage cluster.

In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor executes the computer program to implement the load balancing method of the storage cluster in the foregoing embodiment. , For example, step S10 to step S60 shown in FIG. 1. Or, when the processor executes the computer program, the function of each module/unit of the load balancing apparatus of the storage cluster in the foregoing embodiment is realized, for example, the module 10 to the module 60 shown in FIG. 8. To avoid repetition, I won’t repeat them here.

In one embodiment, a computer-readable storage medium is provided, and a computer program is stored thereon. When the computer program is executed by a processor, the steps of the load balancing method of the storage cluster in the foregoing embodiment are implemented, or the computer program is executed by the processor. The functions of the modules/units of the load balancing device of the storage cluster of the foregoing embodiment are realized during execution. In order to avoid repetition, details are not described herein again.

Optionally, the storage medium involved in this application, such as a computer-readable storage medium, may be non-volatile or volatile.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage. In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed. Module completion, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

The above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still compare the previous embodiments. The recorded technical solutions are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and shall be included in the application Within the scope of protection.

Claims (20)

1. A load balancing method for a storage cluster, wherein the storage cluster includes a plurality of physical nodes, and each physical node contains multiple sets of basic data, and the load balancing method for the storage cluster includes:

   When a load balancing instruction is received, calculate the average number of groups of the basic data of all physical nodes in the storage cluster;

   Determine the number of migration groups of the basic data in each physical node according to the average value of the number of groups and the number of groups of the basic data in each physical node;

   For each physical node, determine the migration data in the basic data according to the number of migration groups;

   Comparing the number of groups of basic data corresponding to all physical nodes of the storage cluster, and using the physical node with the smallest number of groups of basic data as the initial migration node;

   The initial migration node is detected to determine whether the basic data in the initial migration node is the same. When the basic data in the initial migration node is the same, the initial migration node is excluded, and the updated Initially migrate the node, and execute the step of comparing the group numbers of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest group number of basic data as the initial migration node;

   When the basic data in the initial migration node is different, the migration data is migrated to the initial migration node to achieve load balancing.
2. The load balancing method of a storage cluster according to claim 1, wherein the method further comprises:

   When the storage cluster is deployed, a preset number of virtual nodes are selected according to the number of physical nodes included in the storage cluster, and N copies of the basic data on each physical node are replicated to obtain each virtual node. N sets of basic data corresponding to the node, and generate a mapping relationship table between virtual nodes and physical nodes, where N is a positive integer greater than 1;

   The basic data is allocated to the virtual node through a hash algorithm, and the N groups of basic data mapped to the virtual node are stored in a corresponding physical node through the mapping relationship table.
3. 3. The load balancing method of a storage cluster according to claim 2, wherein the distributing the basic data to the virtual node through a hash algorithm comprises:

   Based on a preset memory size, perform fragmentation processing on the basic data to obtain fragmented data, and sequentially number each fragmented data;

   Sequentially number the virtual nodes, and for each of the fragmented data, use the serial number of the fragmented data to perform a remainder operation on the number of the virtual nodes to obtain the numbering modulus of the fragmented data;

   The segment data with the same numbering modulus is divided into the same segmentation set, and each of the segmentation sets is saved on the virtual node corresponding to the sequence number consistent with the numbering model.
4. 3. The load balancing method of a storage cluster according to claim 2, wherein when the initial migration-in node meets a preset restriction condition, the migration data is migrated to the initial migration-in node, After achieving load balancing, it also includes:

   Determining the role tags of the N groups of the basic data corresponding to each virtual node, where the role tags include a leader role and a follower role;

   For each of the physical nodes, the migration threshold is determined according to the ratio of the number of groups of basic data on the physical node and N;

   Perform role label conversion according to the number of groups of the basic data whose role label is a leader role, the number of basic data groups whose role label is a follower role, and the migration threshold corresponding to each of the physical nodes.
5. The load balancing method of a storage cluster according to claim 4, wherein the number of groups of the basic data whose role label is a leader role, the number of groups of basic data whose role label is a follower role, and each Performing role label conversion on the migration threshold corresponding to the physical node includes:

   When the number of groups of the basic data whose role label is the leader role is greater than the migration threshold, the role label that exceeds the migration threshold is converted to the role label of the basic data of the leader role into a follower role, and Selecting one of the basic data of the physical node and setting its role tag as the leader role;

   When the number of groups of the basic data whose role label is the leader role is less than the migration threshold, the basic data whose role label is the leader role is transferred to the physical node, and the role label in the physical node is the leader The basic data of the leader role is converted into the follower role until the number of groups of the basic data whose role label is the leader role in the physical node meets the migration threshold.
6. 5. The load balancing method of a storage cluster according to claim 4, wherein said determining the role tags of the N groups of the basic data corresponding to each virtual node, the role tags including a leader role and a follower role, include:

   Sending a voting request to the physical nodes in the storage cluster;

   Receiving the voting result returned by the physical node corresponding to the voting request;

   According to the voting result, from the N sets of basic data of the virtual node corresponding to the physical node, the role label of one set of basic data is determined to be the leader role, and the role label of the N-1 set of basic data is determined to be the follower Role.
7. The load balancing method of a storage cluster according to claim 1, wherein, when the initial migration-in node satisfies a preset restriction condition, the migration data is migrated to the initial migration-in node , To achieve load balancing, it also includes:

   Traverse all physical nodes in the storage cluster;

   When there is no migration data in the physical node, it is determined that the load of the storage cluster reaches a balanced state.
8. A load balancing device for a storage cluster. The storage cluster includes a plurality of physical nodes, and each physical node contains multiple sets of basic data. The load balancing device for the storage cluster includes:

   An average value calculation module, configured to calculate an average value of the group number of the basic data of all physical nodes in the storage cluster when a load balancing instruction is received;

   A migration group number determination module, configured to determine the migration group number of the basic data in each physical node according to the average value of the group number and the group number of the basic data in each physical node;

   A migration data determination module, configured to determine migration data in the basic data according to the number of migration groups for each physical node;

   A migration node acquisition module, configured to compare the number of groups of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest number of groups of basic data as the initial migration node;

   The migration node detection module is used to detect the initial migration node and determine whether the basic data in the initial migration node is the same. When the basic data in the initial migration node is the same, exclude the initial migration node. Migrate the node to obtain the updated initial migration node, and perform the comparison of the number of groups of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest number of groups of basic data as the initial Steps to move into the node;

   The load balancing module is configured to migrate the migration data into the initial migration node when the basic data in the initial migration node is different, so as to achieve load balancing, wherein the preset restriction condition The basic data in the initial migration node is different.
9. A computer device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the following steps when the processor executes the computer program:

   When a load balancing instruction is received, calculating an average value of the number of groups of basic data of all physical nodes in the storage cluster, the storage cluster includes multiple physical nodes, and each physical node contains multiple sets of basic data;

   Determine the number of migration groups of the basic data in each physical node according to the average value of the number of groups and the number of groups of the basic data in each physical node;

   For each physical node, determine the migration data in the basic data according to the number of migration groups;

   Comparing the number of groups of basic data corresponding to all physical nodes of the storage cluster, and using the physical node with the smallest number of groups of basic data as the initial migration node;

   The initial migration node is detected to determine whether the basic data in the initial migration node is the same. When the basic data in the initial migration node is the same, the initial migration node is excluded, and the updated Initially migrate the node, and execute the step of comparing the group numbers of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest group number of basic data as the initial migration node;

   When the basic data in the initial migration-in node is different, the migration-out data is migrated to the initial migration-in node.
10. 9. The computer device according to claim 9, wherein the processor further implements the following steps when executing the computer program:

    When the storage cluster is deployed, a preset number of virtual nodes are selected according to the number of physical nodes included in the storage cluster, and N copies of the basic data on each physical node are replicated to obtain each virtual node. N sets of basic data corresponding to the node, and generate a mapping relationship table between virtual nodes and physical nodes, where N is a positive integer greater than 1;

    The basic data is allocated to the virtual node through a hash algorithm, and the N groups of basic data mapped to the virtual node are stored in a corresponding physical node through the mapping relationship table.
11. 10. The computer device according to claim 10, wherein when the basic data is allocated to the virtual node through a hash algorithm, the following steps are specifically implemented:

    Based on a preset memory size, perform fragmentation processing on the basic data to obtain fragmented data, and sequentially number each fragmented data;

    Sequentially number the virtual nodes, and for each of the fragmented data, use the serial number of the fragmented data to perform a remainder operation on the number of the virtual nodes to obtain the numbering modulus of the fragmented data;

    The segment data with the same numbering modulus is divided into the same segmentation set, and each of the segmentation sets is saved on the virtual node corresponding to the sequence number consistent with the numbering model.
12. The computer device according to claim 10, wherein, when the initial migration-in node satisfies a preset restriction condition, the migration-out data is migrated to the initial migration-in node to achieve load balancing After that, the processor further implements the following steps when executing the computer program:

    Determining the role tags of the N groups of the basic data corresponding to each virtual node, where the role tags include a leader role and a follower role;

    For each of the physical nodes, the migration threshold is determined according to the ratio of the number of groups of basic data on the physical node and N;

    Perform role label conversion according to the number of groups of the basic data whose role label is a leader role, the number of basic data groups whose role label is a follower role, and the migration threshold corresponding to each of the physical nodes.
13. The computer device according to claim 12, wherein the number of groups of the basic data whose role label is a leader role, the number of groups of basic data whose role label is a follower role, and each of the physical nodes Corresponding to the migration threshold, when performing role label conversion, the following steps are specifically implemented:

    When the number of groups of the basic data whose role label is the leader role is greater than the migration threshold, the role label that exceeds the migration threshold is converted to the role label of the basic data of the leader role into a follower role, and Selecting one of the basic data of the physical node and setting its role tag as the leader role;

    When the number of groups of the basic data whose role label is the leader role is less than the migration threshold, the basic data whose role label is the leader role is transferred to the physical node, and the role label in the physical node is the leader The basic data of the leader role is converted into the follower role until the number of groups of the basic data whose role label is the leader role in the physical node meets the migration threshold.
14. The computer device according to claim 12, wherein when the role tags of the N sets of the basic data corresponding to each virtual node are determined, and the role tags include a leader role and a follower role, the following is specifically implemented step:

    Sending a voting request to the physical nodes in the storage cluster;

    Receiving the voting result returned by the physical node corresponding to the voting request;

    According to the voting result, from the N sets of basic data of the virtual node corresponding to the physical node, the role label of one set of basic data is determined to be the leader role, and the role label of the N-1 set of basic data is determined to be the follower Role.
15. The computer device according to claim 9, wherein, in the case that the initial migration-in node satisfies a preset restriction condition, the migration-out data is migrated to the initial migration-in node to realize the load After the equalization, the processor further implements the following steps when executing the computer program:

    Traverse all physical nodes in the storage cluster;

    When there is no migration data in the physical node, it is determined that the load of the storage cluster reaches a balanced state.
16. A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the following steps:

    When a load balancing instruction is received, calculating an average value of the number of groups of basic data of all physical nodes in the storage cluster, the storage cluster includes multiple physical nodes, and each physical node contains multiple sets of basic data;

    Determine the number of migration groups of the basic data in each physical node according to the average value of the number of groups and the number of groups of the basic data in each physical node;

    For each physical node, determine the migration data in the basic data according to the number of migration groups;

    Comparing the number of groups of basic data corresponding to all physical nodes of the storage cluster, and using the physical node with the smallest number of groups of basic data as the initial migration node;

    The initial migration node is detected to determine whether the basic data in the initial migration node is the same. When the basic data in the initial migration node is the same, the initial migration node is excluded, and the updated Initially migrate the node, and execute the step of comparing the group numbers of basic data corresponding to all physical nodes of the storage cluster, and use the physical node with the smallest group number of basic data as the initial migration node;

    When the basic data in the initial migration-in node is different, the migration-out data is migrated to the initial migration-in node.
17. 15. The computer-readable storage medium of claim 16, wherein the computer program further implements the following steps when being executed by the processor:

    When the storage cluster is deployed, a preset number of virtual nodes are selected according to the number of physical nodes included in the storage cluster, and N copies of the basic data on each physical node are replicated to obtain each virtual node. N sets of basic data corresponding to the node, and generate a mapping relationship table between virtual nodes and physical nodes, where N is a positive integer greater than 1;

    The basic data is allocated to the virtual node through a hash algorithm, and the N groups of basic data mapped to the virtual node are stored in a corresponding physical node through the mapping relationship table.
18. 17. The computer-readable storage medium of claim 17, wherein when the basic data is distributed to the virtual node through a hash algorithm, the following steps are specifically implemented:

    Based on a preset memory size, perform fragmentation processing on the basic data to obtain fragmented data, and sequentially number each fragmented data;

    Sequentially number the virtual nodes, and for each of the fragmented data, use the serial number of the fragmented data to perform a remainder operation on the number of the virtual nodes to obtain the numbering modulus of the fragmented data;

    The segment data with the same numbering modulus is divided into the same segmentation set, and each of the segmentation sets is saved on the virtual node corresponding to the sequence number consistent with the numbering model.
19. The computer-readable storage medium according to claim 17, wherein when the initial migration-in node meets a preset restriction condition, the migration-out data is migrated to the initial migration-in node to After the load balancing is implemented, the following steps are also implemented when the computer program is executed by the processor:

    Determining the role tags of the N groups of the basic data corresponding to each virtual node, where the role tags include a leader role and a follower role;

    For each of the physical nodes, the migration threshold is determined according to the ratio of the number of groups of basic data on the physical node and N;

    Perform role label conversion according to the number of groups of the basic data whose role label is a leader role, the number of basic data groups whose role label is a follower role, and the migration threshold corresponding to each of the physical nodes.
20. The computer-readable storage medium according to claim 19, wherein the number of groups of the basic data whose role label is a leader role, the number of groups of basic data whose role label is a follower role, and each When performing role label conversion for the migration threshold corresponding to the physical node, the following steps are specifically implemented:

    When the number of groups of the basic data whose role label is the leader role is greater than the migration threshold, the role label that exceeds the migration threshold is converted to the role label of the basic data of the leader role into a follower role, and Selecting one of the basic data of the physical node and setting its role tag as the leader role;

    When the number of groups of the basic data whose role label is the leader role is less than the migration threshold, the basic data whose role label is the leader role is transferred to the physical node, and the role label in the physical node is the leader The basic data of the leader role is converted into the follower role until the number of groups of the basic data whose role label is the leader role in the physical node meets the migration threshold.