CN113553179A - Distributed key value storage load balancing method and system - Google Patents
Distributed key value storage load balancing method and system Download PDFInfo
- Publication number
- CN113553179A CN113553179A CN202110806889.2A CN202110806889A CN113553179A CN 113553179 A CN113553179 A CN 113553179A CN 202110806889 A CN202110806889 A CN 202110806889A CN 113553179 A CN113553179 A CN 113553179A
- Authority
- CN
- China
- Prior art keywords
- current carrier
- load balancing
- carrier node
- node
- distributed key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000013508 migration Methods 0.000 claims description 22
- 230000005012 migration Effects 0.000 claims description 22
- 239000012634 fragment Substances 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 10
- 238000013500 data storage Methods 0.000 abstract description 10
- 230000008569 process Effects 0.000 abstract description 10
- 238000013467 fragmentation Methods 0.000 abstract description 8
- 238000006062 fragmentation reaction Methods 0.000 abstract description 8
- 238000004364 calculation method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a distributed key value storage load balancing method and a distributed key value storage load balancing system, wherein the method comprises the following steps: receiving a load state of a current Carrier node sent by a current Carrier node through heartbeat in a distributed key value storage cluster system; judging whether to perform load balancing according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for adjustment of the current Carrier node when determining to perform load balancing; and updating the current Carrier node according to the adjusted execution result sent by the current Carrier node. The system is used for executing the method. The invention can realize the balance of data storage and data access in the distributed key value storage system, and can effectively prevent the jitter of service data access in the scheduling process of data fragmentation, thereby ensuring the stability and high availability of the system.
Description
Technical Field
The invention relates to the technical field of computer distributed databases, in particular to a distributed key value storage load balancing method and system.
Background
In a distributed key-value storage system, in order to support efficient access to data and an increasing amount of data storage, it is necessary to distribute data to a plurality of nodes by using a fragmented storage technique.
At present, most distributed databases realize load balancing by simply migrating data fragments, and often cause jitter of access to service data in the migration process and generate large overhead. For a system that uses a coherence protocol for backup, migrating some fragments at will may also result in the coherence protocol not working properly for a long time.
Therefore, how to guarantee the stability and high availability of the system while performing load balancing in the distributed consistent storage system becomes a problem which needs to be solved urgently at present.
Disclosure of Invention
The distributed key value storage load balancing method and system provided by the invention are used for solving the problems in the prior art, can realize the balance of data storage and data access in a distributed consistent storage system (such as a distributed key value storage system), can effectively prevent the jitter of service data access in the scheduling process of data fragmentation, and ensure the stability and high availability of the distributed key value storage system.
The invention provides a distributed key value storage load balancing method, which comprises the following steps:
receiving a load state of a current Carrier node sent by a current Carrier node through heartbeat in a distributed key value storage system;
judging whether to perform load balancing according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for the current Carrier node to adjust when determining to perform load balancing;
and updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
According to the load balancing method for distributed key value storage provided by the invention, the load state of the current Carrier node comprises the following steps:
configuring information, resource occupation condition and duplicate status of logic fragment Tablet;
the configuration information at least comprises the number of CPU cores and the size of a memory;
the resource occupation condition at least comprises a CPU utilization rate and a memory utilization rate;
the Replica state at least comprises a logic fragment identifier tableid, a copy identifier Replica Id, an Instance identifier InstanceId, a query quantity per second QPS in writing operation, a logic fragment Version tableset Version, a query quantity per second Read QPS in reading operation and a Consensus Group Consenses Group;
the Tablet Version, Read QPS and Consenssus Group are the status that the Replica is the master node.
According to the distributed key value storage load balancing method provided by the invention, whether load balancing is carried out or not is judged according to the load state of the current Carrier node, and when the load balancing is determined, load balancing operation is sent to the current Carrier node so as to be adjusted by the current Carrier node, the method comprises the following steps:
updating the load state of the distributed key value storage system according to the load state of the current Carrier node;
judging whether the load state of the current Carrier node meets a first preset condition or not according to the updated load state of the distributed key value storage system;
if the first preset condition is not met, determining to perform load balancing, selecting a first target Replica from Replicas of the current Carrier node to perform main node switching operation, and updating the load state of the current Carrier node;
if the load state of the current Carrier node after the main node switching operation does not meet the first preset condition, performing migration operation of a second target Replica;
and determining the load balancing operation according to the main node switching operation and the migration operation, and sending the load balancing operation to the current Carrier node for adjustment of the current Carrier node.
According to the distributed key value storage load balancing method provided by the invention, the first preset condition comprises the following steps:
the CPU utilization rate and the memory utilization rate of the current Carrier node are lower than a first preset threshold value; or
The QPS load of each CPU core of the current Carrier node is lower than the preset average QPS; or
The current Carrier node has the load balancing operation not completed.
According to the distributed key value storage load balancing method provided by the invention, the step of selecting the first target Replica from the Replicas of the current Carrier node to carry out the main node switching operation comprises the following steps:
selecting a Replica meeting a second preset condition from replias of the main node of the current Carrier node as the first target Replica to perform main node switching operation;
wherein the second preset condition includes that a Replica exists in the Consensus Group of the Tablet.
According to the distributed key value storage load balancing method provided by the invention, the migration operation of the second target Replica comprises the following steps:
adding the second target Replica on the Carrier node with the lowest load and deleting the second target Replica on the current Carrier node;
wherein the second target Replica is determined according to a Replica satisfying a third preset condition selected from Replicas of slave nodes of the current Carrier node;
the third preset condition includes that the migration operation or the splitting operation is not performed within the preset time of the owned Tablet, and the difference value of the Instance Ids of all the replias in the Consensus Group of the owned Tablet does not exceed a second preset threshold.
According to the distributed key value storage load balancing method provided by the invention, the updating of the current Carrier node according to the adjusted execution result sent by the current Carrier node comprises the following steps:
updating the load balancing operation queues of the current Carrier node and the target Tablet according to the execution result sent by the current Carrier node after the load balancing operation is executed;
wherein the load balancing operation is sent by a Remote Procedure Call (RPC) request;
the load balancing operation queue is determined after the load balancing operation is recorded;
the target Tablet is determined according to the Tablet to which the first target Replica and/or the second target Replica belong.
The invention also provides a distributed key value storage load balancing system, which comprises: the device comprises a receiving module, an adjusting module and an updating module;
the receiving module is used for receiving the load state of the current Carrier node sent by the current Carrier node through heartbeat in the distributed key value storage system;
the adjusting module is used for judging whether load balancing is carried out or not according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for adjusting the current Carrier node when the load balancing is determined;
and the updating module is used for updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the program to realize the steps of any one of the distributed key value storage load balancing methods.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the distributed key-value store load balancing method as described in any of the above.
The distributed key value storage load balancing method and the distributed key value storage load balancing system provided by the invention can realize the balance of data storage and data access in a distributed consistent storage system (such as the distributed key value storage system), can effectively prevent the jitter of service data access in the scheduling process of data fragmentation, and ensure the stability and high availability of the distributed key value storage system.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a distributed key value storage load balancing method provided in the present invention;
FIG. 2 is a schematic structural diagram of a distributed key value storage load balancing system according to the present invention;
fig. 3 is a schematic physical structure diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow diagram of a distributed key value storage load balancing method provided by the present invention, and as shown in fig. 1, the method includes:
s1, receiving the load state of the current Carrier node sent by the current Carrier node through heartbeat in the distributed key value storage system;
s2, judging whether to carry out load balancing according to the load state of the current Carrier node, and sending the load balancing operation to the current Carrier node for the current Carrier node to adjust when determining to carry out load balancing;
and S3, updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
It should be noted that the execution subject of the method may be a computer device, and the following describes in detail the distributed key value storage load balancing method provided by the present invention by taking an example in which the computer device deploys a distributed key value storage system (cluster):
optionally, the distributed key value storage system is deployed with a Carrier node and an arbitration Arbiter node, the china data storage of the distributed key value storage system takes logic partitioned tablets as a unit, each Tablet has 3 copies of a backup, and the synchronization is performed through a paxos consistency protocol, the read operation only occurs on the primary Leader node of the Replica, and the write operation is synchronized to 3 replias through the paxos. The Replica is distributed to a plurality of Carrier nodes. The Arbiter node is a management center of the cluster and is responsible for load balancing scheduling of the Replica.
The Arbiter node receives the load state of the current Carrier node reported by the current Carrier node through heartbeat; the Arbiter node judges whether load balancing is needed according to the received load state of the current Carrier node, if so, proper adjustment operation, namely load balancing operation, is calculated, and the load balancing operation is issued to the corresponding Carrier node, the Carrier node adjusts the related duplicate Replica after receiving the load balancing operation, and an execution result is fed back to the Arbiter node; and the Arbiter node updates the current Carrier node according to the adjusted execution result sent by the current Carrier node.
The distributed key value storage load balancing method provided by the invention can realize the balance of data storage and data access in a distributed consistent storage system (such as a distributed key value storage system), can effectively prevent the jitter of service data access in the scheduling process of data fragmentation, and ensures the stability and high availability of the distributed key value storage system.
Further, in an embodiment, the load status of the current Carrier node in step S1 may specifically include:
configuring information, resource occupation condition and duplicate status of logic fragment Tablet;
the configuration information at least comprises the number of CPU cores and the size of a memory;
the resource occupation condition at least comprises a CPU utilization rate and a memory utilization rate;
the Replica state at least comprises a logic fragment identifier tableid, a copy identifier Replica Id, an Instance identifier InstanceId, a query quantity per second in Write operation Write QPS, a logic fragment Version Tablet Version, a query quantity per second in Read operation Read QPS and a Consensus Group;
tablet Version, Read QPS and Consenssus Group are the Replica states that Replica is the master node.
Optionally, the Carrier reports the load state of the current Carrier node to the Arbiter node through heartbeat, and reports the configuration information and the resource occupation condition of the current Carrier node to the Arbiter node.
Specifically, the configuration information of the Carrier node includes the number of CPU cores and the size of memory, and the resource occupation condition includes the CPU utilization rate and the memory utilization rate of the Carrier node.
In addition, if there is a logical fragment copy Tablet replay on the current Carrier node, the states of the replay need to be reported to the Arbiter node together.
Specifically, the Replica state includes a logical tile identifier, TableId, a copy identifier, Replica Id, an Instance identifier, InstanceId, and a query quantity per second, Write QPS, in a Write operation. If the Replica is the main Leader node of the consistency protocol, the Replica state also comprises a logic fragment Version, a query quantity per second Read QPS in Read operation and Consensus Group information.
The Carrier node detects the load state of the current node by using a periodic task, and firstly obtains the CPU core number, the memory size, the CPU utilization rate and the memory utilization rate of the current node. Then, traversing the Replica existing on the current Carrier node, acquiring the Tablet Id, the Replica Id, the Instance Id and the Write QPS of each Replica, detecting whether the Replica is a Leader node, and if the Replica is the Leader node of the consistency protocol, acquiring the Tablet Version, the Read QPS and the Consensus Group information. After the detection is finished, the load state of the current Carrier node is serialized by Protobuf and then sent to the Arbiter node through a heartbeat RPC.
The distributed key value storage load balancing method comprehensively considers the resource utilization condition of the distributed key value storage system, the Replica read-write load state and the consistency protocol state, can realize rapid and effective load balancing, reduces the jitter of service data access in the load balancing process as much as possible, and ensures the high efficiency and the high availability of the system.
Further, in an embodiment, the step S2 may specifically include:
s21, updating the load state of the distributed key value storage system according to the load state of the current Carrier node;
s22, judging whether the load state of the current Carrier node meets a first preset condition or not according to the updated load state of the distributed key value storage system;
s23, if the first preset condition is not met, determining to perform load balancing, selecting a first target Replica from Replicas of the current Carrier node to perform main node switching operation, and updating the load state of the current Carrier node;
s24, if the load state of the current Carrier node after the main node switching operation still does not meet the first preset condition, performing migration operation of a second target Replica;
and S25, determining load balancing operation according to the main node switching operation and the migration operation, and sending the load balancing operation to the current Carrier node for adjustment of the current Carrier node.
Further, in an embodiment, the first preset condition may specifically include:
the CPU utilization rate and the memory utilization rate of the current Carrier node are lower than a first preset threshold value; or
The QPS load of each CPU core of the current Carrier node is lower than the preset average QPS; or
There are incomplete load balancing operations for current Carrier nodes.
Optionally, the Arbiter node updates the load state of the distributed key value storage cluster according to the load state of the current Carrier node sent by the Carrier node through heartbeat. Specifically, the QPS load of each CPU core of the reply QPS and Write QPS and Carrier node is updated, and the average QPS load of each CPU core of the distributed key-value storage cluster is updated. Wherein the QPS load of the Carrier node is the sum of Read QPS and Write QPS of all replicas on the Carrier node, and the QPS load of each CPU core of the Carrier node is the QPS load of Carrier/number of CPU cores of Carrier. The average QPS load of each CPU core of the distributed key value storage cluster is the sum of QPS loads of all Carrier nodes/the sum of CPU cores of all carriers.
The Arbiter node judges whether the load of the current Carrier node is too high, and judges whether load balancing is needed according to the load state of the updated distributed key value storage system, specifically:
if the system CPU utilization rate and the memory utilization rate of the current Carrier node are lower than the threshold (namely a first preset threshold) set by the distributed key value storage system, or the QPS load of each CPU core of the current Carrier node is lower than the preset average QPS of each CPU core of the cluster, or the current Carrier node has incomplete load balancing operation, the current Carrier node is considered not to need to carry out load balancing, the processing is skipped, otherwise the load of the current Carrier node is considered to be too high, and the load balancing operation required by asynchronous task calculation is started. The preset average QPS is the average QPS (1+ upper limit of variation of the distributed key value storage system configuration).
The upper limit of the variation of the distributed key value storage system configuration represents an acceptable fluctuation ratio, which is usually set to 25%, and if the fluctuation ratio exceeds the acceptable fluctuation ratio, the cluster is considered to have a load skew, and if the fluctuation ratio exceeds the acceptable fluctuation ratio, the threshold of the CPU utilization rate and the memory utilization rate is usually set to 75%, and if the fluctuation ratio is lower than the acceptable fluctuation ratio, the cluster is considered to have no need to be processed even if the cluster has a load skew. The purpose of the asynchronous task is to ensure that the heartbeat of the Carrier node is not blocked.
If the load of the current Carrier node is too high, attempting to perform Leader node switching operation, specifically: in order to reduce the influence of the state change of the distributed key value storage system caused by the heartbeat of other Carrier nodes on the calculation of load balancing operation, a load snapshot of the distributed key value storage system is obtained before the calculation is started, and the subsequent calculation is carried out based on the state of the snapshot.
And selecting a proper Replica (namely a first target Replica) from the Leader node Replica on the current Carrier node to switch the Leader node, and updating the QPS of the current Carrier node and the QPS of the Carrier node where the first target Replica is located.
If no appropriate Leader can be switched, or the load of the Carrier node is still too high after switching (i.e. the Carrier node after switching still does not meet the first preset condition), then the migration operation on the second target Replica is attempted.
The Arbiter node determines load balancing operation according to the main node switching operation and the migration operation, issues a load balancing operation instruction to the current Carrier, and the current Carrier node adjusts the related Replica according to the received load balancing operation instruction.
The distributed key value storage load balancing method provided by the invention can reduce the service data access jitter caused by load balancing through Replica switching and transferring operations, and simultaneously avoids the occurrence of long-time abnormal work of the distributed key value storage system caused by random transfer of certain allocations.
Further, in an embodiment, the step S23 may specifically include:
s231, selecting a Replica meeting a second preset condition from replicas of the main node of the current Carrier node as a first target Replica to perform main node switching operation;
wherein the second preset condition includes that a Replica exists in the Consensus Group of the Tablet.
Optionally, a Replica meeting a second preset condition is selected from replias of the host node of the current Carrier node to serve as a first target Replica to perform host node switching operation, the Replica meeting the second preset condition is a replia existing in a Consensus Group of a Tablet to which the Replica belongs, and after the Leader node is switched from the Replica in the current Carrier node to the Replica, the Carrier node where the Replica belongs does not have an excessively high load.
Meanwhile, in order to reduce the influence of Leader node switching on a Read hot spot, the Leader node replicas are sequenced according to Read QPS, and the Leader node replicas are selected from the middle to two sides until the QPS load of each core of the current Carrier node is reduced to be lower than the preset average QPS of each CPU core of the cluster or the selected number of the replicas exceeds the single switchable Leader node number threshold set by the system.
The distributed key value storage load balancing method provided by the invention can ensure the balance of data storage and data access in the distributed key value storage system by adopting Replica switching, and effectively prevent the jitter of service data access in the scheduling process of data fragmentation.
Further, in an embodiment, the step S24 may specifically include:
s241, adding a second target Replica on the Carrier node with the lowest load and deleting the second target Replica on the current Carrier node;
wherein the second target Replica is determined according to a Replica which satisfies a third preset condition selected from Replicas of slave nodes of the current Carrier node;
the third preset condition includes that the migration operation or the splitting operation is not performed within the preset time of the owned Tablet, and the difference value of the Instance Ids of all the replias in the Consensus Group of the owned Tablet does not exceed the second preset threshold.
Optionally, a replia (i.e. a second target replia) meeting a third preset condition is selected from the replicas of the Follower nodes on the current Carrier node for the migration operation.
Specifically, the third preset condition means that migration or splitting operation is not performed within a preset time of the Tablet to which the Replica belongs, and the difference value of the instant ids of all replias in the Consensus Group does not exceed the second preset threshold.
For the consistency protocol, if the Replica is not synchronized, the new Replica is in a learning state after a new Replica is added, and there are 2 laggard replicas simultaneously under the condition that the default Replica is 3, the consistency protocol cannot exceed half of the proposed solutions to pass, and a situation that the Tablet has no Leader node occurs, so that the system is not usable for a long time, and therefore, the appropriate Replica requires that the Tablet to which the Replica belongs does not perform migration or splitting (split) operation within a certain time, and the difference value of the instant ids of all the replias in the sensussroup does not exceed a second preset threshold.
Specifically, the replias of the following Follower node on the current Carrier node may be sorted by Write QPS, starting with the highest replia of Write QPS.
The Leader Replica can not be directly migrated, the Leader must be switched first, and the migratable Replica can only be a Follower node, so that the Follower Replica on the current Carrier node is sorted according to the Write QPS, and the highest Replica in the Write QPS is selected.
The migration operation of the second target Replica comprises two operations of adding one second target Replica on the Carrier node with the lowest load and deleting the second target Replica on the current Carrier node.
Specifically, the Carrier node of the replia already existing in the Consensus Group is excluded. Sorting the available Carrier nodes according to the QPS load of each core, selecting the Carrier nodes with the lowest load from the Carrier nodes with the lowest load, calculating the load condition of the Carrier nodes after the second target Replica is added, and if the QPS load of each core of the Carrier nodes after the second target Replica is added is still lower than the average QPS of each CPU core of the cluster (1+ the upper limit of the change of the system configuration), considering that the migration is possible and updating the load state of the current Carrier node. Until the QPS load of each core of the current Carrier node falls below the average QPS of each CPU core of the cluster (1+ upper limit of variation of system configuration) or the selected number of Replicas exceeds the single migratable number threshold set by the system.
If the migration is possible, the Write QPS to be migrated is subtracted from the total QPS for the current Carrier node and added to the current Carrier node for subsequent calculation.
The distributed key value storage load balancing method provided by the invention can ensure the balance of data storage and data access in the distributed key value storage system by adopting the Replica migration operation, and effectively prevent the jitter of service data access in the scheduling process of data fragmentation.
Further, in an embodiment, the step S3 may specifically include:
s31, updating the load balancing operation queues of the current Carrier node and the target Tablet according to the execution result sent by the current Carrier node after the load balancing operation is executed;
wherein, the load balancing operation is sent by Remote Procedure Call (RPC) request;
the load balancing operation queue is determined after the load balancing operation is recorded;
the target Tablet is determined according to the Tablet to which the first target Replica and/or the second target Replica belong.
Optionally, the Arbiter node sends the load balancing operation calculated in the above step to the current Carrier node through an RPC request to execute the load balancing operation.
The Arbiter node records the instruction of the load balancing operation into the load balancing operation queues of the current Carrier node and the target Tablet while issuing the instruction of the load balancing operation, and no new operation is allowed to be added after the load balancing operation in the queues reaches a certain number, so that the influence of excessive Replica scheduling on service access in a short time is avoided.
When the current Carrier node receives a load balancing request of the Arbiter node, the Leader node switching operation and the migration operation of the Replica are completed through a consistency protocol.
The Arbiter node cleans up the load balancing operation queue of the current Carrier node and the target Tablet according to the execution result reported by the Carrier node, wherein the target Tablet is determined according to the Tablet to which the first target replay and/or the second target replay belong.
The distributed key value storage load balancing method provided by the invention can ensure the balance of data storage and data access in a distributed key value storage system, prevent the jitter of service data access in the scheduling process of data fragmentation, and avoid the influence of excessive Replica scheduling in a short time on the normal access of services.
The distributed key value storage load balancing system provided by the present invention is described below, and the distributed key value storage load balancing system described below and the distributed key value storage load balancing method described above may be referred to in correspondence with each other.
Fig. 2 is a schematic structural diagram of a distributed key value storage load balancing system provided in the present invention, as shown in fig. 2, a receiving module 210, an adjusting module 211, and an updating module 212;
the receiving module 210 is configured to receive a load state of a current Carrier node, which is sent by a heartbeat and currently bears the Carrier node in the distributed key value storage system;
the adjusting module 211 is configured to determine whether to perform load balancing according to a load state of a current Carrier node, and send a load balancing operation to the current Carrier node when determining to perform load balancing, so that the current Carrier node can be adjusted;
and the updating module 212 is used for updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
The distributed key value storage load balancing system provided by the invention can realize the balance of data storage and data access in a distributed consistent storage system (such as the distributed key value storage system), and can effectively prevent the jitter of service data access in the scheduling process of data fragmentation, thereby ensuring the stability and high availability of the distributed key value storage system.
Fig. 3 is a schematic physical structure diagram of an electronic device provided in the present invention, and as shown in fig. 3, the electronic device may include: a processor (processor)310, a communication interface (communication interface)311, a memory (memory)312 and a bus (bus)313, wherein the processor 310, the communication interface 311 and the memory 312 complete communication with each other through the bus 313. The processor 310 may call logic instructions in the memory 312 to perform the following method:
receiving a load state of a current Carrier node sent by a current Carrier node through heartbeat in a distributed key value storage system;
judging whether to perform load balancing according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for adjustment of the current Carrier node when determining to perform load balancing;
and updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
In addition, the logic instructions in the memory may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention or a part thereof, which essentially contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer power screen (which may be a personal computer, a server, or a network power screen, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like.
Further, the present invention discloses a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which when executed by a computer, the computer is capable of executing the distributed key-value storage load balancing method provided by the above-mentioned method embodiments, for example, the method comprises:
receiving a load state of a current Carrier node sent by a current Carrier node through heartbeat in a distributed key value storage system;
judging whether to perform load balancing according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for adjustment of the current Carrier node when determining to perform load balancing;
and updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
In another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the distributed key value storage load balancing method provided in the foregoing embodiments when executed by a processor, for example, the method includes:
receiving a load state of a current Carrier node sent by a current Carrier node through heartbeat in a distributed key value storage system;
judging whether to perform load balancing according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for adjustment of the current Carrier node when determining to perform load balancing;
and updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
The above-described system embodiments are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on such understanding, the above technical solutions may be essentially or partially implemented in the form of software products, which may be stored in computer readable storage media, such as ROM/RAM, magnetic disk, optical disk, etc., and include instructions for causing a computer power supply screen (which may be a personal computer, a server, or a network power supply screen, etc.) to execute the method according to the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A distributed key value storage load balancing method is characterized by comprising the following steps:
receiving a load state of a current Carrier node sent by a current Carrier node through heartbeat in a distributed key value storage system;
judging whether to perform load balancing according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for the current Carrier node to adjust when determining to perform load balancing;
and updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
2. The distributed key value storage load balancing method according to claim 1, wherein the load status of the current Carrier node includes:
configuring information, resource occupation condition and duplicate status of logic fragment Tablet;
the configuration information at least comprises the number of CPU cores and the size of a memory;
the resource occupation condition at least comprises a CPU utilization rate and a memory utilization rate;
the Replica state at least comprises a logic fragment identifier tableId, a copy identifier ReplicaId, an Instance identifier InstanceId, a query quantity per second in Write operation QPS, a logic fragment Version tableset Version, a query quantity per second in Read operation Read QPS and a Consensus Group Consenses Group;
the Tablet Version, Read QPS and Consenssus Group are the status that the Replica is the master node.
3. The distributed key value storage load balancing method according to claim 1, wherein the determining whether to perform load balancing according to the load state of the current Carrier node, and sending a load balancing operation to the current Carrier node when determining to perform load balancing, so that the current Carrier node can adjust the load balancing method includes:
updating the load state of the distributed key value storage system according to the load state of the current Carrier node;
judging whether the load state of the current Carrier node meets a first preset condition or not according to the updated load state of the distributed key value storage system;
if the first preset condition is not met, determining to perform load balancing, selecting a first target Replica from Replicas of the current Carrier node to perform main node switching operation, and updating the load state of the current Carrier node;
if the load state of the current Carrier node after the main node switching operation does not meet the first preset condition, performing migration operation of a second target Replica;
and determining the load balancing operation according to the main node switching operation and the migration operation, and sending the load balancing operation to the current Carrier node for adjustment of the current Carrier node.
4. The distributed key value storage load balancing method according to claim 3, wherein the first preset condition includes:
the CPU utilization rate and the memory utilization rate of the current Carrier node are lower than a first preset threshold value; or
The QPS load of each CPU core of the current Carrier node is lower than the preset average QPS; or
The current Carrier node has the load balancing operation not completed.
5. The distributed key value storage load balancing method according to claim 3, wherein the selecting a first target Replica from the replicas of the current Carrier node to perform a master node switching operation includes:
selecting a Replica meeting a second preset condition from replias of the main node of the current Carrier node as the first target Replica to perform main node switching operation;
wherein the second preset condition includes that a Replica exists in the Consensus Group of the Tablet.
6. The distributed key value storage load balancing method according to claim 3, wherein the performing migration operation of the second target Replica includes:
adding the second target Replica on the Carrier node with the lowest load and deleting the second target Replica on the current Carrier node;
wherein the second target Replica is determined according to a Replica satisfying a third preset condition selected from Replicas of slave nodes of the current Carrier node;
the third preset condition includes that the migration operation or the splitting operation is not performed within the preset time of the owned Tablet, and the difference value of the Instance Ids of all the replias in the Consensus Group of the owned Tablet does not exceed a second preset threshold.
7. The method for balancing distributed key value storage load according to claim 3, wherein the updating the current Carrier node according to the adjusted execution result sent by the current Carrier node comprises:
updating the load balancing operation queues of the current Carrier node and the target Tablet according to the execution result sent by the current Carrier node after the load balancing operation is executed;
wherein the load balancing operation is sent by a Remote Procedure Call (RPC) request;
the load balancing operation queue is determined after the load balancing operation is recorded;
the target Tablet is determined according to the Tablet to which the first target Replica and/or the second target Replica belong.
8. A distributed key-value store load balancing system, comprising: the device comprises a receiving module, an adjusting module and an updating module;
the receiving module is used for receiving the load state of the current Carrier node sent by the current Carrier node through heartbeat in the distributed key value storage system;
the adjusting module is used for judging whether load balancing is carried out or not according to the load state of the current Carrier node, and sending load balancing operation to the current Carrier node for adjusting the current Carrier node when the load balancing is determined;
and the updating module is used for updating the current Carrier node according to the adjusted execution result sent by the current Carrier node.
9. An electronic device comprising a processor and a memory storing a computer program, wherein the processor implements the steps of the distributed key-value storage load balancing method of any one of claims 1 to 7 when executing the computer program.
10. A processor-readable storage medium, wherein the processor-readable storage medium stores a computer program for causing a processor to perform the steps of the distributed key value store load balancing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110806889.2A CN113553179A (en) | 2021-07-16 | 2021-07-16 | Distributed key value storage load balancing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110806889.2A CN113553179A (en) | 2021-07-16 | 2021-07-16 | Distributed key value storage load balancing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113553179A true CN113553179A (en) | 2021-10-26 |
Family
ID=78131974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110806889.2A Pending CN113553179A (en) | 2021-07-16 | 2021-07-16 | Distributed key value storage load balancing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113553179A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114443057A (en) * | 2022-01-25 | 2022-05-06 | 北京百度网讯科技有限公司 | Deployment of conversation model, conversation method, device, electronic equipment and storage medium |
| CN114666335A (en) * | 2022-03-21 | 2022-06-24 | 北京计算机技术及应用研究所 | DDS-based distributed system load balancing device |
| CN115033390A (en) * | 2022-08-09 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Load balancing method and device |
| CN115794660A (en) * | 2023-02-06 | 2023-03-14 | 青软创新科技集团股份有限公司 | Control method, device and system based on distributed program evaluation |
| CN116991580A (en) * | 2023-07-27 | 2023-11-03 | 上海沄熹科技有限公司 | Distributed database system load balancing method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101013387A (en) * | 2007-02-09 | 2007-08-08 | 华中科技大学 | Load balancing method based on object storage device |
| CN102857577A (en) * | 2012-09-24 | 2013-01-02 | 北京联创信安科技有限公司 | System and method for automatic load balancing of cluster storage |
| US20130332608A1 (en) * | 2012-06-06 | 2013-12-12 | Hitachi, Ltd. | Load balancing for distributed key-value store |
| CN104144127A (en) * | 2013-05-08 | 2014-11-12 | 华为软件技术有限公司 | Load balancing method and device |
| CN108009018A (en) * | 2016-10-28 | 2018-05-08 | 华为技术有限公司 | A kind of load regulation method, apparatus and system |
| CN112748996A (en) * | 2021-01-20 | 2021-05-04 | 北京明略昭辉科技有限公司 | Load balancing strategy method and system of non-centralized topology system |
-
2021
- 2021-07-16 CN CN202110806889.2A patent/CN113553179A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101013387A (en) * | 2007-02-09 | 2007-08-08 | 华中科技大学 | Load balancing method based on object storage device |
| US20130332608A1 (en) * | 2012-06-06 | 2013-12-12 | Hitachi, Ltd. | Load balancing for distributed key-value store |
| CN102857577A (en) * | 2012-09-24 | 2013-01-02 | 北京联创信安科技有限公司 | System and method for automatic load balancing of cluster storage |
| CN104144127A (en) * | 2013-05-08 | 2014-11-12 | 华为软件技术有限公司 | Load balancing method and device |
| CN108009018A (en) * | 2016-10-28 | 2018-05-08 | 华为技术有限公司 | A kind of load regulation method, apparatus and system |
| CN112748996A (en) * | 2021-01-20 | 2021-05-04 | 北京明略昭辉科技有限公司 | Load balancing strategy method and system of non-centralized topology system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114443057A (en) * | 2022-01-25 | 2022-05-06 | 北京百度网讯科技有限公司 | Deployment of conversation model, conversation method, device, electronic equipment and storage medium |
| CN114666335A (en) * | 2022-03-21 | 2022-06-24 | 北京计算机技术及应用研究所 | DDS-based distributed system load balancing device |
| CN115033390A (en) * | 2022-08-09 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Load balancing method and device |
| CN115033390B (en) * | 2022-08-09 | 2022-11-25 | 阿里巴巴（中国）有限公司 | Load balancing method and device |
| CN115794660A (en) * | 2023-02-06 | 2023-03-14 | 青软创新科技集团股份有限公司 | Control method, device and system based on distributed program evaluation |
| CN116991580A (en) * | 2023-07-27 | 2023-11-03 | 上海沄熹科技有限公司 | Distributed database system load balancing method and device |
| CN116991580B (en) * | 2023-07-27 | 2024-09-13 | 上海沄熹科技有限公司 | Distributed database system load balancing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113553179A (en) | Distributed key value storage load balancing method and system | |
| US20200310660A1 (en) | Identifying sub-health object storage devices in a data storage system | |
| US10348827B2 (en) | Distributed storage system, cluster node and range management method thereof | |
| CN102265277B (en) | Operation method and device for data memory system | |
| CN108804465B (en) | Method and system for data migration of distributed cache database | |
| US20070061379A1 (en) | Method and apparatus for sequencing transactions globally in a distributed database cluster | |
| US11068499B2 (en) | Method, device, and system for peer-to-peer data replication and method, device, and system for master node switching | |
| CN107919977B (en) | Online capacity expansion and online capacity reduction method and device based on Paxos protocol | |
| TWI701562B (en) | Database flexible scheduling method and device | |
| CN109739435B (en) | File storage and updating method and device | |
| CN111049928B (en) | Data synchronization method, system, electronic device and computer readable storage medium | |
| CN102577241A (en) | Method, device and system for scheduling distributed buffer resources | |
| CN108132837A (en) | A kind of distributed type assemblies dispatch system and method | |
| WO2014177085A1 (en) | Distributed multicopy data storage method and device | |
| CN109582459A (en) | The method and device that the trustship process of application is migrated | |
| JP2016144169A (en) | Communication system, queue management server, and communication method | |
| CN114124650A (en) | Master-slave deployment method of SPTN (shortest Path bridging) network controller | |
| CN111726388A (en) | Cross-cluster high-availability implementation method, device, system and equipment | |
| CN114265753A (en) | Management method and management system of message queue and electronic equipment | |
| CN116954863A (en) | Database scheduling method, device, equipment and storage medium | |
| CA2619778C (en) | Method and apparatus for sequencing transactions globally in a distributed database cluster with collision monitoring | |
| CN104468674B (en) | Data migration method and device | |
| CN114647701A (en) | Load balancing method and device for distributed database, electronic equipment and medium | |
| US20240028611A1 (en) | Granular Replica Healing for Distributed Databases | |
| JP2024514467A (en) | Geographically distributed hybrid cloud cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |