CN116974731A

CN116974731A - Disk load balancing method and device and electronic equipment

Info

Publication number: CN116974731A
Application number: CN202210424208.0A
Authority: CN
Inventors: 卢杰; 胡晓花
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2022-04-21
Filing date: 2022-04-21
Publication date: 2023-10-31

Abstract

The disclosure provides a disk load balancing device and an electronic device, which are applied to clusters deployed across racks, and the method comprises the following steps: determining a topic master node list and a node topic partition list to be balanced; determining target master nodes corresponding to the first copy partitions of the nodes to be balanced one by one based on the topic master node list and the master node selection rule; determining target slave nodes corresponding to the second copy partitions of the nodes to be balanced one by one based on the disk utilization rate of the disk and the slave node selection rule; and moving the first copy partition of the node to be balanced to the corresponding target master node, and moving the second copy partition of the node to be balanced to the corresponding target slave node. In the cross-rack deployed cluster, the selection of the target master node and the target slave node is enriched, and the disk load of the cross-rack cluster is balanced.

Description

Disk load balancing method and device and electronic equipment

Technical Field

The disclosure relates to the technical field of data processing, and in particular relates to a disk load balancing method, a disk load balancing device and electronic equipment.

Background

When a cluster is deployed, considering the situation that a certain rack in a machine room is not available, a scheme of deploying the cluster across racks is generally adopted, and node servers in the whole cluster are deployed on a plurality of racks.

In the related art, the existing data distribution for a cluster is balanced by the number of partitions of the subject on each node. However, because the speeds of different theme production data are different, the total partition numbers under different nodes are similar, but the corresponding disk usage rates may have great difference, so that the node with high disk usage rate may be occupied and abnormal when the disk is in a high peak value, and the node with low disk usage rate is idle, so that the overall disk usage rate of the cluster is not high, and the usability of the whole cluster is affected.

Balancing disk loads for clusters across racks is thus a current challenge.

Disclosure of Invention

The disclosure provides a disk load balancing method, a disk load balancing device and electronic equipment, which are used for solving the defect that in the prior art, the disk load of a cross-rack cluster cannot be balanced, and realizing the balancing of the disk load of the cross-rack cluster.

The disclosure provides a disk load balancing method, which is applied to a cluster deployed across racks, wherein the cluster comprises a plurality of nodes corresponding to disks one by one, and the disks are deployed on different racks respectively; the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of replica partitions; the method comprises the steps that a plurality of duplicate partitions belonging to the same partition are respectively arranged in different nodes, one node is set as a master node of the partition, and other nodes are set as slave nodes of the partition; the method comprises the following steps:

Determining a topic master node list and a topic partition list of nodes to be balanced, wherein the topic master node list comprises a plurality of master nodes belonging to the same topic, and the topic partition list of the nodes to be balanced comprises copy partition information arranged on the nodes to be balanced;

determining target master nodes corresponding to first replica partitions of nodes to be balanced one by one based on the topic master node list and a master node selection rule, wherein the first replica partitions are determined by the topic partition list of the nodes to be balanced, and the first replica partitions are replica partitions taking the nodes to be balanced as partitions of the master nodes;

determining target slave nodes corresponding to second copy partitions of the nodes to be balanced one by one based on the disk utilization rate of the disk and a slave node selection rule, wherein the second copy partitions are determined by the topic partition list of the nodes to be balanced, and the second copy partitions are copy partitions taking the nodes to be balanced as partitions of the slave nodes;

and moving the first copy partition of the node to be balanced to the corresponding target master node, and moving the second copy partition of the node to be balanced to the corresponding target slave node.

According to the disk load balancing method provided by the disclosure, the step of determining the topic master node list and the topic partition list of the nodes to be balanced comprises the following steps:

obtaining a topic list, wherein the topic list comprises topics of the cluster;

determining topic information according to the topic list, wherein the topic information comprises a partition of the topic, a copy partition of the partition and a mapping relation between the copy partition and a node where the copy partition is located;

and determining the topic master node list and the topic partition list of the nodes to be balanced according to the topic information.

According to the disk load balancing method provided by the present disclosure, the step of determining a target master node corresponding to a first copy partition of a node to be balanced one to one based on the subject master node list and a master node selection rule includes:

determining a first copy partition of an unselected target master node, and selecting candidate master nodes from nodes with non-selected attributes of the cluster, wherein the candidate master nodes are different from master nodes included in a first topic master node list, and the first topic master node list is the topic master node list of a topic corresponding to the first copy partition;

Acquiring a first bearing number of the candidate master node, wherein the first bearing number is the number of partitions taking the candidate master node as a master node in a theme corresponding to the first duplicate partition;

selecting one candidate master node with the first bearing quantity smaller than a first bearing quantity threshold as the target master node;

changing the attribute of the node corresponding to the target master node from non-selection to selected;

and returning to the step of re-executing the first copy partition which determines one unselected target master node, and selecting candidate master nodes from the nodes with non-selected attributes of the cluster until each first copy partition determines the corresponding target master node.

According to the disk load balancing method provided by the present disclosure, the step of determining a target master node corresponding to a first copy partition of a node to be balanced one to one based on the subject master node list and a master node selection rule further includes:

and when the candidate master node cannot be selected or the target master node cannot be selected according to the candidate master node, selecting the node with the minimum disk use rate in the cluster as the target master node of the first copy partition of the unselected target master node.

According to the disk load balancing method provided by the present disclosure, the first load quantity threshold is obtained through the following steps:

acquiring the total number of nodes of the nodes in the cluster and the total number of first topic partitions, wherein the total number of the first topic partitions is the number of partitions of topics corresponding to the first copy partitions;

and calculating the first bearing quantity threshold according to the total number of nodes and the total number of the first subject partitions.

According to the disk load balancing method provided by the present disclosure, the step of determining a target slave node corresponding to a second replica partition of a node to be balanced one to one based on a disk usage rate of the disk and a slave node selection rule includes:

determining a second copy partition of an unselected target slave node, and determining candidate slave nodes in the nodes of the cluster, wherein the candidate slave nodes are different from the target master node, and the disks corresponding to the candidate slave nodes and the disks corresponding to the target master node are distributed on different racks;

selecting a node with non-selected attribute from the candidate slave nodes as the target slave node;

changing the attribute of the node corresponding to the target slave node from non-selection to selected;

And returning to the step of re-executing the second copy partition for determining one unselected target slave node, and determining candidate slave nodes in the nodes with non-selected attributes of the cluster until each second copy partition determines the corresponding target slave node.

According to the method for balancing the disk load provided by the present disclosure, before the step of changing the attribute of the node corresponding to the target slave node from non-selection to selected, the method further includes:

when a node with non-selected attribute cannot be selected from the candidate slave nodes as the target slave node, calculating corresponding weight of disk utilization rate corresponding to the candidate slave node according to a preset segmentation rule;

and determining one target slave node in the candidate slave nodes according to the weight corresponding to the candidate slave nodes.

According to the disk load balancing method provided by the disclosure, the node to be balanced is determined by the following steps:

determining the disk utilization rate corresponding to each node in the cluster;

and determining the node to be balanced according to the variance and the mean value of the disk utilization rate.

According to the disk load balancing method provided by the present disclosure, the step of partitioning and moving the first copy of the node to be balanced to the corresponding target master node includes:

Corresponding to each first copy partition, determining a corresponding target master node prediction utilization rate according to the disk utilization rate of the target master node, the first copy partition space size and the disk size of the target master node, wherein the first copy partition space size is determined by the subject partition list of the nodes to be balanced;

and when the predicted utilization rate of the target master node meets a first preset condition, moving the corresponding first copy partition of the node to be balanced to the corresponding target master node, and updating the disk utilization rate of the nodes in the cluster.

According to the disk load balancing method provided by the present disclosure, the step of moving the second copy partition of the node to be balanced to the corresponding target slave node includes:

corresponding to each second copy partition, determining a corresponding target slave node prediction utilization rate according to the disk utilization rate of the target slave node, the space size of the second copy partition and the disk size of the target slave node, wherein the space size of the second copy partition is determined by the topic partition list of the node to be balanced;

and when the target slave node forecast utilization rate meets a second preset condition, moving the corresponding second copy partition of the node to be balanced to the corresponding target slave node, and updating the disk utilization rate of the nodes in the cluster.

The disclosure also provides a disk load balancing device, which is applied to a cluster deployed across racks, wherein the cluster comprises a plurality of nodes corresponding to the disks one by one, and the disks are deployed on different racks respectively; the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of replica partitions; the method comprises the steps that a plurality of duplicate partitions belonging to the same partition are respectively arranged in different nodes, one node is set as a master node of the partition, and other nodes are set as slave nodes of the partition; the device comprises:

the information determining module is used for determining a theme master node list and a to-be-equalized node theme partition list, wherein the theme master node list comprises a plurality of master nodes belonging to the same theme, and the to-be-equalized node theme partition list comprises copy partition information arranged on the to-be-equalized node;

the target master node selection module is used for determining target master nodes corresponding to first copy partitions of the nodes to be balanced one by one based on the topic master node list and master node selection rules, wherein the first copy partitions are determined by the topic partition list of the nodes to be balanced, and the first copy partitions are copy partitions taking the nodes to be balanced as partitions of the master nodes;

The target slave node selection module is used for determining target slave nodes corresponding to second replica partitions of the nodes to be balanced one by one based on the disk utilization rate of the disk and slave node selection rules, wherein the second replica partitions are determined by the topic partition list of the nodes to be balanced, and the second replica partitions are replica partitions taking the nodes to be balanced as partitions of the slave nodes;

and the migration module is used for moving the first copy partition of the node to be balanced to the corresponding target master node and moving the second copy partition of the node to be balanced to the corresponding target slave node.

The present disclosure also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any one of the disk load balancing methods described above when the program is executed.

The present disclosure also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the disk load balancing method as described in any of the above.

The disk load balancing method, the disk load balancing device and the electronic equipment are applied to clusters deployed across racks, wherein the clusters comprise a plurality of nodes corresponding to the disks one by one, and the disks are deployed on different racks respectively; the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of duplicate partitions; the multiple duplicate partitions belonging to the same partition are respectively arranged in different nodes, one of the nodes is set as a master node of the partition, and the other nodes are set as slave nodes of the partition. Determining a copy partition included in the node to be balanced through a subject partition list of the node to be balanced, selecting a target master node for a first copy partition of the node to be balanced through a master node list and a master node selection rule, selecting a target slave node for a second copy partition of the node to be balanced through a disk utilization rate and a slave node selection rule, and respectively migrating the first copy partition and the second copy partition of the node to be balanced to the target master node and the target slave node, wherein in a cluster deployed across a rack, the selection of the target master node and the target slave node is enriched, and balancing disk loads of the cluster across the rack is realized.

Drawings

In order to more clearly illustrate the technical solutions of the present disclosure or the prior art, the following description will briefly introduce the drawings that are required to be used in the embodiments or the prior art descriptions, and it is obvious that the drawings in the following description are some embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort for a person of ordinary skill in the art.

FIG. 1 is a schematic flow chart of a disk load balancing method provided by the present disclosure;

FIG. 2 is a second flow chart of a disk load balancing method provided by the present disclosure;

FIG. 3 is a third flow chart of the disk load balancing method provided by the present disclosure;

FIG. 4 is a flow chart of a disk load balancing method provided by the present disclosure;

FIG. 5 is a flowchart of a disk load balancing method provided by the present disclosure;

FIG. 6 is a flowchart illustrating a disk load balancing method provided by the present disclosure;

FIG. 7 is a flow chart of a disk load balancing method provided by the present disclosure;

FIG. 8 is one of the disk load balancing architecture schematics provided by the present disclosure;

FIG. 9 is a second schematic diagram of a disk load balancing architecture provided by the present disclosure;

FIG. 10 is a third schematic diagram of a disk load balancing architecture provided by the present disclosure;

FIG. 11 is a fourth schematic diagram of a disk load balancing architecture provided by the present disclosure;

FIG. 12 is a schematic diagram of a disk load balancing apparatus provided by the present disclosure;

fig. 13 is a schematic structural view of an electronic device provided by the present disclosure.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are some embodiments, but not all embodiments of the present disclosure. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the disclosed embodiments, are within the scope of the disclosed embodiments.

For ease of understanding, the technical terms to which the present invention relates will be explained first.

(1) Load balancing (Outbound Load Balancing)

Load balancing is established on the existing network structure, and provides a cheap, effective and transparent method for expanding the bandwidth of network equipment and servers, increasing throughput, enhancing network data processing capacity and improving flexibility and usability of the network. As the traffic increases, the access volume and data traffic increases rapidly, so does the processing power and computational intensity of the current core of the network, making a single server device unable to afford at all. In this case, if the existing device is thrown away to make a large amount of hardware upgrades, this will cause waste of existing resources, and if the next increase of traffic is faced, this will result in high cost investment of hardware upgrade again, and even the device with excellent performance cannot meet the current demand of traffic increase.

(2) Cluster (Cluster)

A cluster is a collection of computing, storage, and network resources. In brief overview, a cluster is a group of computers. They, as a general matter, offer a set of network resources to clients. Each constituent point of a cluster is called a Node (Node), and a cluster is formed by combining the nodes. The Node has the responsibility of running the container application, and the Node is responsible for monitoring and reporting the state of the container and managing the life cycle of the container according to the requirement of the Cluster. The Node runs on the Linux operating system and can be a physical machine or a virtual machine.

(3)Kafka

Kafka is a high throughput distributed message queue system. The Kafka architecture includes a message producer (producer), a message consumer (consumer), a node of the Kafka cluster (broker), and a zookeeper. Wherein the node (reader) is responsible for processing the message read and write requests and storing the message. In addition, under the node (brooker) of the kafka cluster, there are again topics (topic), partitions (partition) and Zookeeper. The Zookeeper is used for storing metadata information of Kafka data and consumer consumption offset, and the metadata information comprises: theme information, partition information.

Specifically, the messages in the kafka cluster are composed of topics (topic), which can be understood as a queue, one queue is just one topic (topic), each topic (topic) can be divided into a plurality of partitions (partitions), and the messages in each partition are strongly ordered and are equivalent to ordered queues. One node (broker) may manage multiple partitions.

(4) leader load balancing mechanism

A partition may have multiple copies, including a master copy and at least one slave copy, where the master copy and the slave copies may be stored on different nodes, the node where the master copy is stored is referred to as the leader node of the partition, and the node where the slave copy is stored is referred to as the follower node of the partition.

In the prior art, when a Kafka cluster is deployed, a scheme of deploying the Kafka cluster across racks is generally adopted to deploy node servers in the whole cluster on a plurality of racks in consideration of the situation that a certain rack in a machine room is not available.

In the related art, the Kafka cluster is often deployed on the same rack, and the existing data distribution is balanced according to the number of partitions of the subject on each node. However, because the speeds of different theme production data are different, the total partition numbers under different nodes are similar, but the corresponding disk usage rates may have great difference, so that the node with high disk usage rate may be occupied and abnormal when the disk is in a high peak value, and the node with low disk usage rate is idle, so that the overall disk usage rate of the cluster is not high, the usability of the cluster is affected, and a scheme for balancing the overall disk of the cluster across the racks is lacking.

Balancing disk loads for clusters across racks is thus a current challenge.

In order to solve the problems, the disclosure provides a disk load balancing method, which is applied to a cluster deployed across racks, wherein the cluster comprises a plurality of nodes corresponding to the disks one by one, and the disks are deployed on different racks respectively; the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of replica partitions; the method comprises the steps that a plurality of duplicate partitions belonging to the same partition are respectively arranged in different nodes, one node is set as a master node of the partition, and other nodes are set as slave nodes of the partition;

in one example, a cluster deployed across racks includes node 1 and node 2, where disks corresponding to node 1 are deployed on rack 1 and disks corresponding to node 2 are deployed on rack 2. The cluster comprises a theme A and a theme B, wherein the theme A comprises a partition a1, and the partition a1 comprises a copy partition a11 and a12; subject B includes partition B1, partition B1 including duplicate partitions B11 and B12. Replica partition a11 and replica partition b12 are disposed in node 1, and replica partitions a12 and b11 are disposed in node 2. At this time, the node with the smaller number of the duplicate partition is taken as the duplicate partition of the corresponding partition, and it can be known that the node 1 is the master node of the partition a1 corresponding to the duplicate partition a11 and is also the slave node of the partition b1 corresponding to the duplicate partition b 12; similarly, node 2 is a slave node to partition a1 corresponding to duplicate partition a12, and is also a master node to partition b1 corresponding to duplicate partition b 11.

As shown in fig. 1, the method comprises the steps of:

s11, determining a topic master node list and a topic partition list of the nodes to be balanced.

The topic master node list comprises a plurality of master nodes belonging to the same topic, and the topic partition list of the nodes to be balanced comprises copy partition information arranged on the nodes to be balanced.

And S12, determining target master nodes corresponding to the first copy partitions of the nodes to be balanced one by one based on the subject master node list and the master node selection rule.

The first duplicate partition is determined by the topic partition list of the node to be balanced, and the first duplicate partition is a duplicate partition taking the node to be balanced as a partition of a main node.

Specifically, the topic partition list of the node to be balanced can determine that one or more topic partitions are set in the replica partition of the node to be balanced, and after determining the replica partition of the node to be balanced, the replica partitions are classified, and the replica partition of the partition taking the node to be balanced as the main node is taken as the first replica partition.

Based on the subject master node list and master node selection principles, a target master node is determined in the cluster for each first replica partition of the nodes to be balanced as the node to be migrated to.

S13, determining target slave nodes corresponding to the second copy partitions of the nodes to be balanced one by one based on the disk utilization rate of the disk and the slave node selection rule.

The second replica partition is determined by the topic partition list of the node to be balanced, and the second replica partition is a replica partition taking the node to be balanced as a partition of a slave node.

Specifically, after determining the duplicate partitions of the nodes to be balanced, classifying the duplicate partitions, and taking the duplicate partition taking the nodes to be balanced as the partitions of the slave nodes as the second duplicate partition.

And determining a target slave node in the cluster as a node to be migrated for each second copy partition of the node to be balanced based on the disk usage rate of the corresponding disk of the node in the cluster and the slave node selection principle.

In one example, the to-be-balanced node M is determined to include a replica partition of the topic a and the topic B through the topic partition list of the to-be-balanced node, and specifically includes a replica partition a11 of the topic a partition a1 with the to-be-balanced node M as a master node, a replica partition a22 of the topic a partition a2 with the to-be-balanced node M as a slave node, and a replica partition B11 of the topic B partition B1 with the to-be-balanced node M as a master node. At this time, the first copy partition of the node to be balanced M is a11 and b11, and the second copy partition of the node to be balanced M is a22.

S14, moving the first copy partition of the node to be balanced to the corresponding target master node, and moving the second copy partition of the node to be balanced to the corresponding target slave node.

In the embodiment of the disclosure, a copy partition included in a node to be balanced is determined through a subject partition list of the node to be balanced, a target master node is selected for a first copy partition of the node to be balanced through a master node list and a master node selection rule, a target slave node is selected for a second copy partition of the node to be balanced through a disk utilization rate and a slave node selection rule, the first copy partition and the second copy partition of the node to be balanced are respectively migrated to the target master node and the target slave node, and in a cluster deployed across a rack, the selection of the target master node and the target slave node is enriched, so that disk loads of the cluster across the rack are balanced.

According to the disk load balancing method provided by the present disclosure, step S11 specifically includes:

s111, obtaining a theme list.

The topic list includes topics of the cluster.

S112, determining theme information according to the theme list.

The topic information comprises a partition of the topic, a copy partition of the partition and a mapping relation between the copy partition and a node where the copy partition is located.

And S113, determining the topic master node list and the topic partition list of the nodes to be balanced according to the topic information.

Specifically, a list of topics on the cluster may be obtained through the listTopics interface of the client of kafka, where the list of topics is used to represent one or more topics included by the cluster. The topic information of each topic may be obtained through a desribetetopics interface using a topic list, which may include, but is not limited to, a transcript list included therein.

The replica list is used to represent the mapping relationship between the replica partitions corresponding to the partitions under one or more topics included in the cluster and the nodes in the cluster.

A topic master node list representing a plurality of master nodes belonging to the same topic can be determined from the topic information.

The topic information contained in the duplicate list is classified according to the node names, so that a topic partition list distributed on each node can be obtained, and further, the fact that the partitions under the topics contained in the nodes to be balanced are arranged in the duplicate partitions of the nodes to be balanced can be determined, and the topic partition list of the nodes to be balanced is obtained.

In the embodiment of the disclosure, the topic list is obtained, the topic information is determined according to the topic list, and the topic master node list and the topic partition list of the nodes to be balanced are determined according to the topic information, so that the topic master node list and the topic partition list of the nodes to be balanced can be accurately determined.

According to the disk load balancing method provided by the present disclosure, as shown in fig. 2, step S12 specifically includes:

s21, determining a first copy partition of a non-selected target master node, and selecting a candidate master node from the nodes with non-selected attributes of the cluster.

The candidate master nodes are different from master nodes included in a first topic master node list, and the first topic master node list is the topic master node list of topics corresponding to the first copy partition.

Specifically, a first copy partition of an unselected target master node is determined, a first topic master node list of topics corresponding to the first copy partition is determined, and a node which is non-selected in attribute and does not belong to the first topic master node list is selected from a plurality of nodes of the cluster to serve as a candidate master node of the first copy partition.

In one example, a first copy partition a11 of an unselected target master node corresponds to a topic a, and the topic master node list of the topic a includes a master node M and a master node N, that is, a node with a non-selected attribute is selected from other nodes except M and N in the cluster as a candidate master node of the first copy partition a 11.

S22, acquiring the first bearing number of the candidate master node.

The first bearing number is the number of partitions taking the candidate master node as the master node in the subject corresponding to the first duplicate partition.

Specifically, the partition of the subject corresponding to this first duplicate partition may be obtained, and the number of partitions in these partitions that have had candidate master nodes as master nodes may be further determined.

In one example, the first duplicate partition a11 is a duplicate partition set by the partition a1 with the node to be balanced as a main node, where the corresponding topic is a, and the topic a includes a partition a2 and a partition a3 in addition to the partition a 1. The candidate master node of the first duplicate partition a11 is a node M, and if the partition a2 and the partition a3 take the node M as respective master nodes, the first bearing number of the candidate master node M at this time is 2.

S23, selecting one candidate master node with the first bearing quantity smaller than a first bearing quantity threshold as the target master node.

In a preferred example, a general polling strategy may be employed to select the target master node from low to high disk usage.

S24, changing the attribute of the node corresponding to the target master node from non-selection to selected.

Specifically, after the attribute of the node corresponding to the target master node is changed from non-selection to selected, the target master node will not participate in the process of selecting the target master node by other first copy partitions of the node to be balanced.

S25, returning to the step of re-executing the first copy partition which determines one unselected target master node, and selecting candidate master nodes from the nodes with non-selected attributes of the cluster until each first copy partition determines the corresponding target master node.

In the embodiment of the disclosure, the first copy partition of the unselected target master node is determined, the candidate master node is determined according to the first copy partition, the target master node is determined through the candidate master node, and the target master node finally determined through a plurality of conditions realizes that the resource deficiency of a plurality of partitions with the same subject and caused by the same node as the master node is avoided as much as possible, thereby reducing the pressure of node bandwidth, magnetic disk and the like. The determined target master node has better bearing capacity.

According to the disk load balancing method provided by the present disclosure, step S12 further includes:

Specifically, if a candidate master node cannot be selected from the non-selected nodes of the cluster, or one of the candidate master nodes with the first bearer number smaller than the first bearer number threshold cannot be selected as the target master node, at this time, a node with low disk usage is preferentially considered, and a node with minimum disk usage in the cluster is used as the target master node of the first copy partition where the target master node is not selected.

In the embodiment of the present disclosure, when the candidate master node cannot be selected or the target master node cannot be selected according to the candidate master node, a node with the minimum disk usage rate in the cluster is selected as the target master node of the first copy partition where the target master node is not selected. The node with the minimum disk utilization rate is used as the target main node of the first copy partition of the node to be balanced as far as possible, and the selected target main node has good bearing capacity.

According to the disk load balancing method provided by the present disclosure, as shown in fig. 3, the first threshold of the number of bearers is obtained by:

s31, acquiring the total number of nodes and the total number of first subject partitions of the nodes in the cluster.

The total number of the first subject partitions is the number of the partitions of the subject corresponding to the first copy partition.

Specifically, the total number of nodes in the cluster may be obtained, and the number of partitions included under the corresponding subject is determined according to the determined first copy partition.

S32, calculating the first bearing quantity threshold according to the total number of nodes and the total number of the first subject partitions.

Specifically, in a preferred example, the first bearer number threshold may be calculated according to the following formula (1):

leader_thershold＝topic_partition_num/cluster_node_num (1)

Wherein, the liquid crystal display device comprises a liquid crystal display device,

the leader_threshold is a first bearer number threshold;

topic_partition_num is the total number of partitions under the first topic;

cluster node num is the total number of nodes.

In the embodiment of the present disclosure, one of the core targets of balancing is to make the distribution of the master nodes of the subject partitions uniform, so that the number of partitions that each node can be used as the master node to carry one subject is limited. And calculating a first bearing quantity threshold value through the total number of nodes of the cluster and the total number of the first subject partitions, so that the obtained first bearing quantity refers to the total number of the subject partitions to which the first copy belongs, and has higher reference value.

According to the disk load balancing method provided by the present disclosure, as shown in fig. 4, step S13 specifically includes:

s41, determining a second copy partition of the unselected target slave node, and determining candidate slave nodes in the nodes of the cluster.

The candidate slave nodes are different from the target master node, and the disks corresponding to the candidate slave nodes and the disks corresponding to the target master node are distributed on different racks.

Specifically, determining a second copy partition of an unselected target slave node, selecting a candidate slave node from nodes except the target master node in the cluster, and distributing disks corresponding to the selected candidate slave node and disks corresponding to the target master node on different racks.

In a preferred example, candidate slave nodes may be selected preferentially from low to high disk usage.

S42, selecting a node with non-selected attribute from the candidate slave nodes as the target slave node.

In a preferred example, the target slave node may be selected preferentially from low to high disk usage.

S43, changing the attribute of the node corresponding to the target slave node from non-selection to selected.

S44, returning to execute the second copy partition of the unselected target slave node again, and determining candidate slave nodes in the nodes with non-selected attributes of the cluster until each second copy partition determines the corresponding target slave node.

In the embodiment of the disclosure, the second copy partition of the unselected target slave node is determined, the candidate slave node is determined according to the second copy partition, the target slave node is determined through the candidate slave node, and the target slave node finally determined through a plurality of conditions has better bearing capacity.

According to the disk load balancing method provided by the present disclosure, step S13 further includes:

s45, when a node with non-selected attribute cannot be selected from the candidate slave nodes as the target slave node, calculating the corresponding weight of the disk utilization rate corresponding to the candidate slave node according to a preset segmentation rule.

Specifically, if all nodes are traversed, and one node meeting the above conditions cannot be found as a target slave node, a weighted polling strategy is adopted to preferentially meet the requirement of low disk use rate.

The weighting of weighted polling is based on the disk usage rate of each candidate slave node, a segmentation factor can be set manually, the disk usage rate is divided into a plurality of segments according to the same or different segment lengths, the corresponding weight of each segment is set according to a preset rule, and all candidate slave nodes in the segment have the same weight. After each node is given a weight, the node with the higher weight is selected with a greater probability.

In one example, the disk usage is divided into 4 segments from 0 to 100% according to the same segment length of 25%, the weight of the candidate slave node A with the disk usage in the segment from 0 to 25% is set to 0.4, the weight of the candidate slave node B with the disk usage in the segment from 25 to 50% is set to 0.3, the weight of the candidate slave node C with the disk usage in the segment from 50 to 75% is set to 0.2, and the weight of the candidate slave node D with the disk usage in the segment from 75 to 100% is set to 0.1. After setting, selecting a target slave node from the candidate slave nodes according to the weight, wherein the probability that the candidate slave node with low disk use rate is selected as the target slave node is larger.

S46, determining one target slave node in the candidate slave nodes according to the weight corresponding to the candidate slave nodes.

In the embodiment of the disclosure, when a node with a non-selected attribute cannot be selected from the candidate slave nodes as the target slave node, a weighted polling strategy is adopted, weights are set according to different disk usage rates, and the candidate slave node with low disk usage rate is preferentially selected as the target slave node of the second copy partition according to the weights, so that the selected target slave node has better bearing capacity.

According to the disk load balancing method provided by the present disclosure, as shown in fig. 5, the node to be balanced is determined by the following steps:

s51, determining the disk utilization rate corresponding to each node in the cluster.

In particular, in one example, the Zookeeper interface may be utilized to communicate Zookeeper cluster IP address information relied upon by the incoming cluster. After the IP address information of the Zookeeper cluster is obtained, a Zookeeper client is constructed, the Zookeeper client sends a request, node information under a designated path of the Zookeeper cluster is circularly traversed and analyzed, and finally a list comprising all nodes in the cluster and disk utilization rates corresponding to all the nodes are obtained.

S52, determining the node to be balanced according to the variance and the mean value of the disk usage rate.

Specifically, the average value of the disk usage rates can be calculated according to the disk usage rates corresponding to the nodes in the cluster, and if the average value of the disk usage rates is not within the preset average value threshold value range, the balancing is abandoned.

And if the average value of the disk utilization rate is within the preset average value threshold value range, further analyzing the clusters. The variance of disk usage of the clusters is calculated by the following equation (2):

s represents the variance of disk usage of the cluster

N represents the total number of nodes of the cluster;

curUsage _k : disk usage of node K;

avgcssage: the average value of disk usage of the cluster.

And when the variance of the disk usage rate is larger than a preset variance threshold, selecting a node to be balanced from the cluster.

In a preferred example, all nodes of the cluster are balanced by selecting nodes from high to low according to disk usage. And for the node k, if the corresponding disk utilization rate is greater than a preset range, taking the node k as a node to be balanced.

In the embodiment of the disclosure, the average value and the variance of the disk use rate of the cluster are calculated in advance, and the equilibrium state of the cluster is evaluated from a quantitative angle, so that the disk use state of the cluster can be more intuitively known, unnecessary equilibrium of the cluster is reduced, and the cost is saved.

According to the disk load balancing method provided by the present disclosure, the step of partitioning the first copy of the node to be balanced to the corresponding target master node includes, as shown in fig. 6:

and S61, corresponding to each first copy partition, and determining the corresponding target master node prediction utilization rate according to the disk utilization rate of the target master node, the space size of the first copy partition and the disk size of the target master node.

The first copy partition space size is determined by the topic partition list of the nodes to be balanced.

Specifically, the monitor pointer path may be stored according to the subject partition list and the partition, connected to the node through the jmx monitor port, and obtained as actual storage size data of the partition on the node. Subsequent partitions and so on until the duplicate partition sizes of the subject partitions on all nodes are obtained. Similarly, the size of the partition space of the first copy, which is set on the node to be balanced, can be obtained according to the topic partition list of the node to be balanced. The disk size of the nodes in the cluster can be obtained in advance, and the disk size of the target master node can be further obtained.

And S62, when the predicted utilization rate of the target master node meets a first preset condition, moving the corresponding first copy partition of the node to be balanced to the corresponding target master node, and updating the disk utilization rate of the nodes in the cluster.

Specifically, the first preset condition may predict that the usage rate is not among the first N cluster disk usage rates for the target master node.

In one example, the target master node predicted usage may be determined by the following equation (3):

the predictUsage represents the predicted usage rate of the node to be migrated, and here, represents the predicted usage rate of the target master node;

curUage represents the disk usage rate of the node to be migrated, and here represents the disk usage rate of the target master node;

PartSize _k the space size of the copy partition to be migrated is represented, and the space size of the first copy partition is represented here;

the total space size of the partition on the node to be migrated is represented, and the total space size of the partition on the target main node is represented here;

the quotient of the size of the partition space of the copy to be migrated and the disk size of the node to be migrated is represented, wherein the quotient of the size of the partition space of the first copy and the disk size of the target main node is represented;

predictUsage _undex the predicted utilization rate sequencing sequence number of the node to be migrated is represented, and the predicted utilization rate sequencing sequence number of the target master node is represented;

topN represents the sequence number of N before cluster disk usage.

And when the predicted utilization rate of the target master node is not in the first N of the cluster disk utilization rates, migrating the first copy partition to the target master node, and updating the disk utilization rates of the nodes in the cluster.

In the embodiment of the disclosure, in the equalization process, a situation that a certain node oscillates back and forth occurs. Aiming at the problem of balancing oscillation, a balancing plan needs to be estimated, the change of the disk usage rate of a target main node after balancing is estimated, if the predicted usage rate of the target main node does not meet the expected condition, balancing aiming at the first copy partition is not worth, and the reasonable utilization of resources is ensured by balancing under the condition that the condition is met.

According to the disk load balancing method provided by the present disclosure, as shown in fig. 7, the step of moving the second copy partition of the node to be balanced to the corresponding target slave node includes:

and S71, corresponding to each second copy partition, and determining the corresponding target slave node prediction utilization rate according to the disk utilization rate of the target slave node, the space size of the second copy partition and the disk size of the target slave node.

The second copy partition space size is determined by the node topic partition list to be balanced.

Specifically, this step is similar to step S61, please refer to the description of step S61, and the details are not repeated here.

And S72, when the target slave node predicts that the utilization rate meets a second preset condition, moving the second copy partition of the corresponding node to be balanced to the corresponding target slave node, and updating the disk utilization rate of the nodes in the cluster.

Specifically, the second preset condition may be that the target slave node predicts that the usage rate is not among the first N cluster disk usage rates.

In one example, the target slave node predicted usage may be determined by the following equation (4):

predictusages represent predicted usage of the nodes to be migrated, and here, represent predicted usage of the target slave nodes;

curUage represents the disk usage rate of the node to be migrated, and here, represents the disk usage rate of the target slave node;

PartSize _k representing the space size of the copy partition to be migrated, here representing the second copy partition space size;

the total space size of the partition on the node to be migrated is represented, and the total space size of the partition on the target slave node is represented here;

representing the quotient of the size of the copy partition space to be migrated and the disk size of the node to be migrated, wherein the quotient of the size of the second copy partition space and the disk size of the target slave node is represented;

predictUsage _index the predicted utilization rate sequencing sequence number of the node to be migrated is represented, and the predicted utilization rate sequencing sequence number of the target slave node is represented;

topN represents the sequence number of N before cluster disk usage.

And when the predicted utilization rate of the target slave node is not in the first N of the cluster disk utilization rates, migrating the second auxiliary partition to the target slave node, and updating the disk utilization rates of the nodes in the cluster.

In one example, an equalization plan is generated prior to performing equalization in accordance with determining a first replica partition, a second replica partition, a target master node, and a target slave node, as follows:

/>

where version represents the version, parts represents the partition, "topic" means "xxxx" represents the name of the topic, "part" means 0 the partition number of the partition under the topic, "parts" means [10167,11673] represents the node into which the duplicate partition is to be migrated.

An alterpartitionreports interface may be invoked, equalization plan parameters may be entered, and replica partition equalization of the partition of the subject of the node to be equalized may be performed. When the method is specifically executed, partition verification is firstly carried out, and after the verification is completed, an equalization scheme is written in a Zookeeper creation node "/admin/compare_parts". If the node exists, then there is ongoing balancing and the current round of balancing operation is aborted. The Controller node in the Kafka cluster registers a listener to monitor a Zookeeper, wherein the partitionReassurementListener is responsible for monitoring "/admin/reassessioncomponents" nodes, and the monitored partitionReassurementevent is transferred to an internal event queue, and is processed by a related thread for subsequent events. In the execution process, the current copy distribution list and the initial copy list are periodically acquired for comparison, and new and old copies exist in the copy list in the processing process at the same time, so that whether the whole equalization process is finished or not is judged. And calling the interface until the whole balancing process is finished, resetting the disk utilization rate and the weight of each node after balancing, and continuing the judgment of the next node to be balanced.

In the embodiment of the disclosure, in the equalization process, a situation that a certain node oscillates back and forth occurs. Aiming at the problem of balancing oscillation, a balancing plan needs to be estimated, the change of the disk utilization rate of the target slave node after balancing is estimated, if the predicted utilization rate of the target slave node does not meet the expected condition, balancing aiming at the second copy partition is not worth, and the reasonable utilization of resources is ensured by balancing under the condition that the condition is met.

In one example, as shown in FIG. 8, the disk load balancing architecture includes the following:

1. and obtaining the disk utilization rate.

Specifically, the disk usage of each node in the cluster may be obtained by referring to the method of the embodiment corresponding to fig. 5, which is not described herein.

2. The dual poll acquires the embedded node.

Specifically, the embedded node may be obtained with reference to the embodiments described in fig. 2 and fig. 4, which are not described herein.

3. And estimating the utilization rate after embedding.

Specifically, the usage rate of the embedded node after the embedding may be estimated with reference to the embodiments described in fig. 5 and fig. 6, which will not be described herein.

4. Request execution.

5. The request is listened to.

6. Equalization is performed.

7. The disk usage is refreshed.

Specifically, the steps 4-7 may be performed with reference to the embodiment described in fig. 7 for performing request execution, listening to requests, performing equalization, and refreshing the disk usage, which will not be described herein.

In another example, as shown in FIG. 9, the disk load balancing process includes:

obtaining a cluster node list, obtaining node disk utilization according to the node list, calculating the average value of the cluster disk utilization, calculating the variance of the cluster disk utilization when the average value of the cluster disk utilization is not greater than a threshold value, selecting a node when the variance of the cluster disk utilization is greater than the threshold value, and executing an equalization flow when the disk utilization of the node is greater than the sum of the average value of the cluster disk utilization and an upper threshold value.

The above disk load balancing process is described in the above embodiments, and will not be described herein.

In another example, as shown in fig. 10, the equalization process specifically includes:

obtaining a partition list of the node to be balanced, determining the partition storage size of the node to be balanced, traversing the copy partition of the node to be balanced, obtaining the partition number of the corresponding subject of the copy partition, obtaining the balancing plan of the copy partition, executing the balancing plan of the copy partition, and updating the disk utilization rate of the node after balancing is completed.

The above equalization process is described in the above embodiments, and will not be described here again.

In another example, as shown in fig. 11, the obtaining an equalization plan for a replica partition specifically includes:

obtaining a subject master node list and a duplicate distribution list, calculating a first bearing number threshold of the nodes, selecting unused nodes which do not reach the first bearing number threshold as duplicate partition allocation target master nodes, selecting available nodes which are not in the same rack attribute as the target master nodes and are unused as duplicate partition allocation target slave nodes, performing weighted polling when the conditions are not met, and allocating the target slave nodes to the duplicate partitions to complete the balance plan.

The disc load balancing device provided by the embodiment of the present disclosure is described below, and the disc load balancing device described below and the disc load balancing method described above may be referred to correspondingly.

The disclosure also provides a disk load balancing device, which is applied to a cluster deployed across racks, wherein the cluster comprises a plurality of nodes corresponding to the disks one by one, and the disks are respectively deployed on different racks;

the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of replica partitions;

The method comprises the steps that a plurality of duplicate partitions belonging to the same partition are respectively arranged in different nodes, one node is set as a master node of the partition, and other nodes are set as slave nodes of the partition;

as shown in fig. 12, the apparatus includes:

the information determining module 121 is configured to determine a topic master node list and a topic partition list of nodes to be balanced, where the topic master node list includes a plurality of master nodes belonging to a same topic, and the topic partition list of nodes to be balanced includes duplicate partition information set in the nodes to be balanced.

The target master node selection module 122 is configured to determine, based on the subject master node list and a master node selection rule, a target master node that corresponds to a first replica partition of a node to be balanced, where the first replica partition is determined by the subject partition list of the node to be balanced, and the first replica partition is a replica partition of a partition that uses the node to be balanced as a master node.

The target slave node selection module 123 is configured to determine, based on a disk usage rate of the disk and a slave node selection rule, a target slave node that corresponds to a second replica partition of a node to be balanced one by one, where the second replica partition is determined by the subject partition list of the node to be balanced, and the second replica partition is a replica partition that uses the node to be balanced as a partition of the slave node.

And the migration module 124 is configured to move the first copy partition of the node to be balanced to the corresponding target master node, and move the second copy partition of the node to be balanced to the corresponding target slave node.

According to the disk load balancing device provided in the embodiment of the present disclosure, the information determining module 121 is specifically configured to:

obtaining a topic list, wherein the topic list comprises topics of the cluster; determining topic information according to the topic list, wherein the topic information comprises a partition of the topic, a copy partition of the partition and a mapping relation between the copy partition and a node where the copy partition is located; and determining the topic master node list and the topic partition list of the nodes to be balanced according to the topic information.

According to the disk load balancing device provided in the embodiment of the present disclosure, the target master node selection module 122 is specifically configured to:

According to the disk load balancing apparatus provided in the embodiments of the present disclosure, the target master node selection module 122 is further configured to:

According to the disk load balancing device provided in the embodiment of the present disclosure, the target slave node selecting module 123 is specifically configured to:

According to the disk load balancing apparatus provided in the embodiment of the present disclosure, the target slave node selecting module 123 is further configured to:

According to the disk load balancing device provided in the embodiment of the present disclosure, the information determining module 121 is further configured to:

According to the disk load balancing device provided in the embodiment of the present disclosure, the migration module 124 is specifically configured to:

Fig. 13 illustrates a physical structure diagram of an electronic device, as shown in fig. 13, which may include: processor 1310, communication interface (Communications Interface) 1320, memory 1330 and communication bus 1340, wherein processor 1310, communication interface 1320, memory 1330 communicate with each other via communication bus 1340. The processor 1310 may call logic instructions in the memory 1330 to execute a disk load balancing method, where the method is applied to a cluster deployed across racks, where the cluster includes a plurality of nodes corresponding to the disks one to one, and the disks are deployed on different racks respectively; the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of replica partitions; the method comprises the steps that a plurality of duplicate partitions belonging to the same partition are respectively arranged in different nodes, one node is set as a master node of the partition, and other nodes are set as slave nodes of the partition; the method comprises the following steps: determining a topic master node list and a topic partition list of nodes to be balanced, wherein the topic master node list comprises a plurality of master nodes belonging to the same topic, and the topic partition list of the nodes to be balanced comprises copy partition information arranged on the nodes to be balanced; determining target master nodes corresponding to first replica partitions of nodes to be balanced one by one based on the topic master node list and a master node selection rule, wherein the first replica partitions are determined by the topic partition list of the nodes to be balanced, and the first replica partitions are replica partitions taking the nodes to be balanced as partitions of the master nodes; determining target slave nodes corresponding to second copy partitions of the nodes to be balanced one by one based on the disk utilization rate of the disk and a slave node selection rule, wherein the second copy partitions are determined by the topic partition list of the nodes to be balanced, and the second copy partitions are copy partitions taking the nodes to be balanced as partitions of the slave nodes; and moving the first copy partition of the node to be balanced to the corresponding target master node, and moving the second copy partition of the node to be balanced to the corresponding target slave node.

Further, the logic instructions in the memory 1330 can be implemented in the form of software functional units and can be stored in a computer readable storage medium when sold or used as a stand alone product. Based on such understanding, the technical solutions of the embodiments of the present disclosure may be essentially or, what contributes to the prior art, or part of the technical solutions, may be embodied in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In another aspect, the disclosure also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer readable storage medium, the computer program including program instructions that, when executed by a computer, are capable of performing the disk load balancing method provided by the methods described above, applied to a cluster deployed across racks, the cluster including a plurality of nodes in one-to-one correspondence with the disks, the disks being deployed on different racks, respectively; the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of replica partitions; the method comprises the steps that a plurality of duplicate partitions belonging to the same partition are respectively arranged in different nodes, one node is set as a master node of the partition, and other nodes are set as slave nodes of the partition; the method comprises the following steps: determining a topic master node list and a topic partition list of nodes to be balanced, wherein the topic master node list comprises a plurality of master nodes belonging to the same topic, and the topic partition list of the nodes to be balanced comprises copy partition information arranged on the nodes to be balanced; determining target master nodes corresponding to first replica partitions of nodes to be balanced one by one based on the topic master node list and a master node selection rule, wherein the first replica partitions are determined by the topic partition list of the nodes to be balanced, and the first replica partitions are replica partitions taking the nodes to be balanced as partitions of the master nodes; determining target slave nodes corresponding to second copy partitions of the nodes to be balanced one by one based on the disk utilization rate of the disk and a slave node selection rule, wherein the second copy partitions are determined by the topic partition list of the nodes to be balanced, and the second copy partitions are copy partitions taking the nodes to be balanced as partitions of the slave nodes; and moving the first copy partition of the node to be balanced to the corresponding target master node, and moving the second copy partition of the node to be balanced to the corresponding target slave node.

In yet another aspect, the present disclosure further provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, is implemented to perform the disk load balancing method provided by the foregoing, and is applied to a cluster deployed across racks, where the cluster includes a plurality of nodes corresponding to the disks one to one, and the disks are deployed on different racks respectively; the cluster includes at least one topic, the topic including at least one partition, the partition including a plurality of replica partitions; the method comprises the steps that a plurality of duplicate partitions belonging to the same partition are respectively arranged in different nodes, one node is set as a master node of the partition, and other nodes are set as slave nodes of the partition; the method comprises the following steps: determining a topic master node list and a topic partition list of nodes to be balanced, wherein the topic master node list comprises a plurality of master nodes belonging to the same topic, and the topic partition list of the nodes to be balanced comprises copy partition information arranged on the nodes to be balanced; determining target master nodes corresponding to first replica partitions of nodes to be balanced one by one based on the topic master node list and a master node selection rule, wherein the first replica partitions are determined by the topic partition list of the nodes to be balanced, and the first replica partitions are replica partitions taking the nodes to be balanced as partitions of the master nodes; determining target slave nodes corresponding to second copy partitions of the nodes to be balanced one by one based on the disk utilization rate of the disk and a slave node selection rule, wherein the second copy partitions are determined by the topic partition list of the nodes to be balanced, and the second copy partitions are copy partitions taking the nodes to be balanced as partitions of the slave nodes; and moving the first copy partition of the node to be balanced to the corresponding target master node, and moving the second copy partition of the node to be balanced to the corresponding target slave node.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are merely for illustrating the technical solution of the present disclosure, and are not limiting thereof; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present disclosure.

Claims

1. The disk load balancing method is characterized by being applied to clusters deployed across racks, wherein the clusters comprise a plurality of nodes corresponding to the disks one by one, and the disks are deployed on different racks respectively;

the method comprises the following steps:

2. The disk load balancing method according to claim 1, wherein the step of determining a subject master node list and a subject partition list of nodes to be balanced includes:

obtaining a topic list, wherein the topic list comprises topics of the cluster;

3. The disk load balancing method according to claim 1, wherein the step of determining a target master node corresponding to the first replica partition of the node to be balanced one-to-one based on the subject master node list and a master node selection rule comprises:

4. The disk load balancing method according to claim 3, wherein the step of determining a target master node corresponding to the first replica partition of the node to be balanced one-to-one based on the subject master node list and a master node selection rule, further comprises:

5. The disk load balancing method according to claim 3, wherein the first bearer number threshold is obtained by:

6. The disk load balancing method according to claim 1, wherein the step of determining a target slave node corresponding to the second replica partition of the node to be balanced one-to-one based on the disk usage of the disk and a slave node selection rule includes:

7. The disk load balancing method according to claim 6, wherein before the step of changing the attribute of the node corresponding to the target slave node from non-selected to selected, the method further comprises:

8. The disk load balancing method according to claim 1, wherein the node to be balanced is determined by the steps of:

9. The disk load balancing method according to claim 1, wherein the step of partitioning the first copy of the node to be balanced to the corresponding target master node includes:

10. The disk load balancing method according to claim 1, wherein the step of moving the second replica partition of the node to be balanced to the corresponding target slave node comprises:

11. The disk load balancing device is characterized by being applied to a cluster deployed across racks, wherein the cluster comprises a plurality of nodes corresponding to the disks one by one, and the disks are deployed on different racks respectively;

the device comprises:

12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the disk load balancing method of any one of claims 1 to 10 when the program is executed by the processor.

13. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor performs the steps of the disk load balancing method according to any of claims 1 to 10.