CN113608870A

CN113608870A - Load balancing method and device of message queue, electronic equipment and storage medium

Info

Publication number: CN113608870A
Application number: CN202110858352.0A
Authority: CN
Inventors: 刘彬
Original assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Current assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date: 2021-07-28
Filing date: 2021-07-28
Publication date: 2021-11-05

Abstract

The application relates to a load balancing method and device for a message queue, electronic equipment and a storage medium. The method comprises the following steps: acquiring the data load of each candidate partition in the target message queue; determining a high-load partition and a low-load partition in all the candidate partitions according to the data load of each candidate partition; according to the data load of the high-load partition and the data load of the low-load partition, a shunting scheme for shunting all messages to be written in a target message queue to be written is obtained; and writing the messages to be written into the candidate partitions in sequence according to the shunting scheme. By the method in the embodiment, the corresponding shunting scheme can be obtained according to the data load of each candidate partition, and the problem of faults caused by unbalanced data loads of different candidate intervals in the using process of the message queue can be effectively avoided by enabling the probability of writing the message to be written into the low-load partition to be higher than the probability of writing the message to be written into the high-load partition.

Description

Load balancing method and device of message queue, electronic equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for load balancing of a message queue, an electronic device, and a storage medium.

Background

The message queue is a middleware for message transmission and is commonly used in the scenes of decoupling, data forwarding, asynchronous communication and traffic peak clipping among various systems; the topic message delivery model supporting publish/subscribe (pub/sub) is a widely used message queue model capable of broadcasting, and comprises a producer, a topic (i.e. a message queue), a partition (partition) and a consumer (consumer).

In the related art, after a producer of data sends data to a subject, there is a problem that the loads of respective partitions in a message queue are unbalanced, and it is easy to cause a failure of the message queue during use in some cases.

Aiming at the technical problem of unbalanced load of each partition in the message queue in the related art, an effective solution is not provided at present.

Disclosure of Invention

In order to solve the technical problem of load imbalance of each partition in the message queue, the application provides a load balancing method and device of the message queue, an electronic device and a storage medium.

In a first aspect, an embodiment of the present application provides a load balancing method for a message queue, including:

acquiring the data load of each candidate partition in a target message queue, wherein the data load of each candidate partition is used for indicating the data volume of the message to be processed of the corresponding candidate partition;

determining a high-load partition and a low-load partition in all the candidate partitions according to the data load of each candidate partition, wherein the data load of the high-load partition is greater than or equal to a load threshold value, and the data load of the low-load partition is smaller than the load threshold value;

according to the data load of the high-load partition and the data load of the low-load partition, a shunting scheme for shunting all messages to be written in the target message queue is obtained;

and sequentially writing the messages to be written into the candidate partitions according to the shunting scheme, wherein one message to be written is only written into one candidate partition, and the probability of writing the messages to be written into the low-load partition according to the shunting scheme is higher than the probability of writing the messages to be written into the high-load partition.

Optionally, as in the foregoing method, the determining, according to the data load of each candidate partition, a high-load partition and a low-load partition in all the candidate partitions includes:

determining the candidate partition as the high-load partition under the condition that the data load of the candidate partition indicates that the current data traffic of the candidate partition is greater than or equal to a data traffic threshold value, or the data load of the candidate partition indicates that the used storage occupied by the messages stored in the candidate partition is greater than or equal to a storage threshold value;

determining the candidate partition as the low-load partition if the data load of the candidate partition indicates that the current data traffic of the candidate partition is less than the data traffic threshold and the data load of the candidate partition indicates that the used storage of the candidate partition is less than the storage threshold.

Optionally, as in the foregoing method, the obtaining, according to the data load of the high-load partition and the data load of the low-load partition, a splitting scheme for splitting a to-be-written message to be written into the target message queue includes:

determining a number of the high load partitions and a number of the low load partitions;

according to a preset corresponding relation, determining a first shunt ratio of all the high-load partitions and a second shunt ratio of all the low-load partitions according to the number of the high-load partitions and the number of the low-load partitions, wherein the first shunt ratio is used for indicating the probability of writing the message to be written into the high-load partitions, and the second shunt ratio is used for indicating the probability of writing the message to be written into the low-load partitions;

determining a third shunt proportion of each high-load partition according to the first shunt proportion and the number of the high-load partitions, wherein the third shunt proportion is used for indicating the probability of writing the message to be written into the corresponding high-load partition;

and determining a fourth shunt proportion of each low-load partition according to the second shunt proportion and the data load of each low-load partition, wherein the fourth shunt proportion is used for indicating the probability of writing the message to be written into the corresponding low-load partition.

Optionally, as in the foregoing method, the determining a fourth split ratio of each low-load partition according to the second split ratio and the data load of each low-load partition includes:

determining a total available data load for each of the low-load partitions;

determining the data load to total available data load ratio of each of the low load partitions;

and obtaining a fourth shunt proportion of each low-load partition according to the proportional relation between the corresponding specific values of the low-load partitions and the second shunt proportion.

Optionally, as in the foregoing method, before the obtaining the data load of each candidate partition in the target message queue, the method further includes:

acquiring capacity expansion information for indicating the target message queue to perform partition capacity expansion in a historical time period, wherein the historical time period is a time period within a preset time length from a current time point;

and triggering and executing a jump operation according to the capacity expansion information, wherein the jump operation is used for jumping to the step of acquiring the data load of each candidate partition in the target message queue.

Optionally, as in the foregoing method, after the determining, according to the data load of each candidate partition, a high-load partition and a low-load partition in all the candidate partitions, the method further includes:

under the condition that all the candidate partitions are determined to be the high-load partitions, generating a target command for performing partition capacity expansion on the target message queue;

and adding a new partition in the target message queue according to the target command.

Optionally, as in the foregoing method, after the sequentially writing the messages to be written into the candidate partitions according to the splitting scheme, the method further includes:

acquiring real-time data loads of the candidate partitions according to a preset period;

and updating the metadata information of the target message queue according to the real-time data load.

In a second aspect, an embodiment of the present application provides a load balancing apparatus for a message queue, including:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring the data load of each candidate partition in a target message queue of a target message queue, and the data load of each candidate partition is used for indicating the data volume of the message to be processed of the corresponding candidate partition;

a determining module, configured to determine a high-load partition and a low-load partition in all the candidate partitions according to a data load of each candidate partition, where a data load of the high-load partition is greater than or equal to a load threshold, and a data load of the low-load partition is smaller than the load threshold;

the distribution module is used for obtaining a distribution scheme for distributing all messages to be written in the target message queue according to the data load of the high-load partition and the data load of the low-load partition;

and a writing module, configured to sequentially write each message to be written into each candidate partition according to the splitting scheme, where one message to be written is only written into one candidate partition, and a probability that the message to be written is written into the low-load partition according to the splitting scheme is higher than a probability that the message to be written is written into the high-load partition.

In a third aspect, an embodiment of the present application provides an electronic device, including: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

the memory is used for storing a computer program;

the processor, when executing the computer program, is configured to implement the method according to any of the preceding claims.

In a fourth aspect, the present application provides a computer-readable storage medium, which includes a stored program, where the program is executed to perform the method according to any one of the preceding claims.

By the method in the embodiment, the corresponding shunting scheme can be obtained according to the data load of each candidate partition, and according to the shunting scheme, the probability of writing the message to be written into the low-load partition is higher than the probability of writing the message to be written into the high-load partition, so that the message to be written which is less than that of the low-load partition is written into the high-load partition, the purpose that the data load of all the messages to be written on the high-load partition is lower than that of all the messages to be written on the low-load partition is achieved, and on the basis that the data load in each candidate partition is reduced along with the consumption progress and the outdated data is gradually cleaned, the problem of faults caused by unbalanced data loads of different candidate partitions in the using process of the message queue can be effectively avoided.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.

Fig. 1 is a flowchart of a load balancing method for a message queue according to an embodiment of the present disclosure;

fig. 2 is a flowchart of a method for load balancing of a message queue according to another embodiment of the present application;

fig. 3 is a flowchart of a method for load balancing of a message queue according to another embodiment of the present application;

fig. 4 is a block diagram of a load balancing apparatus for a message queue according to an embodiment of the present disclosure;

fig. 5 is a diagram of a message queue architecture according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

According to one aspect of the embodiment of the application, a load balancing method for a message queue is provided. Alternatively, in this embodiment, the load balancing method for the message queue may be applied to a hardware environment formed by one or more servers. The server is connected with the terminal through a network, can be used for providing services for the terminal or a client installed on the terminal, and can be provided with a database on the server or independent of the server for providing data storage services for the server.

The network may include, but is not limited to, at least one of: wired networks, wireless networks. The wired network may include, but is not limited to, at least one of: wide area networks, metropolitan area networks, local area networks, which may include, but are not limited to, at least one of the following: WIFI (Wireless Fidelity), bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, and the like.

Taking the server to execute the load balancing method of the message queue in this embodiment as an example, fig. 1 is a load balancing method of the message queue provided in this embodiment, and includes the following steps:

step S101, acquiring a data load of each candidate partition in a target message queue of the target message queue, wherein the data load of each candidate partition is used for indicating the data volume of the message to be processed of the corresponding candidate partition.

The load balancing method for the message queue in this embodiment may be applied to a scenario in which data loads of different candidate partitions (i.e., partitions) in a topic (topic, i.e., message queue) in a topic message transfer model are balanced under the condition that the data loads of the different candidate partitions are different.

The target message queue may be topic that needs to be load balanced. The target message queue may be deployed in a cluster formed by a plurality of nodes, a plurality of candidate partitions may be provided in the target message queue, each candidate partition is provided in one node, and each node may be provided with a plurality of candidate partitions, each candidate partition corresponds to one folder on a physical disk, and all messages and index files of the candidate partition are stored under the folder. For example, there are 3 candidate partitions below a certain topic, and the 3 candidate partitions are in three different nodes respectively. After all candidate partitions of the target message queue are determined, the data load of each candidate partition may be determined from the metadata information of the target message queue.

The data load may be a data amount indicating a message to be processed by the corresponding candidate partition; for example, when there are 3 candidate partitions, there is a data load corresponding to each candidate partition, and then the data amount of the message to be processed for each candidate partition may be obtained based on the data load; the data payload may include messages that have been written into the candidate partition, yet to be consumed, and may be characterized by the amount of data of all messages yet to be consumed, e.g., 500 GB; messages to be written into the candidate partition may also be included, which may be characterized by the number of messages written per second, e.g., 1000qps (queries-per-second, number of requests per second).

Step S102, according to the data load of each candidate partition, determining a high-load partition and a low-load partition in all the candidate partitions, wherein the data load of the high-load partition is greater than or equal to a load threshold value, and the data load of the low-load partition is smaller than the load threshold value.

After determining the data load for each candidate partition, a high load partition and a low load partition may be determined among all candidate partitions. And high-load partitions and low-load partitions can be determined through a preset load threshold value.

The load threshold may be a predetermined threshold for determining whether the data load is a high load. For example, when only one threshold a with the type of A is included in the load thresholds, the load of the type of A in the candidate partition is a ', when a ' ≧ a, the candidate partition is determined to be a high-load partition, and when a ' < a, the candidate partition is determined to be a low-load partition.

Step S103, according to the data load of the high-load partition and the data load of the low-load partition, a shunting scheme for shunting all messages to be written in the target message queue to be written is obtained.

After determining the data load of the high-load partition and the data load of the low-load partition, it is therefore necessary to determine a splitting scheme for splitting all messages to be written to the target message queue, so as to reduce the data load of the high-load partition.

The message to be written, which may be generated by a producer (producer), needs to be written to the message in the target message queue. In general, a producer may include one or more than one producer, and may generate a plurality of messages to be written, where each message to be written may be written in the target message queue in the form of a data stream, so that the data traffic (e.g., the number of messages to be written per second) written in each candidate partition of the target message queue may be controlled by determining a splitting scheme for splitting all messages to be written.

The splitting scheme may be a scheme for splitting all to-be-written messages in the write target message queue into different candidate partitions; for example, when candidate partition I, candidate partition II, and candidate partition III exist in the target message queue, and candidate partition I and candidate partition II are high-load partitions, and candidate partition III is a low-load partition, the offloading scheme may be a scheme for controlling data traffic of a message to be written to candidate partition I or candidate partition II, which is less than data traffic of a message to be written to candidate partition III.

And step S104, writing the messages to be written into the candidate partitions in sequence according to the shunting scheme, wherein one message to be written is only written into one candidate partition, and the probability of writing the message to be written into the low-load partition according to the shunting scheme is higher than the probability of writing the message to be written into the high-load partition.

After the distribution scheme is determined, each message to be written can be written into each candidate partition according to the distribution scheme, and each message to be written is written into only one candidate partition. Since the high-load partition and the low-load partition are determined according to the foregoing steps, and the offloading scheme is to reduce the data traffic written to the high-load partition, the probability that the message to be written is written to the low-load partition according to the offloading scheme is higher than the probability that the message to be written is written to the high-load partition.

Optionally, for example, when the candidate partition 1, the candidate partition 2, and the candidate partition 3 are all low-load partitions, the message to be written may be polled and written into the positive candidate partition 1, the candidate partition 2, and the candidate partition 3, and when the candidate partition 1 and the candidate partition 2 are low-load partitions and the candidate partition 3 is a high-load partition, each message to be written may be written into the target message queue according to the rhythm of the candidate partition 1, the candidate partition 2, and the candidate partition 3, so that the probability that the message to be written is written into the low-load partition is higher than the probability that the message to be written into the high-load partition, and the data traffic of the message to be written into the candidate partition 3 is reduced.

When the candidate partition 1 and the candidate partition 2 are low-load partitions and the candidate partition 3 is a high-load partition, a random integer within 1 to 10 may be configured for each partition, the candidate partition 1 is (1-4), the candidate partition 2 is (5-8), and the candidate partition 3 is (9-10), and when a to-be-written message N to be written into the target message queue is obtained, a random integer within 1 to 10 is randomly allocated to the to-be-written message N, and the to-be-written message N is written into the corresponding candidate partition according to the allocated random integer, for example, when the random integer allocated to the to-be-written message N is 7, the to-be-written message N is written into the corresponding candidate partition 2; thus, the probability that the message to be written will be randomly written to partition 1 and partition 2 will be greater than the probability of writing to partition 3.

As an alternative implementation, as shown in fig. 2, in the foregoing method, the step S102 of determining, according to the data load of each candidate partition, a high-load partition and a low-load partition in all candidate partitions includes the following steps:

step S201, when the data load of the candidate partition indicates that the current data flow of the candidate partition is greater than or equal to a data flow threshold value, or the data load of the candidate partition indicates that the used memory space occupied by the message stored in the candidate partition is greater than or equal to a memory space threshold value, determining the candidate partition as a high-load partition;

in step S202, when the data load of the candidate partition indicates that the current data traffic of the candidate partition is smaller than the data traffic threshold, and the data load of the candidate partition indicates that the used storage capacity of the candidate partition is smaller than the storage capacity threshold, the candidate partition is determined as the low-load partition.

For each candidate partition, the data payload may include the amount of data traffic currently written to the candidate partition (e.g., the number of messages written to the candidate partition per second), and the amount of used storage occupied by messages stored in the candidate partition.

In general, the higher the current data flow of a candidate partition is, the more data volume continuously written into the candidate partition is indicated, so after the current data flow of the candidate partition is determined, whether the candidate partition is a high-load partition may be determined according to the size relationship between the current data flow and the data flow threshold.

The data traffic threshold may be a preset value for indicating that the candidate partition corresponding to the current data traffic is a high-load partition when the current data traffic is greater than or equal to the data traffic threshold.

On the other hand, the total available storage capacity of each candidate partition is limited, when the currently used storage capacity of the candidate partition is higher, the efficiency of reading and writing the candidate partition is poorer, and when the currently used storage capacity of the candidate partition is higher, it is indicated that more messages written by the candidate partition exceed the consumed messages, so after the used storage capacity occupied by the messages stored in the candidate partition is determined, whether the candidate partition is a high-load partition can be judged according to the size relationship between the used storage capacity and the storage capacity threshold value.

The storage threshold may be a preset value for indicating that the candidate partition corresponding to the used storage is a high-load partition in a case that the used storage is greater than or equal to the storage threshold.

On the other hand, if the data load of a candidate partition indicates that the current data traffic of the candidate partition is smaller than the data traffic threshold value, and the data load of the candidate partition indicates that the used storage amount of the candidate partition is smaller than the storage amount threshold value, the condition that the current data traffic and the used storage amount of the candidate partition do not meet the requirement of being a high-load partition is described, and therefore the candidate partition is determined to be a low-load partition.

For example, there are three partitions in total, i.e., candidate partition 1, candidate partition 2, and candidate partition 3, and the data traffic threshold is 10000qps and the storage threshold is 500GB in three different nodes, respectively. Candidate partition 1 has current data traffic and used storage of (1000qps,400GB), candidate partition 2 has current data traffic and used storage of (11000qps,400GB), candidate partition 3 has current data traffic and used storage of (5000qps,600GB), candidate partition 1 has current data traffic less than a data traffic threshold and used storage less than a storage threshold, and therefore candidate partition 1 is a low-load partition; the current data flow of the candidate partition 2 is greater than the data flow threshold, so although the used memory capacity of the candidate partition 2 is less than the memory capacity threshold, the candidate partition 2 is still a high-load partition; the used memory of the candidate partition 3 is greater than the memory threshold, so the candidate partition 3 is still a high load partition although the current data traffic is less than the data traffic threshold.

By the method in the embodiment, whether the candidate partition is a high-load partition or a low-load partition can be quickly determined through two dimensions according to the numerical relationship between the current data flow and the data flow threshold value and the numerical relationship between the used storage capacity and the storage capacity threshold value.

As an optional implementation manner, as in the foregoing method, the step S103 of obtaining, according to the data load of the high-load partition and the data load of the low-load partition, a splitting scheme for splitting a to-be-written message to be written into the target message queue includes the following steps:

in step S301, the number of high load partitions and the number of low load partitions are determined.

After determining the high-load partitions and the low-load partitions of all the candidate partitions, the number of high-load partitions and the number of low-load partitions may be determined.

Step S302, according to a preset corresponding relation, according to the number of the high-load partitions and the number of the low-load partitions, determining first shunt proportions of all the high-load partitions and second shunt proportions of all the low-load partitions, wherein the first shunt proportions are used for indicating the probability of writing the message to be written into the high-load partitions, and the second shunt proportions are used for indicating the probability of writing the message to be written into the low-load partitions.

After the number of the high-load partitions and the number of the low-load partitions are determined, according to a preset corresponding relationship, the first shunt proportions of all the high-load partitions and the second shunt proportions of all the low-load partitions may be determined according to the number of the high-load partitions and the number of the low-load partitions.

The first shunting proportion is the proportion of all messages to be written in all high-load partitions in the messages to be written.

The second shunting proportion is the proportion of all messages to be written in all low-load partitions to all messages to be written.

The preset correspondence may include a correspondence between the number P of high-load partitions and the number Q of low-load partitions and the first split ratios J of all high-load partitions and the second split ratios K of all low-load partitions, i.e., each group (P, Q) has a unique corresponding (J, K); for example, the preset correspondence may be ((1,2), (0.2,0.8)), that is, when P is 1, Q is 2, J is 0.2, and K is 0.8, 2 of 10 messages to be written are written into the high-load partition and 8 are written into the low-load partition; the preset correspondence may be ((1,3), (0.1,0.9)), that is, when P is 1, Q is 3, J is 0.1, and K is 0.9, 1 of 10 messages to be written is written into the high-load partition and 9 is written into the low-load partition.

Step S303, determining a third offload proportion of each high-load partition according to the first offload proportion and the number of the high-load partitions, where the third offload proportion is used to indicate a probability of writing the message to be written into the corresponding high-load partition.

After the first split ratio and the number of high-load partitions are obtained, the third split ratio of each high-load partition can be determined by the quotient of the first split ratio and the number of high-load partitions.

Step S304, determining a fourth shunting proportion of each low-load partition according to the second shunting proportion and the data load of each low-load partition, where the fourth shunting proportion is used to indicate a probability of writing the message to be written into the corresponding low-load partition.

After the second split ratio and the number of low-load partitions are obtained, the fourth split ratio of each low-load partition can be obtained by the quotient of the second split ratio and the number of low-load partitions.

As an alternative implementation, as the foregoing method, determining the fourth split ratio of each low-load partition according to the second split ratio and the data load of each low-load partition may further include the following steps:

in step S401, the total available data load of each low-load partition is determined.

After each low load partition is determined, the total available data load for each low load partition may be determined. Optionally, the total load of available data may include: the highest qps, and the amount of available storage, e.g., the highest data traffic is 20000qps, the amount of available storage is 1000GB, etc.

Step S402, determining the ratio of the data load of each low-load partition to the total available data load.

After determining the data load of each low-load partition and the total data load available for each low-load partition, the data load to total data load available ratio for each low-load partition may be determined.

Optionally, as can be seen from the foregoing embodiment, the data load may include sub-loads of multiple dimensions (i.e., current data traffic, used storage amount), and when the sub-load of a dimension meets the condition of a high-load partition, the candidate partition is the high-load partition, so that the ratio of the sub-load of each dimension in the low-load partition to the highest available load of the dimension may be determined first, and then the largest ratio may be selected as the total data load ratio of the data load of the low-load partition to the available data. For example, when the current data traffic of a low-load partition D is 10000qps and the used storage capacity is 800GB, the proportion of the current data traffic to the highest data traffic is 50%, and the proportion of the used storage capacity to the total available storage capacity is 80%; therefore, the ratio of the data load of the low-load partition to the total available storage is recorded as 80%.

In addition, the ratio can be obtained after weighting the ratios corresponding to the sub-loads of each dimension. For example, when the weight of the data traffic dimension and the weight of the storage volume dimension are 0.5, the ratio of the low-load partition D is (50% × 0.5) + (80% × 0.5) ═ 65%.

Step S403, obtaining a fourth shunt ratio of each low-load partition according to the proportional relationship between the ratios corresponding to the low-load partitions and the second shunt ratio.

After each low-load partition is calculated according to the method, the ratio corresponding to each low-load partition can be obtained, and then the proportional relation among the ratios can be obtained through calculation.

The proportional relationship between the ratios can be obtained by dividing the ratios, for example, when there is a low-load section D1 (ratio of 50%), a low-load section D2 (ratio of 80%), and the resulting proportional relationship is 5/8 for 50%/80%. After the proportional relationship and the second shunt proportions of all the low-load partitions are obtained, a fourth shunt proportion of each low-load partition can be obtained.

The method for obtaining the fourth shunt ratio of each low-load partition may be that, according to the proportional relationship, the fourth shunt ratio allocated to the low-load partition with a small ratio is larger than the fourth shunt ratio allocated to the low-load partition with a large ratio; the split fraction of each low-load partition may be determined according to a proportional relationship, and then the fourth split fraction of each low-load partition may be obtained according to a product between the split fraction and the second split fraction. For example, when there is a proportional relationship between the low-load section D1 (ratio of 50%) and the low-load section D2 (ratio of 80%) of 0.5/0.8, and the second split ratio is 0.8; firstly, the split ratio of the low-load partition D1 is calculated to be (1-0.5)/(1-0.8) to 5/(5+ 2); the split ratio of the low-load partition D2 is 2/(5+ 2); then, the fourth split ratio of the low-load partition D1 is calculated to be (4/5) × (5/7) ═ 4/7; the fourth split ratio of the low-load partition D2 is (4/5) × (2/7) ═ 8/35. And may assign a larger fourth split fraction to low-load partition D1, which may otherwise be less data-loaded.

By the method in the embodiment, the shunting can be further refined according to the data load of each low-load partition, so that the aim of better load balancing can be fulfilled.

As shown in fig. 3, as an alternative implementation, before the step S101 acquires the data load of each candidate partition in the target message queue, the method further includes the following steps:

step S501, obtaining capacity expansion information for indicating that the target message queue performs partition capacity expansion in a historical time period, wherein the historical time period is a time period within a preset time length from a current time point.

After the target message queue is subjected to partition expansion in the historical time period, at least one new partition is added to the target message queue compared with the previous historical time period. The related information of partition expansion each time can be recorded to obtain expansion information, and the expansion information is information for indicating the target message queue to perform partition expansion in the historical time period, so that the expansion information may include related information of candidate partitions obtained by partition expansion, partition expansion time, and the like.

The historical time period may be a time period within a preset time period from the current time point, and the preset time period may be a preset time period, for example, 1 hour, 10 minutes, and the like. After the current time point and the preset time length are determined, the historical time period can be determined by tracing the preset time length forward at the current time point, for example, when the current time point is 12:00:00, and the preset time length is 10 minutes, the historical time period is (11:50:00-12:00: 00).

After determining that the capacity expansion information for performing partition capacity expansion exists in the historical time period, it can be determined that at least one new candidate partition is obtained in the historical time period through partition capacity expansion.

And step S502, triggering and executing a jump operation according to the capacity expansion information, wherein the jump operation is used for jumping to a step of acquiring the data load of each candidate partition in the target message queue.

After the capacity expansion information is obtained, it can be determined that at least one new candidate partition occurs in the historical time period relative to the historical time period; in the current message queue scheme, after the partition capacity expansion, the flow of other partitions with high load cannot be guaranteed to be transferred to the partition obtained by partition capacity expansion, and even if a new partition is built, the pressure of the partition with heavy load cannot be relieved, so that the data balance of the partition is realized; therefore, the jump operation is triggered and executed according to the capacity expansion information, and the step S101 is skipped to perform load balancing on each candidate partition in the target message queue after the partition capacity expansion.

By the method in the embodiment, load balancing can be automatically performed by judging whether partition expansion is performed, and then it can be effectively ensured that the flow of the candidate partition with high load is transferred to the candidate partition obtained by partition expansion after partition expansion, so that the purpose of good load balancing is achieved.

As an alternative implementation manner, as in the foregoing method, after determining, in step S102, a high-load partition and a low-load partition in all candidate partitions according to the data load of each candidate partition, the method further includes the following steps:

step S601, generating a target command for performing partition expansion on the target message queue under the condition that all the candidate partitions are determined to be high-load partitions.

And after the high-load partition and the low-load partition are determined in all the candidate partitions, the current overall data load condition of the target message queue can be obtained. Under the condition that all candidate partitions are high-load partitions, that is, no low-load partition exists, it can be determined that the data load of each candidate partition of the target message queue exceeds the load threshold, and therefore partition expansion needs to be performed on the target message queue to switch part of the traffic to pass through a new partition, and the data load of each high-load partition is further reduced. And may be expanded by generating a target command for partitioning the target message queue.

The target command may be a command that can trigger the addition of a new partition in the target message queue.

Step S602, at least one new partition is newly added in the target message queue according to the target command.

After the target command is generated, at least one new partition may be newly added by executing the target command and performing partition expansion on the target message queue. Furthermore, the corresponding relation between the number of the new partitions and the overall load of the message queue can be preset, the overall target load of the target message queue can be obtained according to the data load of each high-load partition, and the number of the new partitions needing to be newly added can be obtained according to the corresponding relation and the overall target load matching.

By the method in the embodiment, new partitions can be automatically triggered and added under the condition that all candidate partitions are high-load partitions, so that the purpose of load balancing can be further achieved.

As an optional implementation manner, as in the foregoing method, after the step S104 sequentially writes the messages to be written into the candidate partitions according to the split scheme, the method further includes the following steps:

step S701, acquiring real-time data loads of the candidate partitions according to a preset period.

Because the data load of each candidate partition can change along with the change of the flow of the written messages and the influence of the number of the consumed messages and other factors, the real-time data load of each candidate partition is obtained according to a preset period.

The preset period may be a preset time interval for periodically acquiring the real-time data load. For example, when the preset period is 1 minute, the real-time data load of each candidate partition is acquired every 1 minute.

For each candidate partition, there is a one-to-one correspondence between real-time data load and the point in time of acquisition.

Step S702, updating the metadata information of the target message queue according to the real-time data load.

After the real-time data load is acquired, the metadata information of the target message queue can be updated through the real-time data load.

The metadata information may record the distribution of the target message queue and the candidate partition (e.g., the location information corresponding to each candidate partition), and the latest load condition (including real-time data load) of each candidate partition.

By updating the metadata information, data basis can be provided for a subsequent shunting scheme for load balancing, and the obtained shunting scheme can better accord with the load condition of each current candidate partition through real-time data load, so that the purpose of better load balancing is achieved.

As shown in fig. 4, according to an embodiment of another aspect of the present application, there is also provided a load balancing apparatus for a message queue, including:

the system comprises an acquisition module 1, a processing module and a processing module, wherein the acquisition module is used for acquiring the data load of each candidate partition in a target message queue of a target message queue, and the data load of each candidate partition is used for indicating the data volume of a message to be processed of the corresponding candidate partition;

the determining module 2 is configured to determine a high-load partition and a low-load partition in all candidate partitions according to the data load of each candidate partition, where the data load of the high-load partition is greater than or equal to a load threshold, and the data load of the low-load partition is smaller than the load threshold;

the shunting module 3 is used for obtaining a shunting scheme for shunting all messages to be written in the target message queue to be written according to the data load of the high-load partition and the data load of the low-load partition;

and the writing module 4 is configured to sequentially write each message to be written into each candidate partition according to a split scheme, where one message to be written is only written into one candidate partition, and a probability of writing the message to be written into a low-load partition according to the split scheme is higher than a probability of writing the message to be written into a high-load partition.

The load balancing device may be a management module (manager) that is managed by one partition and is disposed in a target message queue (i.e., a topic), the management module may have one each topic in each broker (a node for brokering and storing messages), and the management module is mainly responsible for monitoring and aggregating information of the topics and partitions (i.e., candidate partitions); when a plurality of broker nodes exist in the cluster, summarizing the data load of each candidate partition sent by the management module of each topic in the cluster through heartbeat messages by designating a leader module (leader) in all the management modules; and further managing the load balance of all partitions in the cluster.

For example, as shown in fig. 5, a message queue architecture diagram obtained by the apparatus of this embodiment is shown, when there are

brokers

1,2, and 3; and topoc P0 in brooker 1, topoc P1 in brooker 2, and topoc P2 in brooker 3; and a management module (manager) in the topic P1 is used as a leader to summarize load conditions of each partition in the topic P0, the topic P1 itself, and the topic P2 (that is, to obtain data load of each partition), so that when it is determined that the partition load of a certain topic is high according to information (including current data traffic and used storage of each partition in the topic) reported by the manager in each topic, a offloading scheme corresponding to the topic is obtained, and load balancing of each partition in the topic is realized according to the offloading scheme.

Specifically, the specific process of implementing the functions of each module in the apparatus according to the embodiment of the present invention may refer to the related description in the method embodiment, and is not described herein again.

According to another embodiment of the present application, there is also provided an electronic apparatus including: as shown in fig. 6, the electronic device may include: the system comprises a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502 and the memory 1503 complete communication with each other through the communication bus 1504.

A memory 1503 for storing a computer program;

the processor 1501 is configured to implement the steps of the above-described method embodiments when executing the program stored in the memory 1503.

The bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

The embodiment of the present application further provides a computer-readable storage medium, where the storage medium includes a stored program, and when the program runs, the method steps of the above method embodiment are executed.

It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for load balancing of message queues is characterized by comprising the following steps:

2. The method of claim 1, wherein the determining the high-load partition and the low-load partition among all the candidate partitions according to the data load of each of the candidate partitions comprises:

3. The method according to claim 1, wherein obtaining, according to the data load of the high-load partition and the data load of the low-load partition, a splitting scheme for splitting a message to be written to the target message queue comprises:

4. The method of claim 3, wherein determining the fourth split fraction for each of the low-load partitions according to the second split fraction and the data load for each of the low-load partitions comprises:

determining a total available data load for each of the low-load partitions;

5. The method of claim 1, wherein prior to obtaining the data load for each candidate partition in the target message queue, the method further comprises:

6. The method of claim 1, wherein after determining the high-load partition and the low-load partition in all of the candidate partitions according to the data load of each of the candidate partitions, the method further comprises:

7. The method according to claim 1, wherein after said sequentially writing each of the messages to be written into each of the candidate partitions according to the splitting scheme, the method further comprises:

8. An apparatus for load balancing message queues, comprising:

the acquisition module is used for acquiring the data load of each candidate partition in a target message queue, wherein the data load of each candidate partition is used for indicating the data volume of the message to be processed of the corresponding candidate partition;

9. An electronic device, comprising: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

the memory is used for storing a computer program;

the processor, when executing the computer program, implementing the method of any of claims 1 to 7.

10. A computer-readable storage medium, characterized in that the storage medium comprises a stored program, wherein the program when executed performs the method of any of the preceding claims 1 to 7.