CN111897656B - Main queue load balancing method based on mirror image queue - Google Patents
Main queue load balancing method based on mirror image queue Download PDFInfo
- Publication number
- CN111897656B CN111897656B CN202010784404.XA CN202010784404A CN111897656B CN 111897656 B CN111897656 B CN 111897656B CN 202010784404 A CN202010784404 A CN 202010784404A CN 111897656 B CN111897656 B CN 111897656B
- Authority
- CN
- China
- Prior art keywords
- queue
- main
- queues
- message queue
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
A load balancing method of a main queue based on a mirror image queue is provided with a mechanism for dynamically balancing the main queue, and actively transfers the main queue according to a set rule by automatically acquiring the number of the main queues in a node; the invention can autonomously find the distribution quantity of the main queues in the nodes, smoothly transfer the main queues in the nodes with more distribution quantity of the main queues to the nodes with less distribution quantity of the main queues, and completely cover the scene of unbalanced load of all the main queues; the main queue balancing method is suitable for the pressure imbalance phenomenon caused by the downtime recovery of the nodes, the addition of new nodes and the release of partial message queues after use, and can automatically balance the main queue distribution on the nodes of the message queue cluster.
Description
Technical Field
The invention relates to the field of data processing, in particular to a main queue load balancing method based on a mirror image queue.
Background
Under the current mirror image queue mode of the RabbitMQ cluster, each message queue consists of a main queue and a plurality of slave queues; under normal conditions, the main queue provides message service, the slave queues back up messages, if the main queue of one message queue is abnormal, the message queue cluster can select a new main queue from other nodes; initially, a node has a small number of master queues and relatively many slave queues, and if a new message queue joins the cluster, the master queue of the message queue automatically joins the node with the least current master queue, and the slave queue joins other nodes.
However, the mirror image queue mode still has the following problem that when a node is abnormal, for example, in a down or node restart scenario, the main queue of the node is automatically dispersed to a normal node, but after the node is restored to a normal state, the main queue of the original node is still distributed on the current node, and cannot be automatically migrated and returned to the original node; in addition, in the capacity expansion state, when a new node joins the message queue cluster, the message queues on the nodes with a large number of main queues on the original node cannot be dispersed to the new node. This makes it impossible to share the operating pressure over a longer period of time.
In the mirror image queue mode in the prior art, the newly generated main queue can be automatically balanced, but the existing old main queue cannot be dynamically balanced, so that the pressure of the original node cannot be shared.
Disclosure of Invention
The invention aims to provide a load balancing method of a main queue based on a mirror image queue, so as to solve the problems in the prior art.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
a load balancing method for a main queue based on a mirror image queue comprises the following steps:
s1, a main queue and a slave queue are arranged in the message queue, the message queue is stored in the node, and the node in which the message queue is stored forms a message queue cluster;
s2, acquiring node information in the message queue cluster, acquiring node composition in the message queue through a command, and counting the number of the main queues stored in each node;
s3, calculating the maximum number of the primary queues expected to be stored in the nodes according to the total number of the nodes and the total number of the primary queues counted in the step S2;
s4, selecting the node with the maximum number of the stored main queues and the node with the minimum number of the stored main queues from the step S2, and comparing the number of the main queues in the node with the maximum number of the stored main queues with the maximum number of the main queues which are calculated and expected to be stored in the step S3; when the number of the main queues in the nodes is larger than the maximum number of the main queues expected to be stored, randomly selecting one main queue from the nodes with the maximum number of the stored main queues to migrate to the node with the minimum number of the stored main queues; otherwise, ending the balance;
and S5, after the message queue cluster is migrated through the main queue in the step S4, repeating the loop from the step S2 to the step S4 until the comparison result in the step S4 shows that the number of the main queues in the nodes storing the maximum number of the main queues is less than or equal to the maximum number expected to be stored, and ending the load balance of the message queue cluster.
Preferably, the method for calculating the maximum number of the primary queues expected to be stored in the node of step S3 includes:
the maximum number of expected stored primary queues is (total number of primary queues/total number of nodes) + (total number of primary queues% total number of nodes).
Preferably, in step S4, the migration method of the primary queue between the nodes is as follows:
s401, selecting one message queue from the nodes with the maximum number of the stored main queues, and setting a temporary strategy of the message queue: setting the message queue as a copy 1, and executing the message queue to enable the temporary policy to take effect;
s402, migrating the main queue in the message queue selected in the step S401 from the node with the maximum number of the stored main queues, and setting a temporary strategy of the message queue: setting the node of the message queue as a migration target node; executing the message queue to enable the temporary policy of the message queue to take effect;
s403, the main queue of the selected message queue in the step S401 is migrated to the node with the minimum number of the stored main queues, and the temporary strategy in the step S401 and the step S402 is cleared, so that the strategy of the message queue cluster is restored to a normal state;
s404, the slave queue in the message queue selected in the step S401 is migrated to the nodes except the node with the minimum number of the stored main queues, and the migration of one main queue is finished;
s405, the message queue cluster migrated in steps S401 to S404 returns to step S2 again, and the process loops.
Preferably, there is one and only one master queue in the same message queue, and the master queue and the slave queue of the same message queue are stored in different nodes.
The invention has the beneficial effects that: the invention discloses a load balancing method of a main queue based on a mirror image queue, which is characterized in that a mechanism for dynamically balancing the main queue is arranged, the number of the main queues in a node is automatically acquired, and the main queues are actively transferred according to a set rule; the invention can autonomously find the distribution quantity of the main queues in the nodes, smoothly transfer the main queues in the nodes with more distribution quantity of the main queues to the nodes with less distribution quantity of the main queues, and completely cover the scene of unbalanced load of all the main queues; the main queue balancing method is suitable for the pressure imbalance phenomenon caused by the downtime recovery of the nodes, the addition of new nodes and the release of partial message queues after use, and can automatically balance the main queue distribution in the nodes where the message queue clusters are positioned.
Drawings
FIG. 1 is a flow diagram of primary queue load balancing;
FIG. 2 is a message queue profile for an initial state node;
FIG. 3 is a diagram of a message queue profile for an emigration state node;
FIG. 4 is a diagram of a message queue distribution for a migration status node;
fig. 5 is a diagram of a message queue profile for a migrated node.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
A load balancing method of a main queue based on a mirror image queue is based on a message arrangement mode under the current mirror image queue mode; the method for load balancing of the main queue, as shown in fig. 1, includes the following steps:
s1, storing the message in a message queue, wherein the message queue comprises a main queue and a plurality of slave queues, and the main queue and the slave queues of the same message queue are in different nodes; the nodes storing the message queues form a message queue cluster;
s2, obtaining the node composition in the message queue cluster through a command, and counting the number of the main queues stored in each node;
s3, calculating the maximum number of the main queues expected to be stored in the nodes according to the total number of the nodes and the total number of the main queues counted in the step S2, wherein the calculation method comprises the following steps:
the maximum number of expected stored main queues is (total number of main queues/total number of nodes) + (total number of main queues% total number of nodes);
s4, selecting the node with the maximum number of the stored main queues and the node with the minimum number of the stored main queues from the step S2, and comparing the number of the main queues in the node with the maximum number of the stored main queues with the maximum number of the main queues which are calculated and expected to be stored in the step S3; when the number of the main queues in the nodes is larger than the maximum number of the main queues expected to be stored, randomly selecting one main queue from the nodes with the maximum number of the stored main queues to migrate to the node with the minimum number of the stored main queues; otherwise, ending the load balance of the main queue;
the migration method of the main queue among the nodes comprises the following steps:
s401, selecting one message queue from the nodes with the maximum number of the stored main queues, and setting a temporary strategy of the message queue: setting the message queue as a copy 1, and executing the message queue to enable the temporary policy to take effect;
s402, migrating the main queue in the message queue selected in the step S401 from the node with the maximum number of the stored main queues, and setting a temporary strategy of the message queue: setting the node of the message queue as a migration target node; executing the message queue to enable the temporary policy of the message queue to take effect;
s403, the main queue of the selected message queue in the step S401 is migrated to the node with the minimum number of the stored main queues, and the temporary strategy in the step S401 and the step S402 is cleared, so that the strategy of the message queue cluster is restored to a normal state;
s404, the slave queue in the message queue selected in the step S401 is migrated to the nodes except the node with the minimum number of the stored main queues, and the migration of one main queue is finished;
and S5, after the message queue cluster is migrated through the main queue in the step S4, repeating the loop from the step S2 to the step S4 until the comparison result in the step S4 shows that the number of the main queues in the nodes storing the maximum number of the main queues is less than or equal to the maximum number expected to be stored, and ending the load balance of the message queue cluster.
According to the method for balancing the load of the main queues based on the mirror image queues, only one main queue can be migrated between the nodes every time, and a plurality of main queues can be migrated simultaneously; if the corresponding message queue is accessed by the message consumer in the migration process of the main queue, the access cannot be achieved, but only needs to be accessed again due to the short migration time.
Examples
In this embodiment, the above-mentioned method for load balancing of master queues is applied to a RabbitMQ cluster, and messages are stored by using message queues for dequeuing and enqueuing messages, where the message queues include a master queue and several slave queues, where the master queue is used for providing message services and the slave queues are used for backing up messages; queuing messages in nodes, wherein the master queue and the slave queue are in different nodes, deploying services of the message queues by using the nodes, and controlling dequeue and enqueue of the messages; all nodes of the message queue form a complete message queue cluster. The distribution of the applied message queue cluster is shown in fig. 1, and the application of the main queue load balancing method to the message queue cluster shown in fig. 1 includes the following steps:
s1, setting the default strategy of all messages in the message queue cluster as follows: all, namely all message queues have a master queue node, and the rest nodes are slave queue nodes;
s2, acquiring the node composition in the message queue cluster, and counting the number of the main queues stored in each node; the message queue cluster is composed of three nodes, namely node1, node2 and node3, the total number of main queues of node1 is 5, the total number of main queues of node2 is 3, and the total number of main queues of node3 is 1;
s3, calculating the maximum number of the main queues expected to be stored in the nodes according to the total number of the nodes and the total number of the main queues counted in the step S2, wherein the calculation method comprises the following steps:
the maximum number of expected stored main queues is (total number of main queues/total number of nodes) + (total number of main queues% total number of nodes); the maximum number of expected deposit master queues in the message queue cluster shown in fig. 1 is 9/3-9% 3-3;
s4, selecting the node with the maximum number of the stored main queues and the node with the minimum number of the stored main queues from the step S2; node1 in FIG. 1 is the node storing the largest number of primary queues; node3 is the node with the minimum number of the main queues; the number of the main queues stored in the node1 is 5, which is greater than the maximum number 3 of the expected main queues, and a main queue of message1 in a node1 node is selected to be migrated to a node2 node, wherein the migration steps of the main queue are as follows:
s401, modifying the copy number of the message queue message1 to be 1, and setting the temporary strategy of the message queue message1 to be { "ha-mode": exact "," ha-params ":1 }; executing the sync _ queue message1 to validate the temporary policy; making the message queue cluster as shown in fig. 2, only the main queue of the message queue message1 is stored in node 1;
s402, migrating the message1 to a node3, and setting the temporary strategy of the message1 as { "ha-mode": nodes, ha-params [ "node3" ] }; executing the sync _ queue message1 to validate the temporary policy; making the message queue cluster as shown in fig. 3, only the main queue of the message queue message1 is stored in node 3;
s403, and clearing the temporary policies in step S401 and step S402, and at this time, the policies for all message queues are restored to the default policies in step S1: ha-mode and all;
s404, migrating the message queue message1 from the queue to node1 and node2, wherein the message queue cluster is shown in FIG. 4, and a slave queue of message queue message1 appears in node1 and node2 to complete the migration of a master queue;
s5, after the message queue cluster is migrated through the main queue in step S4, the loop from step S2 to step S4 is performed again, at this time, the number of main queues stored in the node1 is 4, which is still greater than 3, so that the process of looping step S4 again completes the migration of one message queue, at this time, the number of main queues in the message queue cluster is 3, which satisfies the condition of load balancing of the main queues, and completes the load balancing of the main queues of the whole message queue cluster.
The invention selects one message queue to be migrated at one time, reduces the probability that the message queue is accessed by the message consumers in the migration process, and avoids the increase of node pressure caused by the retry of a large number of message consumers.
By adopting the technical scheme disclosed by the invention, the following beneficial effects are obtained:
the invention discloses a load balancing method of a main queue based on a mirror image queue, which is characterized in that a mechanism for dynamically balancing the main queue is arranged, the number of the main queues in a node is automatically acquired, and the main queues are actively migrated according to a set rule; the invention can autonomously find the distribution quantity of the main queues in the nodes, smoothly transfer the main queues in the nodes with more distribution quantity of the main queues to the nodes with less distribution quantity of the main queues, and completely cover the scene of unbalanced load of all the main queues; the main queue balancing method is suitable for the pressure imbalance phenomenon caused by the downtime recovery of the nodes, the addition of new nodes and the release of partial message queues after use, and can automatically balance the main queue distribution in the nodes of the message queue cluster.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements should also be considered within the scope of the present invention.
Claims (4)
1. A load balancing method for a main queue based on a mirror image queue is characterized by comprising the following steps:
s1, a main queue and a slave queue are arranged in the message queue, the message queue is stored in a node, and the node in which the message queue is stored forms a message queue cluster;
s2, acquiring node information in the message queue cluster, acquiring node composition in the message queue through a command, and counting the number of the main queues stored in each node;
s3, calculating the maximum number of the primary queues expected to be stored in the nodes according to the total number of the nodes and the total number of the primary queues counted in the step S2;
s4, selecting the node with the maximum number of the stored main queues and the node with the minimum number of the stored main queues from the step S2, and comparing the number of the main queues in the node with the maximum number of the stored main queues with the maximum number of the main queues which are calculated and expected to be stored in the step S3; when the number of the main queues in the nodes is larger than the maximum number of the main queues expected to be stored, randomly selecting one main queue from the nodes with the maximum number of the stored main queues to migrate to the node with the minimum number of the stored main queues; otherwise, ending the balance;
s5, after the message queue cluster is migrated through the main queue in the step S4, the loop from the step S2 to the step S4 is repeated until the comparison result in the step S4 shows that the number of the main queues in the nodes storing the maximum number of the main queues is less than or equal to the maximum number expected to be stored, and the load balance of the message queue cluster is finished.
2. The method of load balancing primary queues based on mirror image queues according to claim 1, wherein the maximum number of primary queues expected to be stored in the node of step S3 is calculated by:
the maximum number of expected stored primary queues is (total number of primary queues/total number of nodes) + (total number of primary queues% total number of nodes).
3. The method for load balancing a primary queue based on a mirror queue according to claim 1, wherein in step S4, the migration method of the primary queue between the nodes is as follows:
s401, selecting one message queue from the nodes with the maximum number of the stored main queues, and setting a temporary strategy of the message queue: setting the message queue as a copy 1, and executing the message queue to enable the temporary policy to take effect;
s402, migrating the main queue in the message queue selected in the step S401 from the node with the maximum number of the stored main queues, and setting a temporary strategy of the message queue: setting the node of the message queue as a migration target node; executing the message queue to enable the temporary policy of the message queue to take effect;
s403, the main queue of the selected message queue in the step S401 is migrated to the node with the minimum number of the stored main queues, and the temporary strategy in the step S401 and the step S402 is cleared, so that the strategy of the message queue cluster is restored to a normal state;
s404, the slave queue in the message queue selected in the step S401 is migrated to the nodes except the node with the minimum number of the stored main queues, and the migration of one main queue is finished;
s405, the message queue cluster migrated in steps S401 to S404 returns to step S2 again, and the process loops.
4. The method of claim 1, wherein there is one and only one master queue in the same message queue, and the master queue and the slave queue of the same message queue are stored in different nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010784404.XA CN111897656B (en) | 2020-08-06 | 2020-08-06 | Main queue load balancing method based on mirror image queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010784404.XA CN111897656B (en) | 2020-08-06 | 2020-08-06 | Main queue load balancing method based on mirror image queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111897656A CN111897656A (en) | 2020-11-06 |
| CN111897656B true CN111897656B (en) | 2022-08-16 |
Family
ID=73246029
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010784404.XA Active CN111897656B (en) | 2020-08-06 | 2020-08-06 | Main queue load balancing method based on mirror image queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111897656B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112379768B (en) * | 2020-11-12 | 2021-05-18 | 北京中电兴发科技有限公司 | Power consumption reduction method applied to embedded cloud storage system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103227747A (en) * | 2012-03-14 | 2013-07-31 | 微软公司 | High density hosting for messaging service |
| CN104753817A (en) * | 2013-12-25 | 2015-07-01 | 中国移动通信集团公司 | Local analogy method and local analogy system for cloud computing message queue service |
| US10025628B1 (en) * | 2015-06-26 | 2018-07-17 | Amazon Technologies, Inc. | Highly available distributed queue using replicated messages |
| CN110989933A (en) * | 2019-12-05 | 2020-04-10 | 北京首汽智行科技有限公司 | Message queue RockMq smooth migration method |
| CN111343251A (en) * | 2020-02-16 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Message queue service deployment method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9853933B2 (en) * | 2015-08-13 | 2017-12-26 | 60East Technologies, Inc. | Message queue replication with message ownership migration |
-
2020
- 2020-08-06 CN CN202010784404.XA patent/CN111897656B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103227747A (en) * | 2012-03-14 | 2013-07-31 | 微软公司 | High density hosting for messaging service |
| CN104753817A (en) * | 2013-12-25 | 2015-07-01 | 中国移动通信集团公司 | Local analogy method and local analogy system for cloud computing message queue service |
| US10025628B1 (en) * | 2015-06-26 | 2018-07-17 | Amazon Technologies, Inc. | Highly available distributed queue using replicated messages |
| CN110989933A (en) * | 2019-12-05 | 2020-04-10 | 北京首汽智行科技有限公司 | Message queue RockMq smooth migration method |
| CN111343251A (en) * | 2020-02-16 | 2020-06-26 | 苏州浪潮智能科技有限公司 | Message queue service deployment method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111897656A (en) | 2020-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3433759B1 (en) | Method and apparatus for expanding high-availability server cluster | |
| EP2710470B1 (en) | Extensible centralized dynamic resource distribution in a clustered data grid | |
| CN112867050B (en) | UPF network element management method and system | |
| CN108881512B (en) | CTDB virtual IP balance distribution method, device, equipment and medium | |
| EP0822494B1 (en) | Load balancing method and apparatus | |
| EP2493118B1 (en) | Information processing system | |
| CN109194584A (en) | A kind of flux monitoring method, device, computer equipment and storage medium | |
| US20030236887A1 (en) | Cluster bandwidth management algorithms | |
| CN108897606B (en) | Self-adaptive scheduling method and system for virtual network resources of multi-tenant container cloud platform | |
| CN110781006A (en) | Load balancing method, device, node and computer readable storage medium | |
| CN106484528A (en) | It is used in Distributed Architecture realizing the method and device of cluster dynamic retractility | |
| JP5599943B2 (en) | Server cluster | |
| CN108228393A (en) | A kind of implementation method of expansible big data High Availabitity | |
| CN111897656B (en) | Main queue load balancing method based on mirror image queue | |
| CN113553179A (en) | Distributed key value storage load balancing method and system | |
| CN114844912B (en) | Data link distribution method and device and distributed block storage system | |
| CN110515728B (en) | Server scheduling method and device, electronic equipment and machine-readable storage medium | |
| CN109542627A (en) | Node switching method, device, supervisor, node device and distributed system | |
| CN114338670B (en) | Edge cloud platform and network-connected traffic three-level cloud control platform with same | |
| CN114327911A (en) | Remote multi-activity implementation method of distributed service cluster and distributed service system | |
| CN109298949B (en) | Resource scheduling system of distributed file system | |
| JP4309321B2 (en) | Network system operation management method and storage apparatus | |
| CN114064226A (en) | Resource coordination method and device for container cluster and storage medium | |
| CN116643858A (en) | Service priority pod-based rescheduling method, device, equipment and medium | |
| CN111930563B (en) | Fault tolerance method in cloud simulation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |