CN111427711A - Message pushing method based on RabbitMQ - Google Patents
Message pushing method based on RabbitMQ Download PDFInfo
- Publication number
- CN111427711A CN111427711A CN202010247669.6A CN202010247669A CN111427711A CN 111427711 A CN111427711 A CN 111427711A CN 202010247669 A CN202010247669 A CN 202010247669A CN 111427711 A CN111427711 A CN 111427711A
- Authority
- CN
- China
- Prior art keywords
- message
- rabbitmq
- consumption
- pushing method
- consumer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012790 confirmation Methods 0.000 claims abstract description 22
- 238000005457 optimization Methods 0.000 claims abstract description 5
- 230000002159 abnormal effect Effects 0.000 claims description 18
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000010200 validation analysis Methods 0.000 claims description 3
- 238000007689 inspection Methods 0.000 claims 1
- 230000002085 persistent effect Effects 0.000 abstract description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention particularly relates to a message pushing method based on a RabbitMQ. The message pushing method based on the RabbitMQ optimizes a message confirmation mechanism, a timing task mechanism and a producer consumption confirmation mechanism. The message pushing method based on the RabbitMQ combines the confirmation process of the producer and the confirmation process of the consumer in the RabbitMQ for optimization, so that the producer can more stably know that the consumer successfully receives the message, and the sending rate of the persistent message is improved; in the consumer confirmation process, the consumer confirmation process is optimized, the condition that the consumer rejects the message is processed, the consumer can be prevented from receiving repeated messages, the overall system is greatly improved in performance, and the stability is better.
Description
Technical Field
The invention relates to the technical field of web message pushing, in particular to a message pushing method based on a RabbitMQ.
Background
The AMQP, Advanced Message Queuing Protocol, is an open standard of an application layer Protocol, and is designed for Message-oriented middleware.
The message queue is a common technology applied to a plurality of systems, and an asynchronization and loose coupling mechanism of the message queue can improve the utilization rate of system resources and the expandability of the system, and meanwhile, the message queue has a distributed and disaster-tolerant characteristic, thereby improving the performance and the fault-tolerant capability of the system.
The RabbitMQ is an open-source AMQP implementation, and the server side is written in Erlang language and supports various clients. The method is used for storing and forwarding messages in a distributed system and is not popular in the aspects of usability, expansibility, high availability and the like. ConnectionFactory, Connection and Channel are all the most basic objects in the API externally provided by the RabbitMQ. Connection is a socket link of a RabbitMQ, which encapsulates socket protocol related part logic. Connection factory is a factory for Connection.
Queue is an internal object of the RabbitMQ for storing messages.
The producer P sends the message to Exchange, which routes the message to one or more queues (or drops).
Exchange is associated with Queue by Binding in the RabbitMQ so that the RabbitMQ knows how to properly route messages to the specified Queue.
Messages in the RabbitMQ can only be stored in Queue, the producer P produces the messages and finally delivers the messages to the Queue, and the consumer C can obtain the messages from the Queue and consume the messages. In practical applications, it may happen that a consumer receives a message in Queue, but terminates the message unexpectedly without completing the processing, which may result in a message loss.
In order to ensure the reliability of the message, namely the message is really sent, is really consumed and cannot be repeatedly consumed, the invention provides a message pushing method based on a RabbitMQ.
Disclosure of Invention
In order to make up for the defects of the prior art, the invention provides a simple and efficient message pushing method based on the RabbitMQ.
The invention is realized by the following technical scheme:
a message pushing method based on RabbitMQ is characterized in that: optimizing a message sending confirmation mechanism, wherein the optimization is as follows:
first, a message acknowledgement mechanism
Setting a trigger setReturn Callback callback to prevent a switch (Exchange) from discarding the message when not finding a Queue (Queue), and informing a user of a problem to be checked through the callback;
second, the timed task mechanism
Pulling the message of delivery failure by the timing task, and delivering again;
third, producer consumption validation mechanism
And (3) ensuring the idempotency of consumption of multiple times of consumption at a consumption end through AOP (Aspect Oriented Programming) interception and manual ACK (Acknowledgement character).
In the timing task mechanism, the message which fails in delivery is pulled regularly and the message is delivered again by setting a timing task and the number of times of delivery again.
In the timing task mechanism, each message is bound with the Router RouteingKey, and all messages are re-launched to share one timing task.
In the producer consumption confirmation mechanism, when a consumer consumes a message, if a consumer service logic is abnormal, a retry mechanism is started, and AOP is used for intercepting at a bottom layer; if the consumer does not throw the exception, the transaction is automatically committed.
If the AOP obtains program exception by using exception notification interception, automatically realizing a compensation mechanism and caching the message to a RabbitMQ server; and if the consumption message is abnormal, selecting a reasonable retry mechanism according to the thrown abnormal.
If the access cannot be realized by calling a third-party interface, automatically retrying the RabbitMQ; if the RabbitMQ retry fails, compensation is made through logging and health checks and manual work.
If the consumers are abnormal or delay consumption of the consumers occurs in network delay transmission, MQ retry compensation is performed, in order to avoid repeated consumption in the retry process, whether the consumers consume the solution idempotency problem is judged by using the global message ID, and the ID and the state are stored in a log table after the consumption is successful; when MQ retry compensation occurs, the status of the message ID is obtained from the log table and determined.
In the producer consumption confirmation mechanism, after the confirmation message is consumed by the consumer, the server is informed to clear the message in the queue; the ACK is set to be manual, the messages (unacknowledged) which are not ACK can be enqueued again and consumed, the business logic codes can be processed more flexibly or in a humanized mode, the abnormal problems and the data return processing are more convenient to process, and the messages are guaranteed not to be lost.
The invention has the beneficial effects that: the message pushing method based on the RabbitMQ combines the confirmation process of the producer and the confirmation process of the consumer in the RabbitMQ for optimization, so that the producer can more stably know that the consumer successfully receives the message, and the sending rate of the persistent message is improved; in the consumer confirmation process, the consumer confirmation process is optimized, the condition that the consumer rejects the message is processed, the consumer can be prevented from receiving repeated messages, the overall system is greatly improved in performance, and the stability is better.
Drawings
Fig. 1 is a schematic diagram of a RabbitMQ-based message pushing method according to the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention more apparent, the present invention is described in detail below with reference to the embodiments. It should be noted that the specific embodiments described herein are only for explaining the present invention and are not used to limit the present invention.
The message pushing method based on the RabbitMQ optimizes a message sending confirmation mechanism, and the specific optimization is as follows:
first, a message acknowledgement mechanism
Setting a trigger setReturn Callback callback to prevent a switch (Exchange) from discarding the message when not finding a Queue (Queue), and informing a user of a problem to be checked through the callback;
second, the timed task mechanism
Pulling the message of delivery failure by the timing task, and delivering again;
third, producer consumption validation mechanism
And (3) ensuring the idempotency of consumption of multiple times of consumption at a consumption end through AOP (Aspect Oriented Programming) interception and manual ACK (Acknowledgement character).
The message pushing method based on the RabbitMQ comprises three steps of sending a message by a producer, consuming the message by the producer and manually ACK.
The producer sends the message step:
in a RabbitMQ, a producer sends a message to an Exchange switch, which routes the message to one or more queues, or discards the message, where the message may fail to be sent to Exchange. In the message acknowledgement mechanism, setretrencallback callback is set to acknowledge that the message can be successfully routed from Exchange to Queue, preventing Exchange from discarding the message without finding Queue.
In the timing task mechanism, the message which fails in delivery is pulled regularly and the message is delivered again by setting a timing task and the number of times of delivery again.
In the timing task mechanism, each message is bound with the Router RouteingKey, and all messages are re-launched to share one timing task.
The producer consumes the message step:
in the producer consumption confirmation mechanism, when a consumer consumes a message, if a consumer service logic is abnormal, a retry mechanism is started, and AOP is used for intercepting at a bottom layer; if the consumer does not throw the exception, the transaction is automatically committed.
If the AOP obtains program exception by using exception notification interception, automatically realizing a compensation mechanism and caching the message to a RabbitMQ server; and if the consumption message is abnormal, selecting a reasonable retry mechanism according to the thrown abnormal.
If the access cannot be realized by calling a third-party interface, automatically retrying the RabbitMQ; if the RabbitMQ retry fails, compensation is made through logging and health checks and manual work.
If the consumers are abnormal or delay consumption of the consumers occurs in network delay transmission, MQ retry compensation is performed, in order to avoid repeated consumption in the retry process, whether the consumers consume the solution idempotency problem is judged by using the global message ID, and the ID and the state are stored in a log table after the consumption is successful; when MQ retry compensation occurs, the status of the message ID is obtained from the log table and determined. Therefore, the idempotency of the consumption end is ensured, and even if the callback is not triggered due to successful delivery of the network and other reasons, repeated consumption and further abnormal service can not be generated after delivery for many times.
Manual ACK:
in the producer consumption confirmation mechanism, the server is informed to clear the message in the queue after the confirmation message is completely consumed by the consumer. The consumer code may be abnormal, if the consumer code is not processed, the service is executed wrongly, the message is not found, and the message is lost. However, as the consumption end code is captured abnormally, when the service is abnormal, the RabbitMQ is informed that the message consumption fails. The ACK is set to be manual, and messages (unacknowledged) which are not ACK can be re-enqueued and consumed, so that the messages are guaranteed not to be lost and need to be re-enqueued, the messages can be re-delivered to other normal consumption terminals for consumption, and the messages are guaranteed not to be lost. The manual ACK can more flexibly or humanizedly process the service logic code, and more conveniently process the abnormal problem and return processing of data.
The above-described embodiment is only one specific embodiment of the present invention, and general changes and substitutions by those skilled in the art within the technical scope of the present invention are included in the protection scope of the present invention.
Claims (8)
1. A message pushing method based on RabbitMQ is characterized in that: optimizing a message sending confirmation mechanism, wherein the optimization is as follows:
first, a message acknowledgement mechanism
The method comprises the steps of setting a trigger setReturn Callback callback to prevent a switch from discarding messages when a queue is not found, and informing a user of a problem of inspection through the callback;
second, the timed task mechanism
Pulling the message of delivery failure by the timing task, and delivering again;
third, producer consumption validation mechanism
And the consumption idempotency of multiple times of consumption is ensured at the consumption end through AOP interception and manual ACK.
2. The RabbitMQ-based message pushing method according to claim 1, wherein: in the timing task mechanism, the message which fails in delivery is pulled regularly and the message is delivered again by setting a timing task and the number of times of delivery again.
3. The RabbitMQ-based message pushing method according to claim 2, wherein: in the timing task mechanism, each message is bound with the Router RouteingKey, and all messages are re-launched to share one timing task.
4. The RabbitMQ-based message pushing method according to claim 1, wherein: in the producer consumption confirmation mechanism, when a consumer consumes a message, if a consumer service logic is abnormal, a retry mechanism is started, and AOP is used for intercepting at a bottom layer; if the consumer does not throw the exception, the transaction is automatically committed.
5. The RabbitMQ-based message pushing method according to claim 4, wherein: if the AOP obtains program exception by using exception notification interception, automatically realizing a compensation mechanism and caching the message to a RabbitMQ server; and if the consumption message is abnormal, selecting a reasonable retry mechanism according to the thrown abnormal.
6. The RabbitMQ-based message pushing method according to claim 5, wherein: if the access cannot be realized by calling a third-party interface, automatically retrying the RabbitMQ; if the RabbitMQ retry fails, compensation is made through logging and health checks and manual work.
7. The RabbitMQ-based message pushing method according to claim 5, wherein: if the consumers are abnormal or delay consumption of the consumers occurs in network delay transmission, MQ retry compensation is performed, in order to avoid repeated consumption in the retry process, whether the consumers consume the solution idempotency problem is judged by using the global message ID, and the ID and the state are stored in a log table after the consumption is successful; when MQ retry compensation occurs, the status of the message ID is obtained from the log table and determined.
8. The RabbitMQ-based message pushing method according to claim 1, wherein: in the producer consumption confirmation mechanism, after the confirmation message is consumed by the consumer, the server is informed to clear the message in the queue; the ACK is set to be manual, the messages which are not subjected to the ACK can be queued again and consumed, the service logic codes can be processed more flexibly or in a humanized mode, the abnormal problems and the data return processing are more convenient to process, and the messages are prevented from being lost.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010247669.6A CN111427711A (en) | 2020-04-01 | 2020-04-01 | Message pushing method based on RabbitMQ |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010247669.6A CN111427711A (en) | 2020-04-01 | 2020-04-01 | Message pushing method based on RabbitMQ |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111427711A true CN111427711A (en) | 2020-07-17 |
Family
ID=71550369
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010247669.6A Pending CN111427711A (en) | 2020-04-01 | 2020-04-01 | Message pushing method based on RabbitMQ |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111427711A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112131308A (en) * | 2020-08-03 | 2020-12-25 | 深圳市麦谷科技有限公司 | Distributed non-blocking WeChat template method and device based on RabbitMq |
| CN112256461A (en) * | 2020-12-08 | 2021-01-22 | 万邑通商(北京)信息科技有限公司 | Distributed message system based on multi-cloud environment and management method |
| CN112988428A (en) * | 2021-04-26 | 2021-06-18 | 南京蜂泰互联网科技有限公司 | Distributed message asynchronous notification middleware implementation method and system |
| CN113656200A (en) * | 2021-08-24 | 2021-11-16 | 福建天晴数码有限公司 | Method and system for realizing delivery notification by using delay queue |
| CN113849325A (en) * | 2021-09-29 | 2021-12-28 | 四川新网银行股份有限公司 | Distributed consumption credit RabbitMQ fusing method and system |
| CN113938846A (en) * | 2021-10-13 | 2022-01-14 | 福建天泉教育科技有限公司 | Message delay sending method and terminal |
| CN114024901A (en) * | 2022-01-05 | 2022-02-08 | 中邮消费金融有限公司 | Message isolation forwarding method and system |
| CN114979249A (en) * | 2022-03-30 | 2022-08-30 | 阿里巴巴（中国）有限公司 | Message handle creating method, message pushing method, related device and system |
| CN115333978A (en) * | 2022-10-11 | 2022-11-11 | 中化现代农业有限公司 | Data monitoring method and device |
| CN115576714A (en) * | 2022-10-19 | 2023-01-06 | 深圳市中兴新云服务有限公司 | Method for ensuring accuracy of message queue consumption sequence based on MQ framework |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109542639A (en) * | 2018-11-06 | 2019-03-29 | 用友网络科技股份有限公司 | A kind of processing method, processing unit for ensureing micro services and calling data consistency |
| CN110532066A (en) * | 2018-05-23 | 2019-12-03 | 中国移动通信集团浙江有限公司 | A kind of realization method and system of transaction consistency |
-
2020
- 2020-04-01 CN CN202010247669.6A patent/CN111427711A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110532066A (en) * | 2018-05-23 | 2019-12-03 | 中国移动通信集团浙江有限公司 | A kind of realization method and system of transaction consistency |
| CN109542639A (en) * | 2018-11-06 | 2019-03-29 | 用友网络科技股份有限公司 | A kind of processing method, processing unit for ensureing micro services and calling data consistency |
Non-Patent Citations (1)
| Title |
|---|
| APPLE_WEB: ""RabbitMQ解决消费者补偿幂等问题(重复消费问题)"", 《CSDN》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112131308A (en) * | 2020-08-03 | 2020-12-25 | 深圳市麦谷科技有限公司 | Distributed non-blocking WeChat template method and device based on RabbitMq |
| CN112256461A (en) * | 2020-12-08 | 2021-01-22 | 万邑通商(北京)信息科技有限公司 | Distributed message system based on multi-cloud environment and management method |
| CN112988428A (en) * | 2021-04-26 | 2021-06-18 | 南京蜂泰互联网科技有限公司 | Distributed message asynchronous notification middleware implementation method and system |
| CN113656200A (en) * | 2021-08-24 | 2021-11-16 | 福建天晴数码有限公司 | Method and system for realizing delivery notification by using delay queue |
| CN113849325A (en) * | 2021-09-29 | 2021-12-28 | 四川新网银行股份有限公司 | Distributed consumption credit RabbitMQ fusing method and system |
| CN113938846A (en) * | 2021-10-13 | 2022-01-14 | 福建天泉教育科技有限公司 | Message delay sending method and terminal |
| CN114024901A (en) * | 2022-01-05 | 2022-02-08 | 中邮消费金融有限公司 | Message isolation forwarding method and system |
| CN114024901B (en) * | 2022-01-05 | 2022-04-19 | 中邮消费金融有限公司 | Message isolation forwarding method and system |
| CN114979249A (en) * | 2022-03-30 | 2022-08-30 | 阿里巴巴（中国）有限公司 | Message handle creating method, message pushing method, related device and system |
| CN115333978A (en) * | 2022-10-11 | 2022-11-11 | 中化现代农业有限公司 | Data monitoring method and device |
| CN115576714A (en) * | 2022-10-19 | 2023-01-06 | 深圳市中兴新云服务有限公司 | Method for ensuring accuracy of message queue consumption sequence based on MQ framework |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111427711A (en) | Message pushing method based on RabbitMQ | |
| CN110297801B (en) | System and method for just-in-one transaction semantics of transaction system based on fault-tolerant FPGA | |
| JP4515800B2 (en) | How to transparently improve availability and scalability in message exchange systems to applications | |
| US7676580B2 (en) | Message delivery with configurable assurances and features between two endpoints | |
| US8516054B2 (en) | Message handling | |
| US8386586B2 (en) | Distributed kernel operating system | |
| CN109714409B (en) | Message management method and system | |
| US8667184B2 (en) | Distributed kernel operating system | |
| US20110173495A1 (en) | Method and System for Reliable Intersystem Message Notification | |
| JP2009531879A (en) | Method and system for message retransmission and method and system for intersystem message delivery | |
| US8275905B2 (en) | System and method for store-and-forward for highly available message production | |
| US10362131B1 (en) | Fault tolerant message delivery | |
| CN103532822A (en) | Management method for message delivery, server and system thereof | |
| JP2011171867A (en) | Data storage method and mail relay method of data store server in mail system | |
| CN114338651A (en) | File transmission method and device, electronic equipment and readable storage medium | |
| CN101207617A (en) | Method for data transmission of network storing system | |
| WO1999009490A1 (en) | Certified message delivery and queuing in multipoint publish/subscribe communications | |
| US11582320B2 (en) | Message transmitting and receiving method, communication apparatus, and program | |
| US20120096179A1 (en) | Method For Processing Initial SIP Requests By Backends Of A SIP Cluster In The Presence Of A Fault, And Associated Processing Device | |
| CN107623645B (en) | Real-time data exchange system of power system based on data stream forwarding | |
| CN103647622A (en) | Method, apparatus and system for realizing computer room-spanning data transmission | |
| CN110572474B (en) | Method for embedded terminal long-connection communication | |
| WO2024052981A1 (en) | Processing device, processing method, and program | |
| JP5865424B2 (en) | Message system and data store server | |
| US20060288094A1 (en) | Methods for configuring cache memory size |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200717 |
|
| RJ01 | Rejection of invention patent application after publication |