CN105068864B - Method and system for processing asynchronous message queue - Google Patents
Method and system for processing asynchronous message queue Download PDFInfo
- Publication number
- CN105068864B CN105068864B CN201510442427.1A CN201510442427A CN105068864B CN 105068864 B CN105068864 B CN 105068864B CN 201510442427 A CN201510442427 A CN 201510442427A CN 105068864 B CN105068864 B CN 105068864B
- Authority
- CN
- China
- Prior art keywords
- message
- queue
- messages
- consumer
- priority
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a method and a system for processing an asynchronous message queue, which can correctly process according to priority and is simple and easy to implement. Wherein, the method comprises the following steps: the producer sets a priority parameter for each message, and sends a plurality of messages to a message queue in batch, wherein the messages comprise message bodies, message keywords and the priority parameter; the message queue stores messages; the consumer consumes the message from the message queue; and the service processing unit acquires the consumption result of the consumer and processes the consumption result according to the priority parameter.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a system for processing an asynchronous message queue.
Background
The message queue processing permeates various fields of the internet at present due to the characteristics of asynchronous processing, safety, stability and the like. The producer places the generated message in a message queue and the consumer listens to the message queue and consumes the data. Sometimes, a plurality of message producers send messages to the message queue, but due to network delay and other reasons, the message queue cannot be guaranteed to receive messages with a certain priority, and therefore, the consumers cannot be guaranteed to consume data with a certain priority.
Generally, in the prior art, service logic control is used to guarantee the priority of data. Specifically, the method comprises the following steps: the sequence of message processing is judged by comparing the specific service attributes (such as the version numbers corresponding to the messages) of a plurality of messages. The scheme occupies more memory space and consumes more CPU resources, thereby reducing the system performance.
In addition, a database retrieval mode is adopted in the prior art, the scheme can occupy more database connections, and the processing performance of the database can be reduced when the database is greatly concurrent, so that the overall processing speed of the system is reduced.
Disclosure of Invention
In view of the above, the present invention provides a simple and easy method and system for processing an asynchronous message queue, which can correctly process the asynchronous message queue according to priority.
To achieve the above object, according to a first aspect of the present invention, there is provided a method of processing an asynchronous message queue, comprising: the producer sets a priority parameter for each message, and sends a plurality of messages to a message queue in batch, wherein the messages comprise message bodies, message keywords and the priority parameter; the message queue stores the message; the consumer consumes the message from the message queue; and the service processing unit acquires the consumption result of the consumer and refers to the priority parameter for processing.
Optionally, the step of the consumer consuming the message from the message queue comprises: the consumer consuming read messages one by one from the message queue; the consumer stores the message in a first cache queue in an ordered set type, and simultaneously stores the message keyword of the message in a second cache queue in a linked list type, and the step of the service processing unit acquiring the consumption result of the consumer and processing the consumption result by referring to the priority parameter comprises the following steps: the service processing unit monitors the second cache queue in a blocking mode, and when the message keywords are monitored to be stored in the second cache queue, the current local cache data are emptied; and the service processing unit acquires all the message main bodies in the messages which are corresponding to the message keywords and are automatically sequenced according to the priority parameters from the first cache queue, caches the message main bodies in the messages to the local, and performs service processing in sequence.
Optionally, the service processing unit starts a plurality of threads to monitor the second cache queue in a blocking manner.
Optionally, the priority parameter is expressed by a positive integer, and the smaller the number, the higher the priority.
To achieve the above object, according to a second aspect of the present invention, there is provided a system for processing an asynchronous message queue, comprising: the producer is used for setting a priority parameter for each message and sending a plurality of messages to the message queue in batches, wherein the messages comprise message bodies, message keywords and the priority parameter; the message queue is used for storing the messages; a consumer for consuming the message from the message queue; and the service processing unit is used for acquiring the consumption result of the consumer and processing the consumption result by referring to the priority parameter.
Optionally, the method further comprises: the first buffer queue is used for storing the messages in a priority ordered set type; the second buffer queue is used for storing the message keywords of the message in a linked list type; the consumer is also used for reading messages one by one from the message queue, then storing the messages in a first cache queue in a priority ordered set type, and storing the message keywords of the messages in a second cache queue in a linked list type; the service processing unit is further configured to monitor the second cache queue in a blocking manner, when it is monitored that the message keyword is stored in the second cache queue, clear current local cache data, then take out all message bodies in the message, which are automatically sorted according to the priority parameter and correspond to the message keyword, from the first cache queue to cache to the local, and perform service processing in sequence.
Optionally, the service processing unit starts a plurality of threads to monitor the second cache queue in a blocking manner.
Optionally, the priority parameter is expressed by a positive integer, and the smaller the number, the higher the priority.
According to the technical scheme of the invention, the consumer stores the message into the sortedset integrally and stores the message key words into the list at the same time, and then the service processing unit reads the message main body from the sortedset according to the monitored message key words, so that a plurality of messages with the same key words can be processed sequentially according to a certain priority rule, and the invention has the advantages of simplicity, easy implementation and high processing efficiency.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic main flow diagram of a method of processing an asynchronous message queue according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the major components of a system for processing asynchronous message queues, according to an embodiment of the present invention;
fig. 3 is a timing diagram of an aspect of the present invention of handling asynchronous message queues.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
For the skilled person to understand better, some of the terms of art are described below:
the producer: i.e. the provider, sender of the message, note that in the art, the essence here is usually a node and not a natural person.
Message: which may be data or management instructions, etc.
Message queue: message queues are communication means used to asynchronously process a series of incoming messages. The message queue may be implemented by a message queue server.
The consumer: i.e. the user, recipient of the message, it is noted that in the art, the nature here generally refers to a node and not to a natural person.
Key words: the attributes of the message record or set of attributes can be uniquely determined.
Redis: redis is an open-source, network-supported, memory-based key-value pair (key-value) storage database, supports multiple data types and multiple operations, and has a powerful function of managing data. The value types it supports to store include string (string), linked list (list), set (set), and ordered set (zset, e.g., sortedset). These data types all support push/pop, add/remove, and intersect union and difference, and richer operations, and these operations are all atomic.
Priority ordered set type: i.e. sortedset type, is one of zset types supported by redis, and can support data to be automatically sorted according to priority.
Type of linked list: i.e., list type, which is one of the supported types of redis, blocking reads may be supported.
Fig. 1 is a schematic main flow diagram of a method for processing an asynchronous message queue according to an embodiment of the present invention. As shown in fig. 1, the method mainly includes the following steps S101 to S104.
Step S101: the producer sets a priority parameter for each message and sends a plurality of messages to the message queue in batches. Wherein, the message comprises a message body, a message keyword key and a priority parameter sort.
Alternatively, the priority parameter is expressed in positive integers, with smaller numbers giving higher priority. This form of expression has the advantage of being simple and easy to implement.
Step S102: the message queue holds messages.
Through the message queue, transaction messages generated by high concurrency in a short time are stored in the message queue, so that the concurrent transactions in a peak period are flattened, and the system performance is improved.
Step S103: the consumer consumes the message from the message queue. The specific process is as follows:
first, the consumer reads messages from the message queue, one by one. Then, the messages are stored in a first buffer queue in a priority ordered set type, and the message keywords of the messages are stored in a second buffer queue in a linked list type.
It should be noted that, in the process of integrally storing the messages in the first cache queue redis1 in the priority ordered set sortedset type, the messages stored in redis1 may be in the form of: [ message Key priority parameter sort message body content body ]. Every time data is stored in, redis1 will complete the storage of the priority of the message represented by the relevant key using the automatic ordering feature of the sortedset's priority. In other words, assuming that several messages with the same message keyword are input to the redis1 over a period of time, the messages are automatically sorted according to the priority parameter and then stored together collectively and sequentially. Meanwhile, the message keywords of the message are stored in a second cache queue redis2 in a linked list type, and the monitoring queue is realized in a time reminding mode by utilizing the characteristic that the message keywords block a reading list. Note that care is taken to ensure atomicity of both operations stored in redis1 and in redis 2.
Step S104: and the service processing unit acquires the consumption result of the consumer and refers to the priority parameter for processing. The specific process is as follows:
firstly, the service processing unit monitors the second buffer queue in a blocking mode, and when the message key words are monitored to be stored in the second buffer queue, the current local buffer data is emptied. And then, all message bodies in the messages which are corresponding to the message keywords and are automatically sorted according to the priority parameters are obtained from the first cache queue to be cached locally, and service processing is carried out in sequence.
Optionally, the service processing unit starts multiple threads to monitor the second cache queue in a blocking manner, so that multiple keywords can be extracted at the same time, and then multiple message bodies are simultaneously obtained for analysis, thereby finally improving the efficiency of the service processing unit in processing data.
As can be seen from the above, according to the method for processing an asynchronous message queue according to the embodiment of the present invention, a consumer stores a message into a sortedset as a whole and stores a message keyword into a list, and then a service processing unit reads out a message body from the sortedset according to the monitored message keyword, so that a plurality of messages having the same keyword can be sequentially processed according to a certain priority rule, and the method has the advantages of simplicity, easy implementation and high processing efficiency.
FIG. 2 is a schematic diagram of the main components of a system for processing asynchronous message queues, according to an embodiment of the present invention. As shown in FIG. 2, the system 20 for processing asynchronous message queues includes the following components: producer 201, message queue 202, consumer 203, business processing unit 204. The specific introduction of each module is as follows:
the producer 201 is used to set a priority parameter for each message and also to send a plurality of messages in a batch to the message queue 202. Wherein, the message comprises a message body, a message keyword and a priority parameter.
Alternatively, the priority parameter is expressed in positive integers, with smaller numbers giving higher priority. This form of expression has the advantage of being simple and easy to implement.
The message queue 202 is used to hold messages. Through the message queue, transaction messages generated by high concurrency in a short time are stored in the message queue, so that the concurrent transactions in a peak period are flattened, and the system performance is improved.
The consumer 203 is used to consume messages from the message queue 202. Specifically, consumer 203 reads messages from message queue 202 on a per-item basis, then stores the messages in a first cache queue redis1 in a priority ordered set type, and stores the message keys of the messages in a second cache queue redis2 in a linked list type.
The service processing unit 204 is configured to obtain a consumption result of the consumer 203 and perform processing with reference to the priority parameter. Specifically, the service processing unit 204 is configured to monitor the second cache queue redis2 in a blocking manner, when it is monitored that the message keyword is stored in the second cache queue redis2, clear the current local cache data, then take out all message bodies in the message that is automatically sorted according to the priority parameter and corresponds to the message keyword from the first cache queue redis1, cache the message bodies in the message to the local, and perform service processing in sequence.
Optionally, the service processing unit 204 opens multiple threads to snoop the second cache queue redis2 in a blocking manner. Therefore, a plurality of keywords can be extracted at the same time, and then a plurality of message bodies are obtained at the same time for analysis, and finally the data processing efficiency of the service processing unit is improved.
As can be seen from the above, according to the system for processing an asynchronous message queue according to the embodiment of the present invention, a consumer stores a message in a sortedset as a whole and stores a message keyword in a list, and then the service processing unit reads out a message body from the sortedset according to the monitored message keyword, so that a plurality of messages having the same keyword can be sequentially processed according to a certain priority rule, and the system has the advantages of simplicity, easy implementation and high processing efficiency.
To make it better understood by those skilled in the art, a timing diagram providing the technical solution of the present invention for processing an asynchronous message queue is shown in fig. 3.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A method of processing an asynchronous message queue, comprising:
the producer sets a priority parameter for each message, and sends a plurality of messages to a message queue in batch, wherein the messages comprise message bodies, message keywords and the priority parameter;
the message queue stores the message;
the consumer consumes the message from the message queue;
the service processing unit acquires the consumption result of the consumer and refers to the priority parameter for processing;
the step of the consumer consuming the message from the message queue comprises:
the consumer consuming read messages one by one from the message queue;
the consumer storing the message in a first cache queue in a priority ordered set type, while the consumer storing the message key for the message in a second cache queue in a linked list type,
and the number of the first and second electrodes,
the step of acquiring the consumption result of the consumer and processing by the service processing unit by referring to the priority parameter comprises the following steps:
the service processing unit monitors the second cache queue in a blocking mode, and when the message keywords are monitored to be stored in the second cache queue, the current local cache data are emptied;
and the service processing unit acquires all the message main bodies in the messages which are corresponding to the message keywords and are automatically sequenced according to the priority parameters from the first cache queue, caches the message main bodies in the messages to the local, and performs service processing in sequence.
2. The method of processing an asynchronous message queue as recited in claim 1, wherein the service processing unit opens multiple threads to snoop the second cache queue in a blocking manner.
3. A method of handling asynchronous message queues according to claim 1, characterized in that said priority parameter is expressed in positive integers, the lower the number the higher the priority.
4. A system for processing asynchronous message queues, comprising:
the producer is used for setting a priority parameter for each message and sending a plurality of messages to the message queue in batches, wherein the messages comprise message bodies, message keywords and the priority parameter;
the message queue is used for storing the messages;
a consumer for consuming the message from the message queue;
the service processing unit is used for acquiring the consumption result of the consumer and processing the consumption result by referring to the priority parameter;
the consumer is also used for reading messages one by one from the message queue, then storing the messages in a first cache queue in a priority ordered set type, and storing the message keywords of the messages in a second cache queue in a linked list type;
the service processing unit is further configured to monitor the second cache queue in a blocking manner, when it is monitored that the message keyword is stored in the second cache queue, clear current local cache data, then take out all message bodies in the message, which are automatically sorted according to the priority parameter and correspond to the message keyword, from the first cache queue to cache to the local, and perform service processing in sequence.
5. The system for processing the asynchronous message queue of claim 4, wherein the service processing unit opens multiple threads to snoop the second buffer queue in a blocking manner.
6. The system for processing an asynchronous message queue as recited in claim 4, wherein the priority parameter is expressed as a positive integer, with smaller numbers giving higher priority.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510442427.1A CN105068864B (en) | 2015-07-24 | 2015-07-24 | Method and system for processing asynchronous message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510442427.1A CN105068864B (en) | 2015-07-24 | 2015-07-24 | Method and system for processing asynchronous message queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105068864A CN105068864A (en) | 2015-11-18 |
| CN105068864B true CN105068864B (en) | 2020-02-07 |
Family
ID=54498242
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510442427.1A Active CN105068864B (en) | 2015-07-24 | 2015-07-24 | Method and system for processing asynchronous message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105068864B (en) |
Families Citing this family (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105763595A (en) * | 2015-12-23 | 2016-07-13 | 杭州赫智电子科技有限公司 | Method of improving data processing efficiency and server |
| CN105868033A (en) * | 2016-04-06 | 2016-08-17 | 江苏物联网研究发展中心 | Method and system for achieving priority message queues based on Redis |
| CN106095599A (en) * | 2016-06-07 | 2016-11-09 | 深圳证券通信有限公司 | A kind of easy expansion interface method of asynchronous long connection |
| CN106790678B (en) * | 2017-01-23 | 2020-07-17 | 南威软件股份有限公司 | Transmission system and method for ensuring priority transmission consumption of important data |
| CN107171918B (en) * | 2017-04-26 | 2020-06-16 | 成都成电光信科技股份有限公司 | Message transceiving method in GJB289A bus module supporting priority |
| CN108009029B (en) * | 2017-11-30 | 2022-01-04 | 中电福富信息科技有限公司 | Method and system for decoupling and persisting cache data based on Ignite grid |
| CN109344172B (en) * | 2018-08-31 | 2022-05-17 | 深圳市元征科技股份有限公司 | High-concurrency data processing method and device and client server |
| CN110874232B (en) * | 2018-09-04 | 2023-12-29 | 中兴通讯股份有限公司 | Virtual machine component upgrading method, equipment and computer readable storage medium |
| CN109542638A (en) * | 2018-10-26 | 2019-03-29 | 深圳点猫科技有限公司 | A kind of document handling method and device based on educational system |
| CN109451032B (en) * | 2018-11-20 | 2021-11-23 | 上海联寓智能科技有限公司 | Message transmission system |
| CN109688200A (en) * | 2018-11-30 | 2019-04-26 | 北京奇艺世纪科技有限公司 | A kind of message treatment method, device and equipment |
| CN109743137B (en) * | 2019-01-10 | 2022-01-14 | 浙江小泰科技有限公司 | Distributed delay message queue processing system supporting updating |
| CN111488135A (en) * | 2019-01-28 | 2020-08-04 | 珠海格力电器股份有限公司 | Current limiting method and device for high-concurrency system, storage medium and equipment |
| CN110162391A (en) * | 2019-05-27 | 2019-08-23 | 浪潮云信息技术有限公司 | A kind of asynchronous frame and its implementation |
| CN110221927B (en) * | 2019-06-03 | 2021-11-09 | 中国工商银行股份有限公司 | Asynchronous message processing method and device |
| CN110365796B (en) * | 2019-08-01 | 2022-04-29 | 腾讯科技（深圳）有限公司 | Service request processing method and device |
| CN111580939B (en) * | 2020-04-01 | 2023-09-01 | 微梦创科网络科技（中国）有限公司 | Method and device for processing transactions in hierarchical and asynchronous mode |
| CN113535420A (en) * | 2020-04-20 | 2021-10-22 | 北京沃东天骏信息技术有限公司 | Message processing method and device |
| CN112738173A (en) * | 2020-12-23 | 2021-04-30 | 上海创远仪器技术股份有限公司 | Method, system, device and storage medium for realizing extensible asynchronous message processing between radio cross-systems |
| CN112788154A (en) * | 2021-01-29 | 2021-05-11 | 北京奇艺世纪科技有限公司 | Data transmission system and method |
| CN113326150A (en) * | 2021-05-31 | 2021-08-31 | 中国工商银行股份有限公司 | Online small-batch message processing method and device |
| CN113296917B (en) * | 2021-07-26 | 2021-11-02 | 北京元知创智科技有限公司 | Multi-service system service event subscription scheduling method, electronic device and storage medium |
| CN114003469A (en) * | 2021-12-30 | 2022-02-01 | 北京微步在线科技有限公司 | Asset monitoring method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101316145A (en) * | 2008-07-25 | 2008-12-03 | 中兴通讯股份有限公司 | Method and apparatus for implementing multiline message parallel transmission and recovery |
| CN103051521A (en) * | 2013-01-09 | 2013-04-17 | 浪潮电子信息产业股份有限公司 | Retransmitting method of message in cloud environment |
| US8578218B2 (en) * | 2009-04-04 | 2013-11-05 | Oracle International Corporation | Method and system for implementing a scalable, high-performance, fault-tolerant locking mechanism in a multi-process environment |
| CN103390049A (en) * | 2013-07-23 | 2013-11-13 | 南京联创科技集团股份有限公司 | Method for processing high-speed message queue overflow based on memory database cache |
| CN103516585A (en) * | 2012-06-29 | 2014-01-15 | 北京奇虎科技有限公司 | Method and system for distributing messages according to priorities |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103699616B (en) * | 2013-12-17 | 2017-09-29 | 新浪网技术(中国)有限公司 | Data structure creation method, subscription message data sending method and relevant apparatus |
| CN104753769A (en) * | 2015-03-24 | 2015-07-01 | 新余兴邦信息产业有限公司 | Method and device for issuing messages by message queue |
-
2015
- 2015-07-24 CN CN201510442427.1A patent/CN105068864B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101316145A (en) * | 2008-07-25 | 2008-12-03 | 中兴通讯股份有限公司 | Method and apparatus for implementing multiline message parallel transmission and recovery |
| US8578218B2 (en) * | 2009-04-04 | 2013-11-05 | Oracle International Corporation | Method and system for implementing a scalable, high-performance, fault-tolerant locking mechanism in a multi-process environment |
| CN103516585A (en) * | 2012-06-29 | 2014-01-15 | 北京奇虎科技有限公司 | Method and system for distributing messages according to priorities |
| CN103051521A (en) * | 2013-01-09 | 2013-04-17 | 浪潮电子信息产业股份有限公司 | Retransmitting method of message in cloud environment |
| CN103390049A (en) * | 2013-07-23 | 2013-11-13 | 南京联创科技集团股份有限公司 | Method for processing high-speed message queue overflow based on memory database cache |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105068864A (en) | 2015-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105068864B (en) | Method and system for processing asynchronous message queue | |
| US11182098B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| US20180365254A1 (en) | Method and apparatus for processing information flow data | |
| WO2019133928A1 (en) | Hierarchical, parallel models for extracting in real-time high-value information from data streams and system and method for creation of same | |
| US12008027B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| US11301425B2 (en) | Systems and computer implemented methods for semantic data compression | |
| US20150347305A1 (en) | Method and apparatus for outputting log information | |
| US9507821B2 (en) | Mail indexing and searching using hierarchical caches | |
| WO2014145092A2 (en) | Hierarchical, parallel models for extracting in real time high-value information from data streams and system and method for creation of same | |
| CN112307037A (en) | Data synchronization method and device | |
| JP2016194921A (en) | Removal of old item in curated content | |
| CN111522786A (en) | Log processing system and method | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| CN109977139B (en) | Data processing method and device based on class structured query statement | |
| Hurst et al. | Social streams blog crawler | |
| CN108121807B (en) | Method for realizing multi-dimensional Index structure OBF-Index in Hadoop environment | |
| CN114416717A (en) | Data processing method and architecture | |
| CN115809311A (en) | Data processing method and device of knowledge graph and computer equipment | |
| CN115221116A (en) | Data writing method, device and equipment and readable storage medium | |
| CN114205424A (en) | Bill file decompression method and device, computer equipment and storage medium | |
| CN111191103B (en) | Method, device and storage medium for identifying and analyzing enterprise subject information from internet | |
| CN114048228A (en) | State storage updating method, device, equipment and storage medium | |
| EP3380906A1 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| CN116719821B (en) | Concurrent data insertion elastic search weight removing method, device and storage medium | |
| CN118175130A (en) | Short message system based on self-defined RPC |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |