CN114579330A - Sequential message processing method, device, equipment and storage medium - Google Patents

Sequential message processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN114579330A
CN114579330A CN202210146115.6A CN202210146115A CN114579330A CN 114579330 A CN114579330 A CN 114579330A CN 202210146115 A CN202210146115 A CN 202210146115A CN 114579330 A CN114579330 A CN 114579330A
Authority
CN
China
Prior art keywords
message
queue
node
consumption
key information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210146115.6A
Other languages
Chinese (zh)
Inventor
肖文君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bigo Technology Pte Ltd
Original Assignee
Bigo Technology Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bigo Technology Pte Ltd filed Critical Bigo Technology Pte Ltd
Priority to CN202210146115.6A priority Critical patent/CN114579330A/en
Publication of CN114579330A publication Critical patent/CN114579330A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/548—Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer And Data Communications (AREA)

Abstract

The application discloses a sequential message processing method, a device, equipment and a storage medium. The technical scheme provided by the application comprises the following steps: pulling a first message from a first message queue of a message management end; determining whether the first message meets a preset consumption condition according to the key information of the first message; the messages configured with the same key information are stored in the same first message queue according to a preset consumption sequence; and when the first message is determined to meet the consumption condition, consuming the first message, and sending a first transfer instruction to the message management terminal when the first message is failed to be consumed, so that the message management terminal creates a first message node corresponding to the key information, and transfers the first message from the first message queue to a preset message queue through the first message node. Through the technical scheme, the problems that in the prior art, the message consumption efficiency is low and the consumption system is easy to block messages are solved, the message consumption efficiency is improved, and the stability of the consumption system is ensured.

Description

Sequential message processing method, device, equipment and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing a sequence message.
Background
A message producer in the message system can generate a sequence message to a message consumer for consumption, the sequence message comprises a plurality of messages consumed according to a preset consumption sequence, and the message consumer needs to consume the sequence message according to the consumption sequence so as to avoid the message system from generating information confusion.
The current sequential message processing method is to store sequential messages into the same message queue according to the consumption sequence, and a message consumer pulls and consumes the messages at the head of the message queue according to the first-in first-out principle of the message queue. When a message consumer fails to consume the head of the message queue, the message consumer retries to consume the message until the consumption is successful. However, when the message at the non-head part of the message queue is not related to the sequence of the head message, if the consumption time of the message is long, the consumption efficiency of the subsequent part of the message is seriously influenced, which causes abnormal situations such as message blocking and the like to occur in the message system, and influences the stability of the message system.
Disclosure of Invention
The application provides a sequential message processing method, a sequential message processing device and a sequential message processing storage medium, which are used for solving the problems that in the prior art, the message consumption efficiency is low and a consumption system is easy to have message blockage, improving the consumption efficiency of sequential messages and ensuring the stability of the consumption system.
In a first aspect, the present application provides a sequential message processing method, including:
pulling a first message from a first message queue of a message management end;
determining whether the first message meets a preset consumption condition according to the key information of the first message; the messages configured with the same key information are stored in the same first message queue according to a preset consumption sequence;
and when the first message is determined to meet the consumption condition, consuming the first message, and sending a first transfer instruction to the message management terminal when the first message is failed to be consumed, so that the message management terminal creates a first message node corresponding to the key information, and transfers the first message from a first message queue to a preset message queue through the first message node.
In a second aspect, the present application provides a method for processing a sequential message, including:
adding the first messages to corresponding first message queues according to a preset consumption sequence according to key information of the first messages;
when a first transfer instruction sent by a message consumption end is received, a first message node corresponding to the key information is created according to the key information of the first message, and the first message is transferred from a first message queue to a preset message queue through the first message node.
In a third aspect, the present application provides a sequential message processing apparatus, including:
the message management terminal comprises a first message pulling module, a second message pulling module and a message management module, wherein the first message pulling module is configured to pull a first message from a first message queue of the message management terminal;
the condition judging module is configured to determine whether the first message meets a preset consumption condition according to the key information of the first message; the messages configured with the same key information are stored in the same first message queue according to a preset consumption sequence;
a first instruction sending module, configured to consume the first message when it is determined that the first message satisfies the consumption condition, and send a first transfer instruction to the message management end when the first message consumption fails, so that the message management end creates a first message node corresponding to the key information, and transfers the first message from a first message queue to a preset message queue through the first message node.
In a fourth aspect, the present application provides a sequential message processing apparatus, including:
the message adding module is configured to add the first messages to the corresponding first message queues according to the key information of the first messages and the preset consumption sequence;
the message forwarding method includes the steps that when a first forwarding instruction sent by a message consuming end is received, a first message node corresponding to key information is created according to the key information of the first message, and the first message is forwarded from a first message queue to a preset message queue through the first message node.
In a fifth aspect, the present application provides a sequential message processing apparatus, comprising: a memory and one or more processors;
the memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a sequential message processing method as described in the first aspect or the second aspect.
In a sixth aspect, the present application provides a storage medium containing computer-executable instructions for performing the sequential message processing method of the first or second aspect when executed by a computer processor.
According to the method and the device, whether the first message which is consumed currently meets the consumption sequence corresponding to the first message is determined according to the key information of the first message, if not, the first message is not consumed so as to follow the consumption sequence of the sequence message, and therefore consumption errors are avoided. And if so, consuming the first message, and recording key information of the first message by creating a corresponding message node when the consumption fails so as to detect whether the messages belonging to the same sequence message as the first message meet the consumption sequence or not in the following. The first message is transferred to the preset message queue through the message node, so that the first message is prevented from blocking other subsequent messages of the first message queue, the consumption efficiency of the sequential messages is improved, and the stability of a consumption system is ensured.
Drawings
Fig. 1 is a flowchart of a sequential message processing method according to an embodiment of the present application;
fig. 2 is a flowchart of a message consuming side determining whether a first message satisfies a consuming condition according to an embodiment of the present application;
fig. 3 is a flowchart of transferring a first message according to a first transfer instruction by the consumption management side according to an embodiment of the present application;
FIG. 4 is a structural diagram of an index tree provided in an embodiment of the present application;
fig. 5 is a flowchart illustrating that the message management terminal according to the embodiment of the present application transfers the first message according to the second transfer instruction;
fig. 6 is a flowchart illustrating another sequential message processing method according to an embodiment of the present application;
fig. 7 is a flowchart illustrating a message manager adding a third message at the head of a third message queue to the tail of a second message queue according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a sequential message processing apparatus according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of another sequential message processing apparatus according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a sequential message processing apparatus according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, specific embodiments of the present application will be described in detail with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some but not all of the relevant portions of the present application are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.
The sequential message processing method provided in the embodiment of the present application may be executed by a sequential message processing device, where the sequential message processing device may be implemented in a software and/or hardware manner, and the sequential message processing device may be formed by two or more physical entities or may be formed by one physical entity. In this embodiment, the sequential message processing device may be a message system, or may be a message management end and a message consumption end in a consumption system. The description is given by taking the message system as a Kakfa message subscription system as an example, where the Kakfa message subscription system includes Producers, Brokers and Consumers, where the Brokers is used as a message management end in the message system, the Consumers is used as a message consumption end in the message system, and the Producers is used as a message production end in the message system. The Producers produce the messages and then send the messages to the Brokers, and the Brokers stores the message body into the storage space and stores the address information of the message in the storage space into the tail part of the message queue. Consumer pulls the address information of the message from the head of the message queue, and pulls the message body from the corresponding storage space after analyzing the address information, and consumes the message body. In brief, the message management end is used for managing the message generated by the message production end, and the message consumption end is used for pulling the message from the message management end and consuming the message. It should be noted that the message system in this embodiment is not limited to the Kafka message subscription system, and other message systems related to the message management side and the message consumption side are all applicable to this embodiment.
In one embodiment, the sequence message comprises a plurality of messages for consumption in a predetermined consumption order, for example, the sequence message may be an order message of a shopping platform, and the order message comprises an order placing message, a distribution message and a delivery message. The message consumer end needs to process the order placing message before processing the distribution message and the delivery message, and if the distribution message is processed first, the order message confusion of the shopping platform is affected. The traditional sequential message processing method is that a message management end stores sequential messages into the same message queue according to a consumption sequence, and a message consumption end pulls the messages at the head of the message queue and acquires a queue lock of the message queue through a single thread according to a first-in first-out principle of the message queue, so that other threads are prevented from pulling the messages at the head of the message queue, and the messages are repeatedly consumed. When a single thread fails to consume the message, the message is consumed again at intervals until the consumption is successful, and the intervals are generally set to be smaller so as to avoid blocking the application. However, if the message consuming end has network abnormality or program restart, the interval time is not enough to make the message consuming end recover to normal, and the message needs to be delayed for being consumed. The message queue also stores other sequence messages, and because the single thread locks the head message, other threads cannot consume the head message and other sequence messages in the message queue, so that the consumption efficiency of the sequence messages is seriously influenced, the message system has abnormal conditions such as message blocking and the like, and the stability of the message system is influenced.
In order to solve the above problem, an embodiment of the present application provides a sequential message processing method to prevent a message from blocking a message queue, and improve consumption efficiency of the message.
Fig. 1 is a flowchart illustrating a sequential message processing method according to an embodiment of the present application. The sequential message processing method provided by the embodiment of the application can be executed by a message system.
The following description will be made by taking a message system as an example of a main body for executing the sequential message processing method. Referring to fig. 1, the sequential message processing method includes:
s110, the message management terminal adds the first message to the corresponding first message queue according to the preset consumption sequence according to the key information of the first message.
The first messages are messages produced by a message producer, each first message records corresponding key information, the key information is used for marking sequential messages corresponding to the first messages, and the key information of the first messages belonging to the same sequential message is the same. In one embodiment, when defining the message structure of the first message, the message producer reserves a field to store key information of the first message, for example, the message structure of the first message is { "mqKey": "12345", "orderId": "12345", "amount": 100} where the mqKey is key information of the first message and the orderId is a message identification of the first message.
In an embodiment, the key information may be a key word, and the first messages belonging to the same sequence message are stored in the same first message queue through the same key word, so that the indexing efficiency is improved, and the grouping difficulty is reduced. The key information may also be a hash value, and the first messages belonging to the same sequential message are stored in the same first message queue by the same hash value.
Further, after receiving the first message sent by the message producer, the message management end stores the message body of the first message into a preset storage space, and stores the address information of the message body in the storage space into the first message queue. The first message queue refers to a common message queue in a message management terminal, and is used for storing the first message produced by the message producer. In the embodiment, according to the sequence of the output time of the first messages, the first messages with the same key information are sequentially stored at the tail of the same first message queue, the first messages which are output first are stored in the first message queue first, and the first messages which belong to the messages in the same sequence are ensured to be stored in the same first message queue according to the corresponding consumption sequence.
S120, the message consumption end pulls the first message from the first message queue of the message management end.
Illustratively, the message consumer pulls address information to the first message from the first message queue head and locks the first message queue. And the message consumption end pulls the corresponding message body of the first message from the storage space according to the address information, and analyzes the field of the message body to obtain the key information of the first message.
S130, the message consumption end determines whether the first message meets a preset consumption condition according to the key information of the first message; and the messages configured with the same key information are stored in the same first message queue according to a preset consumption sequence.
Wherein the consumption condition is understood to be a condition that needs to be satisfied when the first message is consumed to comply with the corresponding consumption order. For example, when the first message satisfies the consumption condition, it indicates that the message consumption end consumes the first message at the time obeys the consumption order of the first message, so the message consumption end can consume the first message. If the first message does not satisfy the consumption condition, it indicates that the message consumption end consumes the first message at this time without following the consumption sequence of the first message, so the message consumption end cannot consume the first message.
In an embodiment, fig. 2 is a flowchart of a message consuming side determining whether a first message satisfies a consuming condition according to an embodiment of the present application. As shown in fig. 2, the step of determining whether the first message satisfies the consumption condition by the message consuming side specifically includes S1301-S1303:
s1301, the message consumption end determines whether the preset index tree includes a first message node corresponding to the key information according to the key information of the first message.
S1302, when the message consumption end determines that the index tree comprises the first message node, determining that the first message does not meet the consumption condition.
S1303, when the message consumption end determines that the index tree does not include the first message node, it is determined that the first message meets the consumption condition.
The index tree can be understood as a data structure body configured by a message management end and used for recording key information of sequential messages with consumption failure. Illustratively, a plurality of index nodes are arranged on the index tree, one index node is used for recording key information of one sequence message, when consumption of a certain message in the sequence message fails, the message management terminal records the key information of the message in the index node of the index tree, and generates the key information, namely, the message node corresponding to the sequence message. The message consumption end can determine the sequence message of the consumption failure by inquiring the message nodes on the index tree.
In this embodiment, a message node on the index tree is queried according to the key information of the first message, and if a message node corresponding to the key information is queried on the index tree, it may be determined that a message consumed preferentially in sequential messages corresponding to the first message has failed to be consumed, and at this time, the first message is not consumed according to the corresponding consumption sequence, and it is further determined that the first message does not satisfy the consumption condition. Correspondingly, if the message node corresponding to the key information is not queried on the index tree, it may be determined that the message consumed preferentially in the sequential message corresponding to the first message is consumed successfully or the first message is a preferentially consumed message in the sequential message, at this time, the first message of the message may obey the corresponding consumption order, and thus it is determined that the first message satisfies the consumption condition.
S140, when the message consumption end determines that the first message meets the consumption condition, the first message is consumed, and a first transfer instruction is sent to the message management end when the first message consumption fails.
Illustratively, the message consuming side consumes the first message if the first message satisfies the consuming condition. And after the message consumption end successfully consumes the first message, sending a third removal instruction to the message management end. And after receiving the third removal instruction, the message management end removes the first message at the head of the first message queue, and all the first messages of the first message queue move forwards. After the message management end completes the third removal instruction, the message consumption end is informed that the third removal instruction is completed, and the message consumption end unlocks the first message queue so that other message consumption ends can pull and consume a new first message from the head of the first message queue.
Further, if the message consumption end can not normally consume the first message because of network abnormity or program restart, the message consumption end consumes the first message again at a certain time interval. And if the retry times of the message consumption end reach a first preset time, determining that the first message consumption fails. When the first message consumption fails, the message consumption end sends a first transfer instruction to the message management end, the message consumption end is informed of the first message consumption failure through the first transfer instruction, so that the message management end creates a first message node corresponding to the key information, and the first message is transferred from the first message queue to a preset message queue through the first message node. After the message management terminal completes the second transfer instruction, the message consumption terminal is informed that the first transfer instruction is completed, and the message consumption terminal unlocks the first message queue so that other message consumption terminals can pull and consume the messages in the first message queue.
In an embodiment, the first preset number of times may be set according to a message quantity level of the consuming system, and if there are fewer messages in the message system, the first preset number of times may be set to a larger value, and the message system will not be seriously blocked by the consumption failure of the first message. If there are more messages in the messaging system, the first predetermined number of times may be set to a smaller value, and the failure to consume the first message may severely block the messaging system.
S150, when the message management end receives a first transfer instruction sent by the message consumption end, a first message node corresponding to the key information is established according to the key information of the first message, and the first message is transferred from the first message queue to a preset message queue through the first message node.
In an embodiment, the message management end determines key information of a first message of which the message consumption end fails to consume according to the first transfer instruction, creates a corresponding first message node on the index tree according to the key information, and transfers the first message to the tail of the second message queue and the tail of a third message queue corresponding to the first message node. The first message node refers to a message node of a sequential message corresponding to the first message. As can be seen from the foregoing, the first message may be understood as a message that the first consumption of the corresponding sequential message fails, key information of the sequential message is recorded by an index node on the index tree to generate a message node corresponding to the key information, and the message node represents that the sequential message corresponding to the recorded key information fails to be consumed. Further, the message management end creates a third message queue corresponding to the message node at a leaf node of the message node, removes the address information of the first message from the first message queue, stores the address information of the first message into the tail of the third message queue corresponding to the message node, and records the message with consumption failure in the sequential message through the third message queue. Understandably, the first message is the first message stored in the third message queue, so that the address information of the first message is stored at the head of the third message queue. Further, the message management terminal adds the message at the head of the third message queue to the tail of the second message queue. The second message queue may be understood as a non-blocking message queue in which messages at the head of the third message queue corresponding to each message node are stored. The message consumer pulls these consumed failed messages from the non-blocking message queue for secondary consumption.
In this embodiment, fig. 3 is a flowchart of transferring, by the consumption management end according to the first transfer instruction, the first message provided in the embodiment of the present application. As shown in fig. 3, the step of the consumption manager transferring the first message according to the first transfer instruction specifically includes S1501 to S1503:
s1501, creating an index tree and a second message queue.
Illustratively, if the first message is a first message of failed consumption in the message system, the consumption management side first creates an index tree and a second message queue. And recording key information of the sequence message of the consumption failure through the index tree, and ensuring that the sequence message is consumed according to the corresponding message sequence. And caching the consumption-failed sequential messages through the second message queue so as to be consumed again for the subsequent consumption of the consumption-failed sequential messages.
S1502, creating a first message node corresponding to the key information on the index tree according to the key information of the first message, and creating a third message queue corresponding to the first message node.
In one embodiment, the index tree is a B + tree data structure. Fig. 4 is a schematic structural diagram of an index tree provided in an embodiment of the present application. As shown in fig. 4, assuming that the key information of the first message is the first key information, an index node is added to the index tree, and the first key information is stored through the index node, thereby completing creation of the first message node. And creating a corresponding third message queue and a field recorded with the first key information and a field recorded with a flag bit on a leaf node of the index node. The flag bit field is used for storing a flag bit, and the flag bit can represent whether the consumption management end is currently transferring the message of the first message queue to the corresponding third message queue. Referring to fig. 4, the storage space adopts a commit log message storage structure for storing a message body, and the address information of the message refers to a displacement of the message body on the commit log. The first displacement is the displacement of the message body of the first message stored in the third message queue on the commit log. In another embodiment, the index tree may also employ a red-black tree, an AVL tree, or other data structure having a temporal complexity below log 2N.
Similarly, referring to fig. 4, the sequence messages with the key information of mqKey2 and mqKey3 are also stored in the index tree as the sequence messages with the key information of mqKey 1.
S1503, the first message is transferred to the tail of the second message queue and the tail of a third message queue corresponding to the first message node.
Illustratively, after removing the address information of the first message from the head of the first message queue, the message management side adds the address information of the first message to the tail of the corresponding third message queue. The first message at the head of the third message queue is added to the tail of the second message queue.
And S160, when the message consumption end determines that the first message does not meet the consumption condition, sending a second transfer instruction to the message management end.
For example, if the first message does not satisfy the consumption condition, the consumption of the first message fails in the priority consumption message in the sequential message corresponding to the first message, and at this time, consuming the first message disturbs the consumption sequence of the sequential message, so that the first message is not consumed. And sending a second transfer instruction to the message management end, informing the message management end of the first message belonging to the sequence message of consumption failure through the second transfer instruction so that the message management end determines a first message node corresponding to the key information, and transferring the first message from the first message queue to a preset message queue through the first message node. And after the message management terminal finishes the second transfer instruction, the message consumption terminal is informed that the second transfer instruction is finished, and the message consumption terminal unlocks the first message queue so that other message consumption terminals can pull and consume the messages in the first message queue.
S170, when the message management end receives a second transfer instruction sent by the message consumption end, according to the key information of the first message, determining a first message node corresponding to the key information, and transferring the first message from the first message queue to a preset message queue through the first message node.
In an embodiment, fig. 5 is a flowchart of transferring a first message according to a second transfer instruction by a message management terminal according to an embodiment of the present application. As shown in fig. 5, the step of the message management end transferring the first message according to the second transfer instruction specifically includes S1701-S1702:
s1701, determining a corresponding first message node on the index tree according to the key information of the first message.
S1702, transferring the first message to the tail of a third message queue corresponding to the first message node.
For example, as can be seen from the foregoing, if a message node of the first message has been recorded on the index tree with the key information, a new message node does not need to be created, and the corresponding message node is determined according to the key information, and the address information of the first message is added to the tail of the third message queue. Referring to fig. 4, assuming that the key information of the first message is the first key information, and the first message is the message stored in the third message queue corresponding to the second one of the sequential messages, the second displacement of the message body of the first message on the commit log is stored at the tail of the third message queue. And meanwhile, removing the first message at the head of the first message queue so that other message consuming terminals continue to pull and consume other messages of the first message queue.
In an embodiment, after receiving the second transfer instruction, the message manager sets the flag bit corresponding to the first message node as the second flag information, and sets the flag bit as the first flag information after the first message is added to the tail of the third message queue. The flag bit corresponding to the first message node refers to a flag bit field of a leaf node of the first message node. And when the flag bit field is the first flag information, indicating that the consumption management end does not transfer the messages of the first message queue to the corresponding third message queue currently. When the flag bit field is the second flag information, it indicates that the consumption management end is currently transferring the messages of the first message queue to the corresponding third message queue. Illustratively, after receiving the second transfer instruction, the message management end sets the flag bit corresponding to the first message node as the second flag information to notify, through the second flag information, the message consumption end that is consuming the sequential message corresponding to the first message node, and the consumption management end is currently transferring the message in the first message queue to the corresponding third message queue to prevent the message consumption end from consuming the sequential message. After the first message is successfully added to the tail of the third message queue, the message management end sets the flag bit corresponding to the first message node as first flag information, so as to inform the message consumption end which is about to consume the sequential message corresponding to the first message node through the first flag information, and the message consumption management end does not transfer the message of the first message queue to the corresponding third message queue so that the message consumption end can consume the sequential message.
In summary, in the method for processing sequential messages provided in the embodiment of the present application, the message consumption end determines whether the current consumption of the first message satisfies the consumption sequence corresponding to the current consumption of the first message according to the key information of the first message, and if not, the message consumption end does not consume the first message to comply with the consumption sequence of the sequential messages, thereby avoiding consumption errors. And if so, consuming the first message, and recording key information of the first message by creating a corresponding message node when the consumption fails so as to detect whether the messages belonging to the same sequence message as the first message meet the consumption sequence or not in the following. The message management terminal transfers the first message to the preset message queue through the message node, so that the first message is prevented from blocking other subsequent messages of the first message queue, the message consumption efficiency is improved, and the stability of the consumption system is ensured.
On the basis of the foregoing embodiment, fig. 6 is a flowchart illustrating another sequential message processing method provided in this embodiment. The sequential message processing method is an embodiment of the above sequential message processing method. As shown in fig. 6, the sequential message processing method includes:
s210, the message consumption end pulls the second message from the head of the second message queue, and determines a corresponding second message node on the index tree according to the key information of the second message.
In an embodiment, the message management side is configured with a plurality of second message queues, and stores the head message of the third message queue on the index tree through the plurality of second message queues. Because the key information of the messages stored in the second message queue is different, the message management terminal can store the messages at the tail of the second message queue randomly. And the second message is a message stored in the second message queue. The message consumption end can respectively pull the corresponding second messages from the heads of the plurality of second message queues through multithreading, and the message consumption efficiency is improved through the concurrent consumption of the messages of consumption failure in the non-blocking message queues through the multithreading.
In this embodiment, after the message consumer pulls the displacement of the second message on the commit log from the head of the second message queue, the message consumer pulls the message body of the second message from the commit log according to the displacement. And the message consumption end analyzes the field of the message body to acquire the key information of the second message. And querying a corresponding second message node on the index tree according to the key information of the second message.
S220, the message consumption end determines that the zone bit of the second message node is the first zone information or the second zone information; when determining that the flag bit of the second message node is the first flag information, executing step S230; when the flag bit of the second message node is determined to be the second flag information, step S310 is executed.
For example, when the message consumption end determines that the flag bit field of the leaf node of the second message node records the first flag information, it is determined that the consumption management end does not transfer the message of the first message queue to the corresponding third message queue currently, so that the third message can be pulled from the third message queue for consumption. On the contrary, when the message consumption end determines that the flag bit field of the leaf node of the second message node records the second flag information, it determines that the consumption management end is currently transferring the message of the first message queue to the corresponding third message queue. If the third message is pulled from the third message queue for consumption, the message that the current consumption management end transfers from the first message queue to the third message queue may be missed to consume, resulting in consumption errors.
S230, the message consuming end pulls a third message from the head of a third message queue corresponding to the second message node, and consumes the third message; when the third message consumption is successful, performing step S240; when the third message consumption fails, step S290 is performed.
S240, the message consumption end sends a first removal instruction to the message management end.
And S250, removing the third message at the head of the third message queue when the message management terminal receives the first removal instruction sent by the message consumption terminal.
S260, the message consuming end determines whether the third message queue is an empty queue; when the third message queue is an empty queue, executing step S270; when the third message queue is not an empty queue, step S230 is performed.
Wherein the third message refers to a message in the third message queue. For example, it is assumed that the third message queue corresponding to the second message M1 stores third messages M1_1 to M1_ n from head to tail, where the second message M1 is the same as the third message M1_ 1. And after determining that the flag bit of the second message node is the first flag information, the message consumption end acquires a third message M1_1 from the head of the corresponding third message queue and consumes the third message M1_ 1. If the consumption of the third message M1_1 is successful, the message consumption end sends a first removal instruction to the message management end, and notifies the message management end that the message at the head of the third message queue has been consumed through the first removal instruction. After receiving the first removal instruction, the message manager removes the third message M1_1 from the head of the third message queue, and updates the head of the third message queue to the third message M1_ 2. And if the message consumption end determines that the third message queue is not an empty queue, the message consumption end continues to pull the third message M1_2 from the head of the third message queue and consume the third message. When the message consumer consumes the third message M1_ n and the consumption is successful, the message manager removes the third message M1_ n from the head of the third message queue, and the third message queue is an empty queue. At this time, the message consumption end determines that the third message queue is an empty queue, determines that all the sequential messages corresponding to the second message have been consumed, and may send a second removal instruction to the message management end to notify the message management end that the sequential messages corresponding to the second message have been completely consumed through the second removal instruction.
Illustratively, if the message consumption end fails to consume the third message M1_ i due to network fluctuation when consuming the third message M1_ i, a fourth transfer instruction is sent to the message management end to notify the message management end of the failure of consumption of the sequential messages M1_ i to M1_ n corresponding to the second message M1 through the fourth transfer instruction.
S270, the message consumption end sends a second removal instruction to the message management end.
And S280, when the consumption management terminal receives a second removal instruction sent by the message consumption terminal, removing the second message node on the index tree and removing the second message at the head of the second message queue.
For example, the message manager determines that the sequential message M1_1 through M1_ n corresponding to the second message M1 has been completely consumed according to the second removal instruction, and may remove the second message M1 from the second message queue, so that the message consumer pulls the second message M2 of the second message queue. The second message node on the index tree may be removed because the message consumer requires more effort to access the second message queue than it does to access the first message queue. If the message consumption end pulls the first message with the key information same as that of the second message M1 from the first message queue, the query index tree can determine that the sequential message corresponding to the first message is successfully consumed, and further the first message can be consumed, so that the calculation is saved.
Further, after the message management side removes the second message M1 at the head of the second message queue, the consuming side returns to execute step S230.
And S290, the message consumption end sends a fourth transfer instruction to the message management end.
S300, when the message management terminal receives a fourth transfer instruction sent by the message consumption terminal, the second message at the head of the second message queue is removed, and the third message at the head of the third message queue is added to the tail of the second message queue.
Illustratively, the message management end determines that the sequential messages M1_ i-M1 _ n corresponding to the second message M1 have not been consumed according to the fourth branch instruction, and adds the third message M1_ i at the head of the third message queue to the tail of the second message queue, so that when the subsequent message consumption end pulls the message from the head of the second message queue to M1_ i, the message management end consumes M1_ i-M1 _ n again, and ensures that the sequential messages M1_ i-M1 _ n can be consumed again. Further, the message manager removes the second message M1 from the head of the second message queue, so as to avoid the sequential message M1_ i corresponding to the second message M1 from blocking the second message queue due to consumption failure.
In an embodiment, fig. 7 is a flowchart of adding, by the message management side according to the embodiment of the present application, a third message at the head of a third message queue to the tail of a second message queue. As shown in fig. 7, the step of adding, by the message manager, the third message at the head of the third message queue to the tail of the second message queue specifically includes S3001 to S3002:
s3001, creating a delay queue corresponding to the second message queue and adding the third message at the head of the third message queue to the delay queue.
And S3002, setting the trigger time of the delay queue, so as to add the third message in the delay queue to the tail of the corresponding second message queue at the trigger time.
For example, since the message management end only needs to add the third message M1_ i at the head of the third message queue to the tail of the second message queue, the second message queue does not store the message with the same key information as the third message M1_ i, that is, the third message M1_ i can be regarded as a normal message without observing the consumption sequence, and at this time, the time when the third message M1_ i is added to the tail of the second message queue can be set according to the actual requirement. If the current message consuming end fails to consume the third message M1_ i due to network anomaly or program restart, a longer trigger time may be set, so that the delay queue adds the third message M1_ i to the tail of the second message queue at a longer interval.
S310, the message consumption end sends a third transfer instruction to the message management end.
Illustratively, when the message consumption end determines that the consumption management end is currently transferring the messages in the first message queue to the corresponding third message queue, in order to avoid the missing consumption of the messages, a third transfer instruction is sent to the message management end. And informing the message consumption end that the sequence messages M1_ i-M1 _ n corresponding to the second message M1 cannot be consumed currently through the third transfer instruction.
In another embodiment, if the message management side has a short time to transfer the message in the first message queue to the third message queue, the message consumption side may query the flag of the second message node corresponding to the second message M1 at certain time intervals, and execute step S230 when the flag is the first flag information.
S320, when the message management terminal receives a third transfer instruction sent by the message consumption terminal, the second message at the head of the second message queue is removed, and the third message at the head of the third message queue is added to the tail of the second message queue.
For example, after the message management end determines that the message consumption end cannot currently consume the sequential messages M1_1 to M1_ n corresponding to the second message M1 according to the third transfer instruction, the message management end removes the second message M1 at the head of the second message queue to avoid the second message M1 from blocking the second message queue. Further, after delaying a certain time through the delay queue, the message management end adds the message M1_1 at the head of the third message queue to the tail of the second message queue, so that the subsequent message consumption end consumes the sequential messages M1_ 1-M1 _ n. The step of delaying the addition of the message to the tail of the second message queue by the message management side through the delay queue may specifically refer to steps S3001 to S3002.
In an embodiment, after determining that the flag bit of the second message node is the first flag information, the message consuming end pulls a displacement of the third message on the commit log from a head of a third message queue corresponding to the second message node, and determines whether the displacement of the third message is the same as the displacement of the second message. Because the second message stored in the second message queue is generally the third message corresponding to the head of the third message queue, if the displacement of the third message is different from that of the second message, it can be determined that a certain consumption link is in error, and consumption abnormal information can be sent to a worker, so that the worker can check which link is in error, and the consumption accuracy is ensured.
In summary, the second message is pulled from the second message queue through the message consumption end, the third message queue in which the sequence message corresponding to the second message is stored is determined according to the key information of the second message, and the sequence message in the third message queue is consumed, so that the sequential consumption of the consumption failure is re-consumed, the sequential message consumption efficiency is improved, and the stability of the consumption system is ensured.
On the basis of the foregoing embodiment, fig. 8 is a schematic structural diagram of a sequential message processing apparatus according to an embodiment of the present application. As shown in fig. 8, the sequential message processing apparatus includes a first message pulling module 41, a condition judging module 42, and a first instruction transmitting module 43.
The first message pulling module is configured to pull a first message from a first message queue of a message management end;
the condition judging module is configured to determine whether the first message meets a preset consumption condition according to the key information of the first message; the messages configured with the same key information are stored in the same first message queue according to a preset consumption sequence;
a first instruction sending module, configured to consume the first message when it is determined that the first message satisfies the consumption condition, and send a first transfer instruction to the message management end when the first message consumption fails, so that the message management end creates a first message node corresponding to the key information, and transfers the first message from a first message queue to a preset message queue through the first message node.
On the basis of the foregoing embodiment, fig. 9 is a schematic structural diagram of another sequential message processing apparatus provided in the embodiment of the present application. As shown in fig. 9, the sequential message processing apparatus includes a message adding module 51 and a first message transferring module 52.
The message adding module is configured to add the first message to the corresponding first message queue according to a preset consumption sequence according to the key information of the first message;
the message forwarding method includes the steps that when a first forwarding instruction sent by a message consuming end is received, a first message node corresponding to key information is created according to the key information of the first message, and the first message is forwarded from a first message queue to a preset message queue through the first message node.
In summary, in the device for processing sequential messages provided in the embodiment of the present application, the message consumption end determines whether the current consumption of the first message satisfies the consumption sequence corresponding to the current consumption of the first message according to the key information of the first message, and if not, the message consumption end does not consume the first message to comply with the consumption sequence of the sequential message, thereby avoiding the consumption error. And if so, consuming the first message, and recording key information of the first message by creating a corresponding message node when the consumption fails so as to detect whether the messages belonging to the same sequence message as the first message meet the consumption sequence or not in the following. The message management terminal transfers the first message to the preset message queue through the message node, so that the first message is prevented from blocking other subsequent messages of the first message queue, the consumption efficiency of the sequential messages is improved, and the stability of the consumption system is ensured.
The embodiment of the present application further provides a sequential message processing device, which may integrate the sequential message processing apparatus provided in the embodiment of the present application. Fig. 10 is a schematic structural diagram of a sequential message processing apparatus according to an embodiment of the present application. Referring to fig. 10, the sequential message processing apparatus includes: an input device 63, an output device 64, a memory 62, and one or more processors 61; a memory 62 for storing one or more programs; when executed by the one or more processors 61, cause the one or more processors 61 to implement the sequential message handling method as provided in the embodiments above. The sequential message processing device, the equipment and the computer provided by the above can be used for executing the sequential message processing method provided by any of the above embodiments, and have corresponding functions and beneficial effects.
Embodiments of the present application also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are used to perform the sequential message processing method provided in the above embodiments. Of course, the storage medium provided in this embodiment of the present application contains computer-executable instructions, and the computer-executable instructions are not limited to the above-mentioned sequential message processing method, and may also perform related operations in the sequential message processing method provided in any embodiment of the present application. The device, the apparatus, and the storage medium for processing a sequential message provided in any embodiments of the present application may execute the method for processing a sequential message provided in any embodiments of the present application, and refer to the method for processing a sequential message provided in any embodiments of the present application without detailed technical details described in the embodiments.
The foregoing is considered as illustrative of the preferred embodiments of the invention and the technical principles employed. The present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the claims.

Claims (22)

1. A method for sequential message processing, comprising:
pulling a first message from a first message queue of a message management end;
determining whether the first message meets a preset consumption condition according to the key information of the first message; the messages configured with the same key information are stored in the same first message queue according to a preset consumption sequence;
and when the first message is determined to meet the consumption condition, consuming the first message, and sending a first transfer instruction to the message management terminal when the first message is failed to be consumed, so that the message management terminal creates a first message node corresponding to the key information, and transfers the first message from a first message queue to a preset message queue through the first message node.
2. The sequential message processing method of claim 1, further comprising:
and when the first message is determined not to meet the consumption condition, sending a second transfer instruction to the message management terminal so that the message management terminal determines a first message node corresponding to the key information and transfers the first message from the first message queue to a preset message queue through the first message node.
3. The method of claim 2, wherein the determining whether the first message satisfies the consumption condition according to the key information of the first message comprises:
determining whether a preset index tree comprises a first message node corresponding to the key information or not according to the key information of the first message;
determining that the first message does not satisfy the consumption condition when it is determined that the index tree includes the first message node;
determining that the first message satisfies the consumption condition when it is determined that the index tree does not include the first message node.
4. The method according to claim 3, wherein the sending a first transfer instruction to the message manager to enable the message manager to create a first message node corresponding to the key information, and transfer the first message from the first message queue to a preset message queue through the first message node comprises:
and sending the first transfer instruction to the message management terminal so that the message management terminal creates a corresponding first message node on the index tree according to the key information and transfers the first message to the tail of a second message queue and the tail of a third message queue corresponding to the first message node.
5. The method according to claim 3, wherein the sending a second transfer instruction to the message manager to enable the message manager to determine a message node corresponding to the key information, and transfer the first message from the first message queue to a preset message queue through the first message node comprises:
and sending the second transfer instruction to the message management end so that the message management end determines a corresponding first message node on the index tree according to the key information and transfers the first message to the tail of a third message queue corresponding to the first message node.
6. The method according to claim 4 or 5, wherein after the message manager creates a corresponding first message node on the index tree according to the key information and transfers the first message to a second message queue tail and a third message queue tail corresponding to the first message node, the method further comprises:
pulling a second message from the head of the second message queue, and determining a corresponding second message node on the index tree according to key information of the second message;
pulling a third message from the head of a third message queue corresponding to the second message node, and consuming the third message;
and after the third message is successfully consumed, pulling a new third message from the head of the third message queue, and consuming the new third message until the third message queue is empty.
7. The method of sequential message handling according to claim 6, further comprising, prior to said pulling a third message from a third message queue header corresponding to said second message node:
when the flag bit of the second message node is determined to be the first flag information, pulling a third message from the head of a third message queue corresponding to the second message node;
and when the flag bit of the second message node is determined to be second flag information, sending a third transfer instruction to the message management end, so that the message management end removes the second message at the head of the second message queue and adds the third message at the head of the third message queue to the tail of the second message queue.
8. The sequential message processing method of claim 6, further comprising, after said consuming the third message:
when the third message consumption fails, sending a fourth transfer instruction to the message management end, so that the message management end removes the second message at the head of the second message queue and adds the third message at the head of the third message queue to the tail of the second message queue;
and when the third message is successfully consumed, sending a first removal instruction to the message management end so that the message management end removes the third message at the head of the third message queue.
9. The sequential message processing method of claim 6, further comprising, after said consuming the third message:
and when the third message queue is an empty queue, sending a second removal instruction to the message management terminal so that the message management terminal removes the second message node on the index tree and removes the second message at the head of the second message queue.
10. A method for sequential message processing, comprising:
adding the first messages to corresponding first message queues according to a preset consumption sequence according to key information of the first messages;
when a first transfer instruction sent by a message consumption end is received, a first message node corresponding to the key information is created according to the key information of the first message, and the first message is transferred from a first message queue to a preset message queue through the first message node.
11. The sequential message processing method of claim 10, wherein said sequential message processing method further comprises:
and when a second transfer instruction sent by the message consumption end is received, determining a first message node corresponding to the key information according to the key information of the first message, and transferring the first message from a first message queue to a preset message queue through the first message node.
12. The method of claim 11, wherein the creating a first message node corresponding to the key information according to the key information of the first message, and transferring the first message from a first message queue to a preset message queue through the first message node comprises:
creating an index tree and a second message queue;
creating a first message node corresponding to the key information on the index tree according to the key information of the first message, and creating a third message queue corresponding to the first message node;
and transferring the first message to the tail part of the second message queue and the tail part of a third message queue corresponding to the first message node.
13. The method of claim 12, wherein the determining a first message node corresponding to the key information according to the key information of the first message, and transferring the first message from a first message queue to a preset message queue through the first message node comprises:
determining a corresponding first message node on the index tree according to the key information of the first message;
and transferring the first message to the tail part of a third message queue corresponding to the first message node.
14. The method according to claim 12 or 13, wherein said transferring the first message to the tail of a third message queue corresponding to the first message node comprises:
and setting the flag bit corresponding to the first message node as second flag information, and setting the flag bit as first flag information after the first message is added to the tail of the third message queue.
15. The method of claim 12 or 13, further comprising, after the transferring the first message to the second message queue tail and a third message queue tail corresponding to the first message node:
and when a third transfer instruction or a fourth transfer instruction sent by the message consumption end is received, removing the second message at the head of the second message queue, and adding the third message at the head of the third message queue to the tail of the second message queue.
16. The method of sequential message processing according to claim 15, wherein said adding a third message at the head of said third message queue to the tail of said second message queue comprises:
creating a delay queue corresponding to the second message queue and adding a third message at the head of the third message queue to the delay queue;
and setting the trigger time of the delay queue so as to add the third message in the delay queue to the tail part of the corresponding second message queue at the trigger time.
17. The method of claim 12 or 13, further comprising, after the transferring the first message to the second message queue tail and a third message queue tail corresponding to the first message node:
and removing the third message at the head of the third message queue when receiving a first removal instruction sent by the message consumption end.
18. The method of claim 12 or 13, further comprising, after the transferring the first message to the second message queue tail and a third message queue tail corresponding to the first message node:
and when a second removing instruction sent by the message consumption end is received, removing a second message node on the index tree and removing a second message at the head of the second message queue.
19. A sequential message processing apparatus, comprising:
the first message pulling module is configured to pull a first message from a first message queue of a message management end;
the condition judging module is configured to determine whether the first message meets a preset consumption condition according to the key information of the first message; the messages configured with the same key information are stored in the same first message queue according to a preset consumption sequence;
a first instruction sending module, configured to consume the first message when it is determined that the first message satisfies the consumption condition, and send a first transfer instruction to the message management end when the first message consumption fails, so that the message management end creates a first message node corresponding to the key information, and transfers the first message from a first message queue to a preset message queue through the first message node.
20. A sequential message processing apparatus, comprising:
the message adding module is configured to add the first messages to the corresponding first message queues according to the key information of the first messages and the preset consumption sequence;
the message forwarding method includes the steps that when a first forwarding instruction sent by a message consuming end is received, a first message node corresponding to key information is created according to the key information of the first message, and the first message is forwarded from a first message queue to a preset message queue through the first message node.
21. A sequential message processing device, comprising: a memory and one or more processors;
the memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of sequential message processing according to any one of claims 1-18.
22. A storage medium containing computer-executable instructions for performing the method of sequential message processing according to any of claims 1-18 when executed by a computer processor.
CN202210146115.6A 2022-02-17 2022-02-17 Sequential message processing method, device, equipment and storage medium Pending CN114579330A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210146115.6A CN114579330A (en) 2022-02-17 2022-02-17 Sequential message processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210146115.6A CN114579330A (en) 2022-02-17 2022-02-17 Sequential message processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114579330A true CN114579330A (en) 2022-06-03

Family

ID=81773542

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210146115.6A Pending CN114579330A (en) 2022-02-17 2022-02-17 Sequential message processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114579330A (en)

Similar Documents

Publication Publication Date Title
US7702741B2 (en) Configuring or reconfiguring a multi-master information sharing environment
WO2016127903A1 (en) Data synchronization method, device and system
US9009235B2 (en) Ordered message processing
CN107153660B (en) Fault detection processing method and system for distributed database system
EP3172668B1 (en) Speculative data processing of streaming data
EP2723017A1 (en) Method, apparatus and system for implementing distributed auto-incrementing counting
CN103500120A (en) Distributed cache high-availability processing method and system based on multithreading asynchronous double writing
CN106484329A (en) A kind of big data transmission integrity protection mechanism based on multistage storage
CN107153595B (en) Fault detection method and system for distributed database system
CN110941502A (en) Message processing method, device, storage medium and equipment
CN109544344B (en) Block chain transaction processing method and equipment based on DAG
CN109213828B (en) Block generation method, device, equipment and storage medium
CN109298976B (en) Heterogeneous database cluster backup system and method
CN111489079A (en) Capacity bottleneck detection method and device and computer readable storage medium
CN113342893B (en) Node synchronization method and device based on block chain, storage medium and server
EP3031172B1 (en) Managing data feeds
CN112612855B (en) High-availability database log receiving queue, synchronization method and device
CN114579330A (en) Sequential message processing method, device, equipment and storage medium
CN110888739B (en) Distributed processing method and device for delayed tasks
CN109947828B (en) Method and device for processing report data
CN114416717A (en) Data processing method and architecture
CN109241362B (en) Block generation method, device, equipment and storage medium
CN114884906A (en) Failure retry notification method and device based on quick recovery
CN113364874A (en) Node synchronization method and device based on block chain, storage medium and server
CN112905322A (en) Resource locking method, computing device and computer storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination