WO2020238365A1

WO2020238365A1 - Message consumption method, apparatus and device, and computer storage medium

Info

Publication number: WO2020238365A1
Application number: PCT/CN2020/080930
Authority: WO
Inventors: 黄山; 尹强; 王和平; 刘有; 杨峙岳; 邸帅; 卢道和
Original assignee: 深圳前海微众银行股份有限公司
Priority date: 2019-05-31
Filing date: 2020-03-24
Publication date: 2020-12-03
Also published as: CN110213371A; CN110213371B

Abstract

Disclosed is a message consumption method, relating to the field of fintech. The method comprises: acquiring a target node in a service registration directory of a distributed coordination service ZooKeeper, and monitoring, on the basis of the target node, a preset number of message queues to determine whether the message queues include a new message queue monitored for the first time; if there is a new message queue monitored for the first time, traversing messages in the new message queue in sequence, acquiring a current message traversed currently, and determining, on the basis of a preset feature recognition algorithm, a target thread corresponding to the current message; and transmitting the current message to the target thread for consumption until the messages in the new message queue are all traversed. Further disclosed are a message consumption apparatus and device, and a computer storage medium.

Description

Message consumption method, device, equipment and computer storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910479100.X, and the invention title is "message consumption methods, devices, equipment, and computer storage media" on May 31, 2019. The reference is incorporated in the application.

Technical field

This application relates to the field of big data technology of financial technology (Fintech), in particular to message consumption methods, devices, equipment and computer storage media.

Background technique

With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are applied in the financial field. The traditional financial industry is gradually transforming to Fintech. However, due to financial The industry's security and real-time requirements also place higher requirements on technology. At present, in the big data statistics process, messages are generated and consumed for large amounts of data. Among them, the generation of messages is transmitted by the message producer to the message server, and the consumption of messages is by the consumer thread The consumption operation of the message of the message server.

In the message consumption process, the prior art generally has single-threaded consumption and multi-threaded consumption. Single-threaded consumption uses a single thread to monitor the message queue. Once the consumer thread obtains a new message, it needs to complete the delivery of the message to all locals. The next message can be consumed only after the process of other services, but this method has serious defects in performance. Multi-threaded consumption uses multiple threads to monitor the same message queue at the same time. As long as a new message appears in the queue, there will be a random thread to obtain it, and then independently complete the delivery process to other local services, but this method cannot guarantee that the message is consumed It is not applicable in scenarios where the order of message state migration is required, which has a great impact on the available range of message consumption and makes the available range smaller. Therefore, how to achieve orderly consumption of messages with high concurrency and high availability has become a technical problem to be solved urgently.

Summary of the invention

technical problem

The solution to the problem

Technical solutions

The main purpose of this application is to propose a message consumption method, device, equipment, and computer storage medium, aiming to solve the technical problem of how to achieve orderly consumption of messages with high concurrency and availability.

In order to achieve the above objective, this application provides a message consumption method, which includes the following steps:

Obtain the target node in the service registration directory of the distributed coordination service zookeeper, and based on the target node monitor a preset number of message queues, determine whether there is a new message queue monitored for the first time in each of the message queues;

If there is a new message queue monitored for the first time, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm;

The current message is delivered to the target thread for consumption until the traversal of messages in the new message queue is completed.

In addition, in order to achieve the above objective, the present application also provides a message consumption device, the message consumption device including:

The obtaining module is used to obtain the target node in the zookeeper service registration directory, and determine whether there is a new message queue monitored for the first time in each of the message queues based on the target node monitoring a preset number of message queues;

The determining module is configured to, if there is a new message queue monitored for the first time, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm;

The delivery module is used to deliver the current message to the target thread for consumption until the message traversal in the new message queue is completed.

In addition, in order to achieve the above objective, the present application also provides a message consumption device, the message consumption device comprising: a memory, a processor, and computer-readable instructions stored on the memory and running on the processor, When the computer-readable instructions are executed by the processor, the steps of the message consumption method described above are implemented.

In addition, in order to achieve the above objective, the present application also provides a computer storage medium having computer readable instructions stored thereon, and when the computer readable instructions are executed by a processor, the above message consumption method is implemented. step.

This application obtains the target node in the zookeeper's service registration directory and monitors a preset number of message queues based on the target node to determine whether there is a new message queue monitored for the first time in each of the message queues; if there is a new message queue monitored for the first time Message queue, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm; deliver the current message to the target thread Consumption is performed until the traversal of the messages in the new message queue is completed. By monitoring a preset number of message queues according to the target node in zookeeper, and when it finds that there is a new message queue that is monitored for the first time, the messages in the new message queue will be traversed and consumed sequentially, thereby ensuring the orderly consumption of messages , The order in which the messages are consumed is consistent with the order in the queue, and since the target thread is determined according to the message, and then the message is passed to the target thread for consumption, it also achieves orderly consumption of messages with high concurrency and high availability Effect.

The beneficial effects of the invention

Brief description of the drawings

Description of the drawings

FIG. 1 is a schematic diagram of a device structure of a hardware operating environment involved in a solution of an embodiment of the present application;

2 is a schematic flowchart of the first embodiment of the message consumption method of this application;

FIG. 3 is a schematic diagram of device modules of the message consumption device of this application;

Figure 4 is a schematic diagram of the node structure of zookeeper in the message consumption method of this application;

Figure 5 is a flow chart of message flow in the message consumption method of this application.

The realization of the purpose, functional characteristics and advantages of the application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Invention embodiment

Embodiments of the invention

It should be understood that the specific embodiments described here are only used to explain the application, and are not used to limit the application.

As shown in FIG. 1, FIG. 1 is a schematic diagram of the device structure of the hardware operating environment involved in the solution of the embodiment of the present application.

The message consumption device in the embodiment of the present application may be a PC or a server device, on which a Java virtual machine runs.

As shown in FIG. 1, the message consumption device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to implement connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a magnetic disk memory. Optionally, the memory 1005 may also be a storage device independent of the foregoing processor 1001.

Those skilled in the art can understand that the structure of the device shown in FIG. 1 does not constitute a limitation on the device, and may include more or fewer components than those shown in the figure, or a combination of certain components, or different component arrangements.

As shown in FIG. 1, a memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and computer readable instructions.

In the device shown in FIG. 1, the network interface 1004 is mainly used to connect to a back-end server and communicate with the back-end server; the user interface 1003 is mainly used to connect to a client (user side) and communicate with the client; and the processor 1001 can be used to call computer-readable instructions stored in the memory 1005 and perform operations in the following message consumption method.

Based on the above hardware structure, an embodiment of the message consumption method of this application is proposed.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of a first embodiment of a message consumption method according to this application. The method includes:

Step S10: Obtain a target node in the service registration directory of the distributed coordination service zookeeper, and determine whether there is a new message queue monitored for the first time in each of the message queues based on the target node monitoring a preset number of message queues;

Zookeeper (distributed coordination service), which is an open source distributed coordination service. Distributed applications can implement functions such as data publishing/subscription, load balancing, naming services, distributed coordination/notification, cluster management, Master election, distributed locks, and distributed queues based on Zookeeper. The message queue can be responsible for the orderly storage of messages for the storage mechanism. The new message queue can be a message queue that has not been monitored by the message consumer corresponding to the temporary node before the current moment, and the message consumer can be used as a client to obtain messages from the message server, and the message server is used as a server to provide core message services . The target node can be a node representing itself created by the message consumer under the zookeeper's service registration directory, and there can be many types of target nodes. In this embodiment, only the target node is a temporary node for illustration, but not The representative target node is limited to one type of temporary node.

In this embodiment, a directory is established under zookeeper as a message queue, and a monitoring mechanism provided by zookeeper is used to implement a message broadcasting mechanism. And each message consumer will establish a temporary node (that is, the target node) on behalf of itself in the zookeeper's service registration directory during the initialization phase, so you need to first obtain the temporary node in the zookeeper's service registration directory (the temporary node is in the message Consumers will be automatically deleted after losing connection with zookeeper), and according to the temporary node, they will monitor a preset number of message queues (any number set by the user in advance, which can be one or multiple) to determine which ones are being monitored. Whether there is a new message queue that is monitored for the first time in the message queue, if there is a new message queue that is monitored for the first time, the message consumer will obtain all the existing messages in the queue and consume them in order. If there is no new message queue monitored for the first time, it will continue to detect whether there are new message node generations in the monitored message queue, and perform different operations according to the detection results.

Among them, when there are multiple temporary nodes obtained, different message queues can be monitored respectively according to each temporary node, that is, each temporary node can simultaneously monitor their corresponding message queues. For example, as shown in Figure 4, when there are three temporary nodes in the zookeeper's service registration directory, namely service 1, service 2, and service 3, and there are two message queues, namely queue 1 and queue 2, in Queue 1 includes message 1, message 2, message 3, and message 4, and queue 2 also includes message 1, message 2, message 3, and message 4. Assuming that both service 1 and service 2 need to listen to message 1 in queue 1, service 1 and service 2 can simultaneously obtain message 1 in queue 1 for consumption, and after consumption is completed, it will be generated in the child node under message 1 Temporary child node service 1 and service 2. Assume that service 2 not only monitors message 1 in queue 1, but also monitors message 2 in queue 1 and message 1 in queue 2. Similarly, when message consumption is completed After that, temporary child node service 1 is generated in the child nodes of message 2 in queue 1 and message 1 in queue 2 respectively.

It should be noted that the nodes in zookeeper can be divided into four types of nodes, namely persistent nodes, persistent sequential nodes, temporary nodes, and temporary sequential nodes. Among them, after the persistent node is created, it will always exist until there is a delete operation to actively clear the node. In this embodiment, the directory where the message queue is located can use this node. The basic characteristics of persistent sequential nodes are the same as those of persistent nodes. Compared with persistent nodes, the additional characteristic of persistent sequential nodes is that each parent node maintains a sequence for its first-level child nodes, and records the creation of each child node In this embodiment, this node can be used to carry the message body, and use its natural order to realize the orderly storage of messages in the queue. The message body can be used as a carrier to carry the message. The content is generally a data packet encoded according to the protocol. The temporary node is automatically deleted after the client loses the connection with the zookeeper. In this embodiment, the automatic removal function of this node can be used to ensure that abnormal consumers will not affect the cleaning of messages and reduce the accumulation of messages. The basic characteristics of the temporary sequence node are the same as the temporary node. Compared with the temporary node, the additional feature of the temporary sequence node is that each parent node maintains a sequence for its first-level child nodes, and records the creation of each child node In order, in this embodiment, this node can be used to implement distributed locks. The message producer (that is, the client is responsible for transmitting messages to the message server) can create this kind of node to realize the queuing mechanism for the same lock , And be notified in real time when the lock is released.

Step S20: If there is a new message queue monitored for the first time, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm;

The current message can be a message being traversed in the new message queue. A thread can be the smallest unit that the operating system can perform operation scheduling. When it is judged that there is a message queue that is monitored for the first time, you can traverse each message in the new message queue in turn, and obtain the current message currently traversed in each message, and then determine the current message corresponding to the preset feature recognition algorithm The target thread. It should be noted that if there are two or more new message queues that are monitored for the first time, each new message queue can be traversed separately at the same time to consume messages.

Among them, the feature recognition algorithm is to ensure that messages of the same category can be distributed locally to the same thread for orderly consumption. When users define a message, they can select some fields in the message as the basis for feature recognition. If the values of these fields are consistent, they can be considered as the same type of message, which means that each type of message needs to be given a guarantee Numbers that will not be repeated are used as characteristic information. When the message consumer receives the message, it will read the characteristic information, and perform the residual operation according to the capacity of the local consumption thread pool, and put the message into the thread marked with the residual result. Among them, the feature recognition algorithm will be combined with the current abnormal monitoring results of the thread pool. Under the premise that the thread processing sequence is understood and synchronized, the thread mark is adjusted to prevent the peak consumption of a certain type of message from being too long.

In step S30, the current message is delivered to the target thread for consumption until the traversal of messages in the new message queue is completed.

When the target thread is obtained, the current message can be directly passed to the target thread for consumption, and when the current message is consumed through the target thread, the message consumer will have a separate monitoring thread to poll the distribution thread Robustness. If a thread is found to be in an abnormal state, the standby thread will be activated immediately to ensure the stability of consumption. And after the current message is consumed in the target thread, it will continue to traverse each message in the new message queue until the traversal of each message is completed.

In order to help understand the consumption of messages, an example is given below.

For example, as shown in Figure 5, when a message producer (for example, a business party that has business interactions with financial institutions such as banks, where the business includes but is not limited to loan business, deposit business, etc.) establishes the correspondence between message nodes under the zookeeper directory When the message queue sends a message, it will trigger the zookeeper's monitoring mechanism. Zookeeper will send a notification to all message consumers listening to the directory. After receiving the notification, the message consumer (such as a bank and other financial institutions) will respond according to the content of the notification. Get the message in the message node. Among them, because the message consumer manages all the threads responsible for local distribution of messages through an ordered thread pool, when receiving a message, it will analyze the characteristics of the message according to the feature recognition algorithm, and combine the size of the thread pool. Determine which distribution thread the message should enter to ensure that messages with the same characteristics can always be consumed in an orderly manner. And the message consumer will have a separate monitoring line to detect each distribution thread, assuming that there are consumer thread 1, consumer thread 2, consumer thread 3 and consumer thread 4 in the thread pool, and the message consumer monitors the thread pool through the monitor at this time Each consumer thread in the consuming thread will obtain monitoring information in the future, and when the consuming thread 4 is found to be in an abnormal state based on the monitoring information, the backup thread will be activated immediately, that is, the backup thread 1 and the consuming thread 4 are selected for exceptions in the backup thread 1 and the backup thread 2. Switch. Among them, each message node is periodically detected in zookeeper, and after it is determined that a message node has been consumed by all message consumers, the message in the message node can be cleaned up.

In this embodiment, by acquiring the target node in the zookeeper's service registration directory, and monitoring a preset number of message queues based on the target node, it is determined whether there is a new message queue monitored for the first time in each of the message queues; For the new message queue, the messages in the new message queue are sequentially traversed, the current message currently traversed is obtained, and the target thread corresponding to the current message is determined based on the preset feature recognition algorithm; the current message is delivered to the target The thread consumes until the traversal of the messages in the new message queue is completed. By monitoring a preset number of message queues according to the target node in zookeeper, and when it finds that there is a new message queue that is monitored for the first time, the messages in the new message queue will be traversed and consumed sequentially, thereby ensuring the orderly consumption of messages , The order in which the messages are consumed is consistent with the order in the queue, and since the target thread is determined according to the message, and then the message is passed to the target thread for consumption, it also achieves orderly consumption of messages with high concurrency and high availability Effect.

Further, based on the first embodiment of the message consumption method of this application, a second embodiment of the message consumption method of this application is proposed. This embodiment is a refinement of the step S20 of the first embodiment of the present application. The step of determining the target thread corresponding to the current message based on a preset feature recognition algorithm includes:

Step S21, receiving feature information corresponding to the current message at the top-level interface corresponding to the current message, and identifying the feature information based on a preset feature recognition algorithm;

The user-defined feature information corresponding to the current message can be received in the top-level interface corresponding to the current message, and when the user defines the message, some fields in the message can be selected as the basis for feature identification, and the values of these fields can be used as feature information. When the message consumer receives the current message, the feature information in the current message can be identified and restored through the preset feature recognition algorithm, so as to determine which thread the message should be consumed by the feature information, and to ensure that messages with the same feature must have Ordered to be consumed.

Step S22: Determine the target thread corresponding to the current message according to the characteristic information.

Since the characteristic information of the same type of messages is the same, and the characteristic information of different types of messages are different, after obtaining the characteristic information corresponding to the current information, you can determine which thread is consuming the characteristic information of the message in the thread pool The characteristic information of the current message is the same, that is, the target thread is determined.

In this embodiment, the characteristic information of the current message is acquired according to the preset characteristic recognition algorithm, and the target thread is determined according to the characteristic information, thereby ensuring that the current message acquires a suitable thread for consumption and guarantees the efficiency of message consumption.

Specifically, the step of determining the target thread corresponding to the current message according to the characteristic information includes:

Step S221: Obtain a thread pool corresponding to the target node, perform a remainder operation according to the capacity of the thread pool and the characteristic information, acquire a first thread with a remainder result mark in the thread pool, and detect all State whether the first thread is in an abnormal state;

The thread pool can be a collection of all threads. After obtaining the characteristic information corresponding to the current message, it is also necessary to obtain the thread pool corresponding to the target node, and determine the capacity of the thread pool, and then perform the remainder operation according to the capacity and characteristic information of the thread pool to determine which thread in the thread pool The characteristic information of the consumed message is the same as the characteristic information of the current message, and the capacity is the largest, so it can be used as the first thread with the remainder result mark. And after acquiring the first thread, it is also necessary to check whether the first thread is in an abnormal state (that is, to check whether the first thread can normally perform the operation of consuming messages), and perform different operations according to the check result.

Step S222, if the first thread is in an abnormal state, enable the standby thread in the thread pool, and use the standby thread as the target thread corresponding to the current message;

When the first thread is found to be in an abnormal state after judgment, the emergency mechanism will be triggered. The mechanism will use the pre-initialized backup thread to replace the abnormal thread, and roll back the local state according to the abnormal scope, and then continue to consume. The thread is the target thread corresponding to the current message. It should be noted that in this embodiment, each step of message consumption is monitored. Once a link fails, times out, or the throughput is lower than the threshold, the emergency mechanism will be triggered. It is not limited to the detection of the first step. Triggered when a thread is in an abnormal state.

Step S223: If the first thread is in a normal state, use the first thread as the target thread corresponding to the current message.

When the first thread is found to be in a normal state after judgment, the first thread can be directly used as the target thread corresponding to the current message, and the current message is delivered to the target thread for consumption.

In this embodiment, after the first thread is determined, whether the first thread is in an abnormal state is detected, and different operations are performed according to different detection results, thereby ensuring the stability of message consumption.

Further, on the basis of any one of the first to second embodiments of the present application, a third embodiment of the message consumption method of the present application is proposed. In this embodiment, in step S10 of the first embodiment of the present application, each location is determined After the steps of whether there is a new message queue monitored for the first time in the message queue, include:

Step S50, if there is no new message queue monitored for the first time, detect whether there is a new message node in each of the message queues;

When it is judged that there is no new message queue monitored for the first time, it is necessary to check again whether there is a new message node generation in each message queue, that is, to determine whether there is a message producer to create a new message node in the message queue, and based on Different test results perform different operations.

Step S60: If there is a new message node, obtain the new message in the new message node, and deliver the new message to the second thread corresponding to the new message node for consumption.

When it is judged that a new message node is generated, the monitoring mechanism of zookeeper will be triggered. Zookeeper will send a notification to all consumers who monitor the message queue where the new message node is located. After the message consumer receives the notification, it will follow The content of the notification pulls the new message in the new message node from zookeeper, calculates and determines the second thread corresponding to the new message node through a preset feature recognition algorithm, and delivers the new message to the second thread for consumption. Among them, the new message may be a message carried by the new message node. The second thread may be the thread used by the new message to be consumed.

In this embodiment, by detecting whether a new message node is generated, and when it is determined that a new message node is generated, the new message is directly obtained and consumed through the second thread, thereby ensuring the efficiency of message consumption.

Specifically, after the step of delivering the new message to the second thread corresponding to the new message for consumption, it includes:

Step S70: Obtain the total message queue in zookeeper, and periodically determine whether there is a consumed message node whose consumption is completed in each message node in the total message queue, wherein the total message queue includes a message queue;

The total message queue can be all message queues in zookeeper. The preset time period can be any time period set by the user. The consumed message node may be a message node where the message has been consumed by each message consumer.

Because the consumption of messages is not highly transactional. Existing technologies cannot guarantee that messages can be consumed 100%, and there are no protective measures after consumption failure, which makes the existing technologies unable to cope with scenarios that require high state synchronization accuracy, and the message body itself provides judgments The status information of expired or not, so the messages in the transitional state can be cleaned up in time, and a message merging mechanism is provided. Part of the important information in the elapsed message can be separately extracted and put into the message of the persistent state.

Obtain all the message queues in zookeeper, and there will be a timing task to check the messages in all the message nodes in each message queue at every preset time period, that is, check the status of the message. In this embodiment, the message Defined as "start", "transition", and "terminate" states to achieve differentiated treatment of messages of different natures, and the identification of message states can be identified through user-defined or automatic middleware identification. User self-definition is when users of middleware define messages, they can assign status to the message by inheriting interface fields. The automatic identification of middleware is to actively analyze the field value of the same type of message in the process of message cleaning to determine whether the message has been merged by other messages. If the information carried by other messages is merged, it intelligently judges the status information of the message. To ensure that even if the user-defined status information has errors, it can still be corrected to maintain the effectiveness of message cleaning. Among them, the messages in the transition state will be cleaned up in time after confirming that they are completely consumed. Therefore, it is necessary to determine whether each message node is a transition message that does not need to be persisted. If it is judged that there is a message node as a transition message, it will be compared. If the temporary child node under the message node and the temporary node under the service registration directory are consistent, it proves that the message in the message node has been consumed by all message consumers, and it can be considered as each message node in the total message queue. There is a consumed message node that has completed consumption, and the messages in the message node can be cleaned up at this time. Among them, the temporary child node may be a temporary child node that the message consumer creates under the message node after receiving the message in the message node, which represents itself, as a basis for the message to be delivered.

Step S80, if it exists, extract and save important information in the consumed message node, and clear the consumed message node in the total message queue.

When it is judged that there is a consumed message node in the total message queue, that is, it can be consumed by all message consumers listening to the message node by default. At this time, the message in the message node is in a transition state, then the transition The important information in the message is extracted, and the important information can be stored in the persistent state according to the message merging mechanism in zookeeper, so that when the user wants to view the important information later, he can query it in time and conveniently. After extracting and saving When the important information in the message node has been consumed, the consumed message node can be cleaned up. Among them, if there is no persistent state message in zookeeper, you can create a persistent node by yourself to save important information. When it is found after judgment that the message node in the total message queue has not been consumed, the message in the message node is kept unchanged.

In this embodiment, by detecting whether there is a consumed message node in the total message queue, and if there is, the consumed message node is cleaned up, thereby improving the efficiency of message cleanup and improving the user experience.

Further, on the basis of the first embodiment of this application, a fourth embodiment of the message consumption method of this application is proposed. In this embodiment, in step S30 of the first embodiment of this application, the current message is transferred to all After the steps for the target thread to consume, include:

Step A10, detecting whether the current message is successfully consumed in the target thread;

When the current message is passed to the target thread for consumption, it is also necessary to check whether the current message is successfully consumed in the target thread. If the consumption is successful, the next message consumption will be executed. If the consumption is not successful, the recovery mechanism will be triggered to compensate consumption.

Step A20: If the current message is not successfully consumed in the target thread, obtain the version value corresponding to the current message, and compensate the consumption according to the version value.

When it is judged that the current message is not successfully consumed in the target thread, it will start the recovery mechanism after the message consumption failed, because each directory node of the consumption queue has a version (description) value, and when the message in the message queue is generated When it changes, zookeeper's own mechanism will increment the version value of the consumption queue directory node. Therefore, you can first obtain the version value corresponding to the current message, and then calculate the number of messages with errors based on the version value to compensate for consumption until the consumption is successful.

In this embodiment, by detecting whether the consumption of the current message is successful, and when the consumption is unsuccessful, the consumption is compensated until the consumption is successful, thereby avoiding the influence of the consumption failure on the entire system logic.

Specifically, the step of obtaining the version value corresponding to the current message and compensating consumption according to the version value includes:

Step A21: Obtain the queue version value of the current message in the message queue and the local version value of the current message in the zookeeper, and calculate the difference between the local version value and the queue version value Whether the difference is greater than the preset value;

In this embodiment, the version value includes the local version value and the queue version value.

The message consumer will record the version value of the current message queue after each message consumption, and before each message consumption, it will also compare the local version value in zookeeper with the version value in the message queue. If two are found If the difference between the two is greater than the preset value, a compensation operation is performed, and if the difference between the two is less than or equal to the preset value, the current message consumption operation is continued. Among them, the preset value can be any value set by the user, such as 1.

Step A22: If the difference between the local version value and the queue version value is greater than a preset value, the current message is compensated for consumption according to the difference.

When it is judged that the difference between the local version value and the queue version value is greater than the preset value, the number of messages with errors will be calculated based on the difference, and the consumption will be compensated until the consumption is successful.

In this embodiment, by calculating whether the difference between the local version value and the queue version value is greater than a preset value, and when the difference is greater than the preset value, compensation consumption is performed, thereby increasing the success rate of message consumption.

The present application also provides a message consumption device. Referring to FIG. 3, the message consumption device includes:

Further, the determining module is also used for:

Receiving the feature information corresponding to the current message at the top-level interface corresponding to the current message, and identifying the feature information based on a preset feature recognition algorithm;

Determine the target thread corresponding to the current message according to the characteristic information.

Further, the determining module is also used for:

Obtain the thread pool corresponding to the target node, perform the remainder operation according to the capacity of the thread pool and the characteristic information, acquire the first thread with the remainder result mark in the thread pool, and detect the first thread Whether the thread is in an abnormal state;

If the first thread is in an abnormal state, enable the standby thread in the thread pool, and use the standby thread as the target thread corresponding to the current message;

If the first thread is in a normal state, use the first thread as the target thread corresponding to the current message.

Further, the message consumption device further includes:

If there is no new message queue monitored for the first time, detecting whether there is a new message node in each of the message queues;

If there is a new message node, the new message in the new message node is acquired, and the new message is delivered to the second thread corresponding to the new message node for consumption.

Further, the message consumption device further includes:

Acquire the total message queue in zookeeper, and periodically determine whether there is a consumed node whose consumption has been completed among each message node in the total message queue, where the total message queue includes a message queue;

If it exists, extract and save important information in the consumed message node, and clean up the consumed message node in the total message queue.

Further, the message consumption device further includes:

Detecting whether the current message is successfully consumed in the target thread;

If the current message is not successfully consumed in the target thread, the description version value corresponding to the current message is obtained, and the consumption is compensated according to the version value.

Further, the version value includes a local version value and a queue version value, and the message consumption device further includes:

Obtain the queue version value of the current message in the message queue and the local version value of the current message in the zookeeper, and calculate whether the difference between the local version value and the queue version value is Greater than the preset value;

If the difference between the local version value and the queue version value is greater than a preset value, the current message is compensated for consumption according to the difference.

For the method executed by the above-mentioned program modules, please refer to each embodiment of the message consumption method of this application, which will not be repeated here.

The present application also provides a computer storage medium, which may be a non-volatile readable storage medium.

The computer storage medium of the present application stores computer-readable instructions, and the computer-readable instructions implement the steps of the message consumption method described above when the computer-readable instructions are executed by the processor.

For the method implemented when the computer-readable instruction running on the processor is executed, please refer to each embodiment of the message consumption method of the present application, which will not be repeated here.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, It also includes other elements that are not explicitly listed, or elements inherent to the process, method, article, or system. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article or system that includes the element.

The serial numbers of the foregoing embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disk, optical disk), including several instructions to make a terminal device (can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the method described in each embodiment of the present application.

The above are only preferred embodiments of this application, and do not limit the scope of this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields , The same reason is included in the scope of patent protection of this application.

Claims

A message consumption method, wherein the message consumption method includes the following steps:

Obtain the target node in the service registration directory of the distributed coordination service zookeeper, and based on the target node monitor a preset number of message queues, determine whether there is a new message queue monitored for the first time in each of the message queues;

If there is a new message queue monitored for the first time, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm;

The current message is delivered to the target thread for consumption until the traversal of messages in the new message queue is completed.
The message consumption method according to claim 1, wherein the step of determining the target thread corresponding to the current message based on a preset feature recognition algorithm comprises:

Receiving the feature information corresponding to the current message at the top-level interface corresponding to the current message, and identifying the feature information based on a preset feature recognition algorithm;

Determine the target thread corresponding to the current message according to the characteristic information.
3. The message consumption method of claim 2, wherein the step of determining the target thread corresponding to the current message according to the characteristic information comprises:

Obtain the thread pool corresponding to the target node, perform the remainder operation according to the capacity of the thread pool and the characteristic information, acquire the first thread with the remainder result mark in the thread pool, and detect the first thread Whether the thread is in an abnormal state;

If the first thread is in an abnormal state, enable the standby thread in the thread pool, and use the standby thread as the target thread corresponding to the current message;

If the first thread is in a normal state, use the first thread as the target thread corresponding to the current message.
5. The message consumption method according to claim 1, wherein after the step of determining whether there is a new message queue monitored for the first time in each of the message queues, the method comprises:

If there is no new message queue monitored for the first time, detecting whether there is a new message node in each of the message queues;

If there is a new message node, the new message in the new message node is acquired, and the new message is delivered to the second thread corresponding to the new message node for consumption.
The message consumption method according to claim 4, wherein after the step of delivering the new message to the second thread corresponding to the new message for consumption, the method comprises:

Acquire the total message queue in zookeeper, and periodically determine whether there is a consumed message node whose consumption is completed in each message node of the total message queue, wherein the total message queue includes a message queue;

If it exists, extract and save important information in the consumed message node, and clean up the consumed message node in the total message queue.
The message consumption method according to claim 1, wherein after the step of passing the current message to the target thread for consumption, it comprises:

Detecting whether the current message is successfully consumed in the target thread;

If the current message is not successfully consumed in the target thread, the description version value corresponding to the current message is obtained, and the consumption is compensated according to the version value.
8. The message consumption method of claim 6, wherein the version value includes a local version value and a queue version value,

The step of obtaining the version value corresponding to the current message and compensating consumption according to the version value includes:

Obtain the queue version value of the current message in the message queue and the local version value of the current message in the zookeeper, and calculate whether the difference between the local version value and the queue version value is Greater than the preset value;

If the difference between the local version value and the queue version value is greater than a preset value, the current message is compensated for consumption according to the difference.
A message consumption device, wherein the message consumption device includes:

The obtaining module is used to obtain the target node in the zookeeper service registration directory, and determine whether there is a new message queue monitored for the first time in each of the message queues based on the target node monitoring a preset number of message queues;

The determining module is configured to, if there is a new message queue monitored for the first time, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm;

The delivery module is used to deliver the current message to the target thread for consumption until the message traversal in the new message queue is completed.
A message consumption device, wherein the message consumption device includes: a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor, and the computer-readable instructions are The processor implements the following steps when executing:

Obtain the target node in the service registration directory of the distributed coordination service zookeeper, and based on the target node monitor a preset number of message queues, determine whether there is a new message queue monitored for the first time in each of the message queues;

If there is a new message queue monitored for the first time, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm;

The current message is delivered to the target thread for consumption until the traversal of messages in the new message queue is completed.
9. The message consumption device according to claim 9, wherein the step of determining the target thread corresponding to the current message based on a preset feature recognition algorithm comprises:

Receiving the feature information corresponding to the current message at the top-level interface corresponding to the current message, and identifying the feature information based on a preset feature recognition algorithm;

Determine the target thread corresponding to the current message according to the characteristic information.
10. The message consuming device according to claim 10, wherein the step of determining the target thread corresponding to the current message according to the characteristic information comprises:

Obtain the thread pool corresponding to the target node, perform the remainder operation according to the capacity of the thread pool and the characteristic information, acquire the first thread with the remainder result mark in the thread pool, and detect the first thread Whether the thread is in an abnormal state;

If the first thread is in an abnormal state, enable the standby thread in the thread pool, and use the standby thread as the target thread corresponding to the current message;

If the first thread is in a normal state, use the first thread as the target thread corresponding to the current message.
9. The message consuming device according to claim 9, wherein after the step of determining whether there is a new message queue monitored for the first time in each of the message queues, the method comprises:

If there is no new message queue monitored for the first time, detecting whether there is a new message node in each of the message queues;

If there is a new message node, the new message in the new message node is acquired, and the new message is delivered to the second thread corresponding to the new message node for consumption.
The message consumption device according to claim 12, wherein, after the step of delivering the new message to the second thread corresponding to the new message for consumption, it comprises:

Acquire the total message queue in zookeeper, and periodically determine whether there is a consumed message node whose consumption is completed in each message node of the total message queue, wherein the total message queue includes a message queue;

If it exists, extract and save important information in the consumed message node, and clean up the consumed message node in the total message queue.
9. The message consumption device according to claim 9, wherein after the step of passing the current message to the target thread for consumption, it comprises:

Detecting whether the current message is successfully consumed in the target thread;

If the current message is not successfully consumed in the target thread, the description version value corresponding to the current message is obtained, and the consumption is compensated according to the version value.
A computer storage medium, wherein computer readable instructions are stored on the computer storage medium, and when the computer readable instructions are executed by a processor, the following steps are implemented:

Obtain the target node in the service registration directory of the distributed coordination service zookeeper, and based on the target node monitor a preset number of message queues, determine whether there is a new message queue monitored for the first time in each of the message queues;

If there is a new message queue monitored for the first time, sequentially traverse the messages in the new message queue, obtain the current message currently traversed, and determine the target thread corresponding to the current message based on a preset feature recognition algorithm;

The current message is delivered to the target thread for consumption until the traversal of messages in the new message queue is completed.
15. The computer storage medium of claim 15, wherein the step of determining the target thread corresponding to the current message based on a preset feature recognition algorithm comprises:

Receiving the feature information corresponding to the current message at the top-level interface corresponding to the current message, and identifying the feature information based on a preset feature recognition algorithm;

Determine the target thread corresponding to the current message according to the characteristic information.
15. The computer storage medium of claim 16, wherein the step of determining the target thread corresponding to the current message according to the characteristic information comprises:

Obtain the thread pool corresponding to the target node, perform the remainder operation according to the capacity of the thread pool and the characteristic information, acquire the first thread with the remainder result mark in the thread pool, and detect the first thread Whether the thread is in an abnormal state;

If the first thread is in an abnormal state, enable the standby thread in the thread pool, and use the standby thread as the target thread corresponding to the current message;

If the first thread is in a normal state, use the first thread as the target thread corresponding to the current message.
15. The computer storage medium according to claim 15, wherein after the step of determining whether there is a new message queue monitored for the first time in each of the message queues, the method comprises:

If there is no new message queue monitored for the first time, detecting whether there is a new message node in each of the message queues;

If there is a new message node, the new message in the new message node is acquired, and the new message is delivered to the second thread corresponding to the new message node for consumption.
18. The computer storage medium according to claim 18, wherein after the step of delivering the new message to the second thread corresponding to the new message for consumption, it comprises:

Acquire the total message queue in zookeeper, and periodically determine whether there is a consumed message node whose consumption has been completed in each message node of the total message queue, where the total message queue includes a message queue;

If it exists, extract and save important information in the consumed message node, and clean up the consumed message node in the total message queue.
15. The computer storage medium of claim 15, wherein after the step of passing the current message to the target thread for consumption, it comprises:

Detecting whether the current message is successfully consumed in the target thread;

If the current message is not successfully consumed in the target thread, the description version value corresponding to the current message is obtained, and the consumption is compensated according to the version value.