CN117093387B

CN117093387B - Message processing method, device, electronic equipment and storage medium

Info

Publication number: CN117093387B
Application number: CN202311360899.3A
Authority: CN
Inventors: 李达统; 陈云云; 李斌; 曾楚伟
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-10-20
Filing date: 2023-10-20
Publication date: 2024-02-13
Anticipated expiration: 2043-10-20
Also published as: CN117093387A

Abstract

The embodiment of the application discloses a message processing method, a message processing device, electronic equipment and a storage medium; the method comprises the steps of detecting the state of a consuming thread in a consuming process, determining the consuming thread in an idle state as a target consuming thread, wherein the consuming process comprises at least one consuming thread, and the consuming thread is used for consuming and processing information to be processed; extracting a message to be processed from a message queue; pushing the message to be processed to a target consumption thread to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after the processing is completed; extracting a new message to be processed from the message queue based on the message pulling request; and sending the new message to be processed to the target consumption thread. The scheme can improve the processing efficiency of the information.

Description

Message processing method, device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a message processing method, a device, an electronic device, and a storage medium.

Background

In recent years, with the advent of the mobile internet era, the widespread use of social software has made instant messaging and interaction between users an important component in daily life. To achieve efficient messaging and processing, message queues are often employed in social software as a key technology.

The message queue is a reliable and asynchronous message transmission mechanism, and social software can realize efficient, stable and extensible message transmission and processing by reasonably utilizing the message queue, so that the user experience is improved and the ever-increasing user demands are met. When using a message queue, message distribution is a very critical process, which refers to the process of distributing messages from the message queue to the corresponding recipients or handlers.

However, current message distribution mainly depends on Consumer (Consumer) pulling, and in order to ensure stability of a message Broker (Broker), a Consumer pulling time interval is often large (especially when a message queue is empty, pulling is performed at intervals of 1 second), which affects real-time performance and processing efficiency of message distribution.

Disclosure of Invention

The embodiment of the application provides a message processing method, a device, electronic equipment and a storage medium, which can improve the message processing efficiency.

The embodiment of the application provides a message processing method, which is used for detecting the state of a consumption thread in a consumption process, determining the consumption thread in an idle state as a target consumption thread, wherein the consumption process comprises at least one consumption thread, and the consumption thread is used for carrying out consumption processing on a message to be processed;

Extracting a message to be processed from a message queue;

pushing the message to be processed to the target consumption thread to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after the processing is completed;

extracting a new message to be processed from the message queue based on the message pulling request;

and sending the new message to be processed to the target consumption thread.

The embodiment of the application also provides a message processing method, which comprises the following steps:

detecting a message to be processed pushed by a message management process, wherein the message management process is used for managing the message to be processed in a message queue, the message to be processed is pushed when the message management process detects that a consumption thread in an idle state exists in a consumption process, and the consumption process comprises at least one consumption thread which is used for consuming and processing the message to be processed;

when the message to be processed is detected, carrying out consumption processing on the message to be processed based on a target consumption thread in the consumption process, wherein the target consumption thread is a consumption thread in an idle state in the consumption process;

When the target consumption thread finishes processing the message to be processed, sending a message pulling request to the message management process, wherein the message pulling request is used for pulling a new message to be processed from a message queue of the message management process;

and receiving the new message to be processed, and carrying out consumption processing on the new message to be processed through the target consumption thread.

The embodiment of the application also provides a message processing device, which comprises:

the state detection unit is used for detecting the state of a consuming thread in a consuming process, determining the consuming thread in an idle state as a target consuming thread, wherein the consuming process comprises at least one consuming thread, and the consuming thread is used for consuming and processing the message to be processed;

the first extraction unit is used for extracting the message to be processed from the message queue;

the pushing unit is used for pushing the message to be processed to the target consumption thread so as to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after the processing is completed;

a second extracting unit, configured to extract a new message to be processed from the message queue based on the message pulling request;

And the sending unit is used for sending the new message to be processed to the target consumption thread.

In some embodiments, the second extraction unit comprises:

a message detection subunit, configured to detect whether a new message to be processed exists in the message queue based on the message pulling request;

and the extraction subunit is used for extracting the new message to be processed from the message queue when the new message to be processed exists in the message queue, and adding the target consumption thread to a busy list, wherein the busy list is used for recording the consumption thread in the current working state.

In some embodiments, the second extraction unit is further configured to:

and when no new message to be processed exists in the message queue, adding the target consumption thread to an idle list, wherein the idle list is used for recording the consumption thread in the idle state currently.

In some embodiments, the apparatus further comprises:

a new message detection unit, configured to detect a newly listed message in the message queue;

the screening unit is used for screening a consumption thread from the idle list as a candidate consumption thread when the newly listed message is detected;

And the new message pushing unit is used for pushing the newly listed message to the candidate consumption thread so as to instruct the candidate consumption thread to perform consumption processing on the newly listed message.

In some embodiments, the apparatus further comprises:

the connection detection unit is used for detecting the connection request of each consumption thread in the consumption process;

the channel establishing unit is used for establishing a long-chain message channel between a message management process and the consumption thread when the connection request is detected;

and the execution unit is used for executing the step of detecting the state of each consumption thread in the consumption process when the establishment of the long-chain message channel is successful.

In some embodiments, the channel establishing unit is specifically configured to:

when a connection request is detected, detecting a registration event of the consumption thread, wherein the registration event is an event of the consumption thread applying for registering to the message management process;

and when the registration event is detected within a specified time period, establishing a long-chain message channel between the message management process and the consumption thread.

In some embodiments, the number of consumption processes is multiple, the pushing unit includes:

A number obtaining subunit, configured to obtain a number of target consumption threads included in each consumption process in the plurality of consumption processes;

a priority determining subunit, configured to determine, for each consumption process, a push priority of the consumption process based on a number of target consumption threads included in the consumption process;

and the pushing subunit is used for pushing the message to be processed to the target consumption thread based on the pushing priority.

In some embodiments, the priority determination subunit is specifically configured to:

obtaining the total number of at least one consumption thread contained in the consumption process;

calculating the ratio between the number of the target consumption threads and the total number of the target consumption threads to obtain a target ratio;

and determining the push priority of the consumption process according to the target ratio, wherein the target ratio is positively related to the push priority.

the message detection unit is used for detecting a message to be processed pushed by the message management process, wherein the message management process is used for managing the message to be processed in the message queue, the message to be processed is pushed when the message management process detects that a consumption thread in an idle state exists in a consumption process, the consumption process comprises at least one consumption thread, and the consumption thread is used for carrying out consumption processing on the message to be processed;

The first processing unit is used for carrying out consumption processing on the message to be processed based on a target consumption thread in the consumption process when the message to be processed is detected, wherein the target consumption thread is a consumption thread in an idle state in the consumption process;

a request unit, configured to send a message pulling request to the message management process when the target consumption thread finishes processing the message to be processed, where the message pulling request is used to pull a new message to be processed from a message queue of the message management process;

and the second processing unit is used for receiving the new message to be processed and carrying out consumption processing on the new message to be processed through the target consumption thread.

In some embodiments, the apparatus further comprises:

and the pulling request sending unit is used for sending a message pulling request to the message management process when the target consumption thread does not detect the message to be processed in a preset time period.

In some embodiments, the apparatus further comprises:

and the connection request unit is used for sending a connection request to the message management process, wherein the connection request is used for requesting to establish a long-chain message channel between the message management process and the consumption thread.

The embodiment of the application also provides electronic equipment, which comprises a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to perform steps in any of the message processing methods provided by the embodiments of the present application.

Embodiments of the present application also provide a computer readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform steps in any of the message processing methods provided by the embodiments of the present application.

Embodiments of the present application also provide a computer program product comprising computer programs/instructions which, when executed by a processor, implement steps in any of the message processing methods provided in the embodiments of the present application.

The embodiment of the application can detect the state of the consumption thread in the consumption process and determine the consumption thread in the idle state as the target consumption thread; then, extracting the message to be processed from the message queue; pushing the message to be processed to a target consumption thread to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after the processing is completed; then, extracting new message to be processed from the message queue based on the message pulling request; and finally, sending the new message to be processed to the target consumption thread. That is, after the target consuming thread in the idle state is detected, the message to be processed can be actively pushed to the target consuming thread, and after the target consuming thread processes the message to be processed, a message pulling request can be automatically returned to pull the next message to be processed in the message queue, so that the message distribution is completed in a push-pull combination mode, and the message processing efficiency is improved. In addition, in the push-pull combination mode, the consumption thread does not actively pull the message in most cases, but waits for the message management process to push, so that the problem of overload of the message management process caused by independent repeated pulling of the consumption thread is avoided, and the stability of message processing is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1a is a schematic view of a scenario of a message processing method provided in an embodiment of the present application;

fig. 1b is a schematic flow chart of a message processing method according to an embodiment of the present application;

FIG. 2 is a flow chart of another message processing method according to an embodiment of the present application;

FIG. 3a is a flow chart of yet another message processing method according to an embodiment of the present application;

FIG. 3b is a schematic diagram of an architecture of a message queue system provided by an embodiment of the present application;

fig. 4 is a schematic structural diagram of a message processing apparatus according to an embodiment of the present application;

fig. 5 is another schematic structural diagram of a message processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The embodiment of the application provides a message processing method, a message processing device, electronic equipment and a storage medium.

The message processing apparatus may be integrated in an electronic device, which may be a server or other device. The server may be a single server or a server cluster composed of a plurality of servers.

In some embodiments, the message processing apparatus may also be integrated in a plurality of electronic devices, for example, the message processing apparatus may be integrated in a plurality of servers, and the message processing method of the present application is implemented by the plurality of servers.

In some embodiments, the server may also be implemented in the form of a terminal.

For example, referring to fig. 1a, fig. 1a shows a schematic view of a scenario of a message processing method provided in an embodiment of the present application, and as shown in fig. 1a, the scenario may include a server 100 and at least one client 200, where the server 100 may be communicatively connected to the client 200. Alternatively, the client may be a mobile phone, tablet computer, smart bluetooth device, notebook computer, or personal computer (Personal Computer, PC) or the like.

The server 100 may perform the following steps:

and detecting the state of the consuming thread in the consuming process, and determining the consuming thread in the idle state as a target consuming thread.

And extracting the message to be processed from the message queue.

Pushing the message to be processed to a target consumption thread to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after the processing is completed.

And extracting a new message to be processed from the message queue based on the message pulling request.

And sending the new message to be processed to the target consumption thread.

The client 200 may install a messaging application, through which the client 200 may send messages to the server 100, and the server 100 may receive the messages sent by the client and process the received messages. Wherein the server may be a message queue server.

Wherein the message queue server (Message Queue Server) is a middleware system for enabling asynchronous communication between distributed applications. It provides a reliable messaging mechanism that allows a sender to send a message to a message queue and then asynchronously acquire and process the message from the queue by a receiver.

The message queue server is typically composed of several core components:

producer (Producer): is responsible for sending messages to the message queue server. The producer may be any application or service that requires a message to be sent (e.g., social software with communication capabilities, office software, etc.).

Message Queue (Message Queue): is a persistent storage area for holding messages to be processed. Message queue servers typically support multiple queues and provide flexible message management mechanisms.

Consumer (Consumer): is responsible for retrieving and processing messages from the message queue. The consumer may be a plurality of independent applications or services that can fetch messages from the queue for processing according to their own needs.

Message Broker (Broker): is the core component of the message queue server, responsible for receiving messages from the producer and sending them to the corresponding queues. At the same time, it is also responsible for receiving requests from consumers and transmitting messages in the queue to consumers.

The workflow of the message queue server is as follows:

the producer sends the message to the message queue server, typically by specifying the name of the target queue and the message content.

After receiving the message, the message queue server stores the message in a corresponding queue and ensures the persistence of the message so as to prevent data loss.

The consumer requests a message from the message queue server. The consumer can select different acquisition modes according to own requirements, such as requesting all messages, acquiring the messages according to priority, and the like.

The message queue server transmits the message to the consumer. Once the message is successfully delivered to the consumer, the message queue server will delete the message, ensuring that each message is consumed only once.

Advantages of message queue servers include decoupling, asynchronous processing, enhanced scalability, high availability, and the like. It can help build a reliable and efficient distributed system and provide good performance and reliability in the face of high concurrency and large data volume scenarios. Common message queue servers include: rabbitMQ, apache Kafka, activeMQ, etc.

In this embodiment, an exemplary message queue system is provided, where the message queue system may correspond to the middleware system, and the message queue system may include a message management process (corresponding to the Broker and the Consumer process (corresponding to the Consumer) for managing a message to be processed in the message queue, where the Consumer process includes at least one Consumer thread for performing Consumer processing on the message to be processed.

In this embodiment, a message processing method related to a message queue technology is provided, where the message processing method may be applied to the message queue system, and specifically may be applied to the message management process, as shown in fig. 1b, and a specific flow of the message processing method may be as follows:

101. And detecting the state of the consuming thread in the consuming process, and determining the consuming thread in the idle state as a target consuming thread.

The idle state refers to a state that a thread is idle, has no task execution or has no workload.

As one example, a particular implementation of state detection for a consuming thread in a consuming process may include:

a status flag bit is set for each consuming thread indicating the state of the thread, e.g. "idle" or "busy". When the thread is idle, the flag position is "idle" and the status of the flag bit is checked periodically. If the thread with the flag bit of idle exists, the idle consumption thread can be judged.

As another example, the consuming thread periodically sends a heartbeat signal to indicate its active state. By continuously detecting the heartbeat signal of a consuming thread, if a thread does not send a heartbeat signal for a certain period of time, the thread can be considered to be in an idle state.

In some embodiments, before the state detection is performed on each consuming thread in the consuming process, whether an idle consuming thread exists in the consuming process may be further determined online, and specific embodiments may include:

A counter is maintained that records the number of consuming threads currently running. Each time a thread starts processing a message, the counter is incremented by one; when the thread has finished processing the message, the counter is decremented by one. By continuously detecting the value of the counter, it can be determined whether there is an idle consuming thread.

It will be appreciated that in practical applications, the suitable method in the above example may be selected to detect whether the consuming thread is idle according to a specific application scenario and system architecture. In general, combining multiple methods can improve the accuracy and stability of the assay.

In some embodiments, before performing state detection on the consuming thread in the consuming process and determining the consuming thread in the idle state as the target consuming thread in step 101, the method may further include:

s1, detecting a connection request of each consumption thread in the consumption process.

The connection request is a request sent by a consuming thread to a message management process for establishing inter-process communication. Illustratively, in this embodiment, the inter-process communication may employ a Socket (Socket) inter-process communication mechanism. Alternatively, the connection request may carry a thread identifier of the consuming thread that issued the connection request.

In some embodiments, the message management process may detect whether a connection request is received at a specified period, e.g., may detect whether a connection request is received every N seconds, where N is a positive integer.

S2, when the connection request is detected, a long-chain message channel between the message management process and the consumption thread is established.

Wherein long-chain message channel is a network communication mechanism for implementing real-time messaging. By establishing a persistent connection, the sending end can push messages to the receiving end without the receiving end frequently sending requests.

In some embodiments, in step S2, when a connection request is detected, a specific embodiment of establishing a long-chain message channel between the message management process and the consuming thread may include:

when the connection request is detected, detecting a registration event of the consuming thread, wherein the registration event is an event of the consuming thread applying for registering to the message management process.

Wherein, the consumption thread registers to the message management process means that the consumption thread registers its information to the message management process, and in the context of inter-process communication, the purpose of the registration of information is to establish a connection between each other and enable each other to identify and communicate with each other. By registering information, one process can inform other processes of its own presence, services or functions provided, and the manner in which it communicates with other processes. In this embodiment, the consuming thread may allow the message management process to identify and locate the consuming thread after registering its information with the message management process. Illustratively, a consuming thread may register a process with a unique identifier or name so that a message management process can accurately find and communicate with the consuming thread.

When a registration event is detected within a specified time period, a long-chain message channel between the message management process and the consuming thread is established.

The designated time period may be from when the message management process receives the connection request to when the designated time period is after the connection request is received, for example, the designated time period is 1 minute, the time when the connection request is received is 12:00, and then the designated time period is 12:00-12:01.

In some embodiments, the message management process may continuously detect whether the registration information sent by the consuming thread is received within a specified duration after receiving the connection request, and if so, determine that the registration event is detected within a specified time period, and establish a long-chain message channel between the message management process and the consuming thread.

Alternatively, the message management process may close the connection with the consuming thread when the message management process does not detect a registration event of the consuming thread for a specified period of time.

And S3, when the long-chain message channel is successfully established, executing the step of detecting the state of each consumption thread in the consumption process.

In this embodiment, after detecting the connection request and registration event of the consuming thread, a long-chain message channel is additionally established with the consuming thread, so that the state of the consuming thread can be stably and effectively detected after the long-chain message channel is established.

102. And extracting the message to be processed from the message queue.

In some embodiments, if the message management process detects that there are multiple target consuming threads in the consuming process, the number of the multiple target consuming threads may be obtained, and then a corresponding number of messages to be processed may be extracted from the message queue based on the number of the multiple target consuming threads.

103. Pushing the message to be processed to a target consumption thread to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after the processing is completed.

The message pulling request is used for pulling a new message to be processed from a message queue managed by the message management process.

The consumption processing refers to actual processing operation of the received message by the consumption process. In a messaging system, there are a number of ways in which the message may be consumed, such as:

and (3) direct treatment: the simplest way is to process the message immediately after it is received. This applies to scenarios where the message processing logic is lightweight and time consuming. For example, after receiving a notification message, a corresponding prompt or record is locally performed.

Asynchronous processing: for longer time consuming message processing logic, messages may be placed in a queue or task pool for processing asynchronously by a background thread or specialized processor. This may improve concurrency and throughput of the system while avoiding blocking the main thread. Asynchronous processing is often used for processing intensive or time consuming computations, I/O operations, etc.

Batch processing: when the number of messages is large and processing logic can be executed in batches, the received messages can be processed in batches. By aggregating multiple messages for centralized processing, the overhead of resources and the load of the system can be reduced. For example, multiple data update requests are processed at once, reducing the number of interactions with the database.

Distributed processing: in a distributed system, messages may be distributed to different nodes or service instances for processing to achieve horizontal expansion and load balancing. Different nodes can perform consumption processing according to the routing rule of the message, so that distributed computation or parallel processing is realized.

Error handling and retry: during message processing, errors or anomalies may occur. To ensure reliability of the message and stability of the system, error handling and retry mechanisms are required. When an error occurs, the message may be marked as failed and retried according to a set policy until the message is successfully processed or a maximum number of retries is reached.

The target consumption thread can obtain a processing result after the target consumption thread finishes processing the message to be processed, and the target consumption thread can return the processing result and the message pulling request to the message management process.

In some embodiments, since different consuming processes are used to process different types of messages, for example, the plurality of consuming processes may include consuming process Q1 and consuming process Q1, where consuming process Q1 is used to process one type of message and consuming process Q2 is used to process another type of message. Therefore, before pushing the pending message to the target consuming thread, the method may further comprise:

and acquiring the type of the message to be processed, and determining whether the type of the message to be processed is matched with the consumption process corresponding to the target consumption thread. And pushing the message to be processed to the target consumption thread when the type of the message to be processed is determined to be matched with the consumption process corresponding to the target consumption thread.

In other embodiments, the message queue and the consuming process that are matched with each other may be associated in advance, and after the target consuming thread is detected, the message to be processed may be extracted from the message queue associated with the consuming process where the target consuming thread is located, and the message to be processed may be pushed to the target consuming thread.

It will be appreciated that the different consumption processes will also differ in the manner in which the message to be processed is consumed.

In some embodiments, the number of consumption processes may be plural, and in step 103, pushing the message to be processed to the target consumption thread may include:

a1, acquiring the number of target consumption threads included in each consumption process in the plurality of consumption processes.

In some embodiments, after all the target consuming threads are detected, the number of target consuming threads included in each consuming process may be counted for the consuming process.

It will be appreciated that in this embodiment, the plurality of consuming processes described above are all matched to the type of message to be processed.

A2, determining the push priority of the consumption process based on the number of target consumption threads included in the consumption process aiming at each consumption process.

The push priority may be a priority recommendation level of the consumption process, and when the message management process pushes the message to be processed to the target consumption thread in the consumption process, the consumption process with a higher recommendation level may be pushed the message to be processed first.

In some implementations, the push priority of a consuming process may be positively correlated to the number of target consuming threads that the consuming process includes.

Illustratively, the plurality of consuming processes include a consuming process Q3, a consuming process Q4, and a consuming process Q5, where the number of target consuming threads in the consuming process Q3 is k3, the number of target consuming threads in the consuming process Q4 is k4, and the number of target consuming threads in the consuming process Q5 is k5, where k3, k4, and k5 are positive integers. As an embodiment, the number of target consuming threads included in the consuming process may be multiplied by a preset coefficient to obtain the push priority of the consuming process, for example, when the preset coefficient is 1, the push priority of the consuming process Q3 may be directly determined as k3. The preset coefficient may be set in a user-defined manner according to actual requirements, and is not limited herein.

Considering that the more target consumption threads in the consumption process are, the smaller the current load of the consumption process is indicated, in the embodiment, the push priority of the consumption process is determined based on the number of the target consumption threads included in the consumption process, so that the message to be processed can be preferentially pushed to the consumption process with smaller current load, the load balance of a plurality of consumption processes is guaranteed, and the processing efficiency and the stability of the message to be processed are further improved.

In some embodiments, in step A2, determining the push priority of the consumption process based on the number of target consumption threads included in the consumption process may include:

a21, obtaining the total number of at least one consumption thread contained in the consumption process.

The total number of consuming threads contained in the consuming process can be registered in the message management process through a consuming thread application registration event in the consuming process.

Along with the above examples, for example, the obtained total number of consuming processes Q3 corresponds to m3, the total number of consuming processes Q4 corresponds to m4, and the total number of consuming processes Q5 corresponds to m5. Wherein m3, m4 and m5 are positive integers.

A22, calculating the ratio between the number of the target consumption threads and the total number of the target consumption threads to obtain a target ratio.

Along the above example, the target ratio corresponding to the consuming process Q3 is k3/m3, the target ratio corresponding to the consuming process Q4 is k4/m4, and the target ratio corresponding to the consuming process Q5 is k5/m5.

A23, determining the push priority of the consumption process according to the target ratio, wherein the target ratio is positively related to the push priority.

For example, the target ratio corresponding to the consumption process may be multiplied by a specified coefficient to obtain the push priority of the consumption process, for example, when the preset coefficient is 1, the push priority of the consumption process Q3 may be directly determined as k3/m3. The specified coefficient can be set in a self-defined mode according to actual requirements, and optionally, the set specified coefficient can meet the requirement that the product of the specified coefficient and all target ratios is a positive integer.

Considering that the two consumption processes with the same number of the target consumption threads are smaller in load, the smaller the total number of the threads (i.e. the larger the target ratio), in the embodiment, the target ratio is obtained by calculating the ratio between the number of the target consumption threads and the total number of the target consumption threads, and the push priority of the consumption processes is determined according to the target ratio, and the target ratio is positively correlated with the push priority, so that the message to be processed can be preferentially pushed to the consumption process with smaller current load, the load balance of a plurality of consumption processes is guaranteed, and the processing efficiency and the stability of the message to be processed are improved.

A3, pushing the message to be processed to the target consumption thread based on the pushing priority.

In some implementations, the pending message may be pushed to the target consuming thread in the highest-pushing priority consuming process.

In other embodiments, a consuming process with a push priority greater than or equal to a priority threshold may be determined as an alternative consuming process, and then the message to be pushed is randomly pushed to a target consuming thread in the alternative consuming process.

104. And extracting a new message to be processed from the message queue based on the message pulling request.

In some embodiments, at least one pending message may be extracted from the remaining pending messages in the message queue as a new pending message.

For example, the message queue originally includes a to-be-processed message a, a to-be-processed message B, and a to-be-processed message C, where the to-be-processed message a was pushed when the last message was pushed, and after receiving the message pulling request, at least one of the to-be-processed message B and the to-be-processed message C may be extracted as a new to-be-processed message.

In step 104, the specific embodiment of extracting the new pending message from the message queue based on the message pull request may include:

b1, detecting whether a new message to be processed exists in the message queue based on the message pulling request.

In some embodiments, the message management process may detect the number of remaining pending messages in the message queue after receiving the message pull request, and if the number of remaining pending messages is 0, may determine that no new pending messages exist in the message queue.

And B2, when a new message to be processed exists in the message queue, extracting the new message to be processed from the message queue, and adding the target consumption thread to a busy list, wherein the busy list is used for recording the consumption thread in the current working state.

When a new message to be processed exists in the message queue, the target consumption thread is indicated to be pushed with the new message to be processed, and the new message to be processed is processed, so that the target consumption thread is not idle temporarily, and therefore the target consumption thread can be added to a busy list to avoid that a message management process repeatedly pushes the message to be processed to the target message thread.

In some embodiments, the message processing method may further include:

When no new message to be processed exists in the message queue, the target consumption thread is indicated not to be pushed with the new message to be processed, so that the message to be processed is not needed to be processed next, and the message to be processed is in an idle state, and therefore the target consumption thread can be added into the idle list, so that the detection of the idle state of the target consumption thread is not needed to be carried out later.

In some implementations, after adding the target consuming thread to the free list, the method can further include:

A newly enqueued message in the message queue is detected.

After receiving a new message sent by a client, the message queue system can add the new message to a message queue managed by the message queue system through a message management process to obtain a newly-listed message, and can update the number of the messages contained in the message queue.

When a newly listed message is detected, a consuming thread is screened from the free list to serve as a candidate consuming thread.

Along with the above example, when it is detected that the number of messages contained in the message queue is not 0, one consuming thread may be randomly selected from the free list as a candidate consuming thread.

Pushing the newly listed message to the candidate consumption thread to instruct the candidate consumption thread to consume the newly listed message.

The embodiment of pushing the newly listed message to the candidate consumption thread to instruct the candidate consumption thread to perform the consumption processing on the newly listed message may refer to the embodiment of pushing the message to be processed to the target consumption thread in step 103, which is not described herein.

105. And sending the new message to be processed to the target consumption thread.

In some embodiments, after sending the new pending message to the target consuming thread, the target consuming thread may consume the new pending message and return a new message pull request to the message management process after processing is complete. The message management process may then return to executing the operation of extracting the new pending message from the message queue based on the message pull request in step 104 based on the new message pull request until no new pending message exists in the message queue.

The message processing scheme provided by the embodiment of the application can be applied to various message queue scenes. Such as a communication scenario, an order processing scenario, etc.

As can be seen from the above, the embodiments of the present application may perform state detection on a consuming thread in a consuming process, and determine a consuming thread in an idle state as a target consuming thread; then, extracting the message to be processed from the message queue; pushing the message to be processed to a target consumption thread to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after the processing is completed; then, extracting new message to be processed from the message queue based on the message pulling request; and finally, sending the new message to be processed to the target consumption thread. That is, after the target consuming thread in the idle state is detected, the message to be processed can be actively pushed to the target consuming thread, and after the target consuming thread processes the message to be processed, a message pulling request can be automatically returned to pull the next message to be processed in the message queue, so that the message distribution is completed in a push-pull combination mode, and the message processing efficiency is improved. In addition, in the push-pull combination mode, the consumption thread does not actively pull the message in most cases, but waits for the message management process to push, so that the problem of overload of the message management process caused by independent and repeated pulling and extinguishing of the consumption thread is avoided, and the stability of message processing is improved.

In this embodiment, a message processing method related to a message queue technology is provided, and the method may be applied to, as shown in fig. 2, a specific flow of the message processing method may be as follows:

201. and detecting a message to be processed pushed by the message management process, wherein the message to be processed is pushed when the message management process detects that the consumption thread in an idle state exists in the consumption process.

In some embodiments, before detecting the pending message pushed by the message management process, the message processing method may further include:

and sending a connection request to the message management process, wherein the connection request is used for requesting to establish a long-chain message channel between the message management process and the consumption process.

Illustratively, after the consuming process is started, a connection request may be sent to the message management process by a consuming thread in the consuming process, and information of the consuming thread is registered in the message management process. Wherein registering information of the consuming thread into the message management process may trigger a registration event.

After the message management process detects the connection request and registration event, a long-chain message channel between the message management process and the consumption process can be established based on the connection request.

In some embodiments, after detecting the pending message pushed by the message management process, the message processing method may further include:

and when the target consumption thread does not detect the message to be processed in the preset time period, sending a message pulling request to the message management process.

The preset time period may be a time from when the long-chain message channel between the message management process and the consumption process is successfully established to after a preset time period. The preset time length can be set according to actual requirements, and is not limited.

For example, the consumption process may start to detect whether a message to be processed is received when the long-chain message channel is established successfully, and if the message to be processed is not detected after the preset duration, may send a message pull request to the message management process. As an embodiment, after sending the message pull request, if the consumption process still does not detect the pending message after waiting for a certain period of time, the consumption process may stop sending the message pull request to the message management process.

202. And when the message to be processed is detected, carrying out consumption processing on the message to be processed based on a target consumption thread in the consumption process, wherein the target consumption thread is a consumption thread in an idle state in the consumption process.

The target consuming thread is detected by the message management process, and the target consuming thread pushes the message to be processed to the target consuming thread, so that the consuming thread in the consuming process receives the message to be processed, and the consuming thread can be determined to be the target consuming thread. Then, the message to be processed can be directly processed through the target consumption thread.

203. When the target consumption thread finishes processing the message to be processed, a message pulling request is sent to the message management process, and the message pulling request is used for pulling a new message to be processed from a message queue of the message management process.

204. And receiving the new message to be processed, and carrying out consumption processing on the new message to be processed through the target consumption thread.

For more specific embodiments of step 201 to step 204, reference is made to step 101 to step 105 in the above embodiments, and thus will not be described herein.

The method described in the above embodiments will be described in further detail below.

In this embodiment, a message queue system will be taken as an example, and a method of the embodiment of the present application will be described in detail.

As shown in fig. 3a, a specific flow of a message processing method is as follows:

301. The message management process detects the state of the consuming thread in the consuming process, and determines the consuming thread in the idle state as a target consuming thread.

302. The message management process extracts the pending message from the message queue.

303. The message management process pushes the pending message to the target consuming thread.

304. The consumption process detects the message to be processed pushed by the message management process.

305. And when the consumption process detects the message to be processed, carrying out consumption processing on the message to be processed based on a target consumption thread in the consumption process.

306. And the consumption process sends a message pulling request to the message management process when the target consumption thread finishes processing the message to be processed.

307. The message management process extracts new pending messages from the message queue based on the message pull request.

308. The message management process sends the new pending message to the target consuming thread.

309. And the consumption process receives the new message to be processed and consumes the new message to be processed through the target consumption thread.

For example, in practical application, the message processing method may be implemented by a framework of a message queue system as shown in fig. 3b, where the framework may include a Broker and a Consumer, where the Broker may be equivalent to a message manager, where the message manager may include a message management process, and the Consumer may be regarded as a Consumer, where the Consumer may include at least one consuming process (such as Q1 and Q2 in fig. 3 b), and where each consuming process may include at least one consuming thread (such as worker1, worker2 …, and worker n in fig. 3 b). Among other things, a coroutine pool and at least one scheduler (Dispatcher) may be included in the Broker.

The Dispatcher is used for detecting connection of a consumption thread (worker) and perceiving enqueuing of new messages, and when the worker is found to be idle, the newly enqueued messages can be actively pushed to a consumer for consumption.

The coroutine pool is responsible for specifically executing the Dispatcher push message task and processing the processing result message (ack message) and the message pull request of the consuming thread. Consumer is used for actively registering with a browser at the starting time and keeping a push-pull channel long connection, and processing after receiving a message.

Wherein, a Broker may create 1 or more dispatchers and use epoll to detect Socket events (such as connection events and read-write events).

Among other things, epoll is an efficient event-driven mechanism that can be used to achieve high performance concurrent network programming. It is a technology capable of detecting a plurality of file descriptors (sockets, files, etc.) simultaneously and capable of efficiently handling a large number of concurrent connections.

Wherein, the detecting the connection event of the consuming thread by the Dispatcher may comprise the following steps:

1. dispatcher detects designated ports and unix domain sockets (co-machine deployment). Wherein detecting the designated port and unix domain socket is a common operation in network programming for accepting connection requests from other hosts or processes.

2. After the Dispatcher receives the connection request, the connection which is not registered in the timeout is closed after continuing to wait for a registration event.

3. After receiving the registration event, the long connection fd is added into the connection list and registered into the epoll, and further detects the read-write event.

In the event of detecting a read-write event, when a new message enqueues in the queue, all dispatchers are notified and the notification is awakened. The Dispatcher then throws a push message task to the coroutine pool. The message pushing task can try to acquire the idle worker, and if the idle worker is not available, the task is ended; otherwise, the message is obtained from the message queue and pushed to fd corresponding to the idle worker. The fd senses a write event, and sends a message to the worker through a socket, and when the worker initiates an ack or message pulling request, the Dispatcher detects and processes a read event on the fd.

Where fd is an abbreviation for file descriptor in epoll, representing an identifier of an open file or socket.

The coroutine pool mainly processes three types of events:

1. new enqueue notify event: and detecting whether the connection associated with the queue is idle, if so, distributing the message, otherwise, ending the task.

2. ack event: after the Consumer processes the message, a message processing result is returned to the brooker through an ack request; and after the Broker receives the ack message, returning to the consumer if the queue has the message to be distributed, otherwise, ending the task, and adding the connection into the idle list.

3. Consumer pull cancel event: if the queue has the information to be distributed, returning to the consumer, and adding the connection into a busy list; if the queue has no message to be distributed, the task is ended, and the connection is added into the idle list.

The connection management may be performed in the Consumer, and specifically, the connection management may include the following steps:

1. after the worker is started, connection is actively initiated to the browser and registration is carried out.

2. After the connection is successful, blocking the read socket and waiting for the brooker push message.

3. And in the process of pushing the message by the reader, if the reading time-out occurs, initiating a message pulling request. Wherein a read timeout generally means that a jam occurs while reading data and the waiting time exceeds a preset time.

Exemplary, tests show that in the common polling pull mode, the consumption delay of fewer scenes of the queue message reaches 200-300 milliseconds, and in the message processing method provided by the implementation, the consumption delay is reduced to 3-4 milliseconds in the push-pull combined mode; in a full load scenario, the poll pull mode consumes 300-400 milliseconds while the push-pull combined mode reduces to 30 milliseconds. Therefore, the message processing method provided by the embodiment effectively optimizes the real-time performance of message processing and improves the service stability.

In order to better implement the method, the embodiment of the application also provides a message processing device, which can be integrated in an electronic device, and the electronic device can be a server or other devices. The server may be a single server or a server cluster composed of a plurality of servers.

For example, in the present embodiment, a method of the embodiment of the present application will be described in detail by taking a specific integration of a message processing apparatus in a server as an example.

For example, as shown in fig. 4, the message processing apparatus may be applied to the above-described message management process, and the message processing apparatus may include a state detection unit 401, a first extraction unit 402, a push unit 403, a second extraction unit 404, and a transmission unit 405, as follows:

a state detection unit 401, configured to perform state detection on a consuming thread in the consuming process, and determine a consuming thread in an idle state as a target consuming thread;

a first extracting unit 402, configured to extract a message to be processed from the message queue;

a pushing unit 403, configured to push the message to be processed to the target consuming thread, so as to instruct the target consuming thread to perform consuming processing on the message to be processed, and return a message pulling request after processing is completed;

A second extracting unit 404, configured to extract a new message to be processed from the message queue based on the message pulling request;

and a sending unit 405 for sending the new message to be processed to the target consuming thread.

In some embodiments, the second extraction unit 404 includes:

In some embodiments, the second extraction unit 404 is further configured to:

In some embodiments, the apparatus further comprises:

a new message detection unit 501, configured to detect a newly listed message in the message queue;

a new message pushing unit 403, configured to push the newly listed message to the candidate consumption thread, so as to instruct the candidate consumption thread to perform consumption processing on the newly listed message.

In some embodiments, the apparatus further comprises:

a channel establishing unit, configured to establish a long-chain message channel between the message management process and the consuming thread when the connection request is detected;

and the execution unit is used for executing the step of detecting the state of each consuming thread in the consuming process when the establishment of the long-chain message channel is successful.

when a connection request is detected, detecting a registration event of the consuming thread, wherein the registration event is an event of the consuming thread applying for registration to the message management process;

In some embodiments, the pushing unit 403 includes:

As shown in fig. 5, an embodiment of the present application further provides a message processing apparatus, where the apparatus is applied to a message process in a message queue system, where the message queue system includes the consuming process and a message management process, where the message management process is used to manage a message to be processed in a message queue, where the consuming process includes at least one consuming thread, where the consuming thread is used to perform consuming processing on the message to be processed, and the apparatus includes:

A message detection unit 501, configured to detect a message to be processed pushed by the message management process, where the message to be processed is pushed when the message management process detects that a consuming thread in an idle state exists in the consuming process;

the first processing unit 502 is configured to perform, when the message to be processed is detected, consumption processing on the message to be processed based on a target consumption thread in the consumption process, where the target consumption thread is a consumption thread in an idle state in the consumption process;

a request unit 503, configured to send a message pulling request to the message management process when the target consuming thread finishes processing the message to be processed, where the message pulling request is used to pull a new message to be processed from a message queue of the message management process;

and the second processing unit 504 is configured to receive the new message to be processed, and perform consumption processing on the new message to be processed through the target consumption thread.

In some embodiments, the apparatus further comprises:

and a pull request sending unit 405, configured to send a message pull request to the message management process when the target consumption thread does not detect the pending message within a preset period of time.

In some embodiments, the apparatus further comprises:

a connection request unit 503, configured to send a connection request to the message management process, where the connection request is used to request to establish a long-chain message channel between the message management process and the consuming thread.

In the implementation, each unit may be implemented as an independent entity, or may be implemented as the same entity or several entities in any combination, and the implementation of each unit may be referred to the foregoing method embodiment, which is not described herein again.

In the present embodiment, a detailed description will be given taking an example in which the electronic device of the present embodiment is a server, for example, as shown in fig. 6, which shows a schematic structural diagram of the electronic device according to the embodiment of the present application, specifically:

the electronic device may include one or more processing cores 'processors 601, one or more computer-readable storage media's memory 602, a power supply 603, an input module 604, and a communication module 605, among other components. It will be appreciated by those skilled in the art that the electronic device structure shown in fig. 6 is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components. Wherein:

The processor 601 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 602, and calling data stored in the memory 602, thereby performing overall detection of the electronic device. In some embodiments, processor 601 may include one or more processing cores; in some embodiments, processor 601 may integrate an application processor that primarily handles operating systems, user interfaces, applications, and the like, with a modem processor that primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 601.

The memory 602 may be used to store software programs and modules, and the processor 601 may execute various functional applications and data processing by executing the software programs and modules stored in the memory 602. The memory 602 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide access to the memory 602 by the processor 601.

The electronic device also includes a power supply 603 that powers the various components, and in some embodiments, the power supply 603 may be logically connected to the processor 601 through a power management system, so as to perform functions of managing charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more of any components, such as a direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

The electronic device may also include an input module 604, which input module 604 may be used to receive entered numeric or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

The electronic device may also include a communication module 605, and in some embodiments the communication module 605 may include a wireless module, through which the electronic device may wirelessly transmit over short distances, thereby providing wireless broadband internet access to the user. For example, the communication module 605 may be used to assist a user in e-mail, browsing web pages, accessing streaming media, and the like.

Although not shown, the electronic device may further include a display unit or the like, which is not described herein. In particular, in this embodiment, the processor 601 in the electronic device loads executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 executes the application programs stored in the memory 602, thereby implementing various functions.

The specific implementation of each operation above may be referred to the previous embodiments, and will not be described herein.

Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, embodiments of the present application provide a computer readable storage medium having stored therein a plurality of instructions capable of being loaded by a processor to perform steps in any of the message processing methods provided by the embodiments of the present application.

Wherein the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the above-described embodiment.

The steps in any message processing method provided in the embodiments of the present application may be executed by the instructions stored in the storage medium, so that the beneficial effects that any message processing method provided in the embodiments of the present application may be achieved are detailed in the previous embodiments, and are not repeated herein.

The foregoing has described in detail the methods, apparatuses, electronic devices and computer readable storage medium for processing messages provided in the embodiments of the present application, and specific examples have been applied to illustrate the principles and embodiments of the present application, where the foregoing examples are provided to assist in understanding the methods and core ideas of the present application; meanwhile, those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, and the present description should not be construed as limiting the present application in view of the above.

Claims

1. A method of message processing, the method comprising:

detecting states of consumption threads in the consumption process, determining the consumption threads in an idle state as target consumption threads, wherein the number of the consumption processes is multiple, the consumption process comprises at least one consumption thread, and the consumption thread is used for carrying out consumption processing on information to be processed;

extracting a message to be processed from a message queue;

acquiring the number of target consumption threads included in each consumption process in a plurality of consumption processes;

for each consumption process, acquiring the total number of at least one consumption thread contained in the consumption process;

determining a push priority of the consumption process according to the target ratio, wherein the target ratio is positively related to the push priority;

pushing the message to be processed to the target consumption thread based on the pushing priority so as to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after processing is completed;

and sending the new message to be processed to the target consumption thread.

2. The message processing method according to claim 1, wherein extracting a new message to be processed from the message queue based on the message pull request comprises:

detecting whether a new message to be processed exists in the message queue or not based on the message pulling request;

when a new message to be processed exists in the message queue, extracting the new message to be processed from the message queue, and adding the target consumption thread to a busy list, wherein the busy list is used for recording the consumption thread in a working state currently.

3. The message processing method according to claim 2, characterized in that the method further comprises:

4. A message processing method according to claim 3, further comprising, after said adding said target consuming thread to a free list:

Detecting newly listed messages in the message queue;

when the newly listed message is detected, a consumption thread is screened out from the idle list to serve as a candidate consumption thread;

5. The message processing method according to claim 1, further comprising, prior to said detecting the state of each consuming thread in said consuming process:

detecting a connection request of each consumption thread in the consumption process;

when the connection request is detected, a long-chain message channel between a message management process and the consumption thread is established;

and when the long-chain message channel is successfully established, executing the step of detecting the state of each consuming thread in the consuming process.

6. The message processing method of claim 5, wherein establishing a long-chain message path between a message management process and the consuming thread when a connection request is detected comprises:

7. A method of message processing, the method comprising:

detecting a message to be processed pushed by a message management process, wherein the message management process is used for managing the message to be processed in a message queue, the message to be processed is pushed by the message management process based on the push priority of each consumption process in a plurality of consumption processes, the push priority of the consumption process is positively correlated with a target ratio of the consumption process, the target ratio of the consumption process is a ratio between the number of target consumption threads in the consumption process and the total number of consumption threads in the consumption process, and the target consumption threads are consumption threads in an idle state in the consumption process and are used for carrying out consumption processing on the message to be processed;

8. The message processing method according to claim 7, further comprising, after the detecting the message to be processed pushed by the message management process:

and when the target consumption thread does not detect the message to be processed in a preset time period, sending a message pulling request to the message management process.

9. The message processing method according to claim 7 or 8, further comprising, before the detecting the message to be processed pushed by the message management process:

and sending a connection request to the message management process, wherein the connection request is used for requesting to establish a long-chain message channel between the message management process and the consumption thread.

10. A message processing apparatus, the apparatus comprising:

The state detection unit is used for detecting the states of the consumption threads in the consumption processes, determining the consumption threads in the idle state as target consumption threads, wherein the number of the consumption processes is multiple, the consumption processes comprise at least one consumption thread, and the consumption threads are used for carrying out consumption processing on the information to be processed;

the pushing unit is used for acquiring the number of target consumption threads included in each consumption process in the plurality of consumption processes; for each consumption process, acquiring the total number of at least one consumption thread contained in the consumption process; calculating the ratio between the number of the target consumption threads and the total number of the target consumption threads to obtain a target ratio; determining a push priority of the consumption process according to the target ratio, wherein the target ratio is positively related to the push priority; pushing the message to be processed to the target consumption thread based on the pushing priority so as to instruct the target consumption thread to consume the message to be processed, and returning a message pulling request after processing is completed;

11. A message processing apparatus, the apparatus comprising:

the message detection unit is used for detecting a message to be processed pushed by a message management process, wherein the message management process is used for managing the message to be processed in a message queue, the message to be processed is pushed by the message management process based on the push priority of each consumption process in a plurality of consumption processes, the push priority of the consumption process is positively correlated with the target ratio of the consumption process, the target ratio of the consumption process is the ratio between the number of target consumption threads in the consumption process and the total number of consumption threads in the consumption process, and the target consumption threads are idle consumption threads in the consumption process and are used for carrying out consumption processing on the message to be processed;

12. An electronic device comprising a processor and a memory, the memory storing a plurality of instructions; the processor loads instructions from the memory to perform the steps of the message processing method according to any of claims 1 to 9.

13. A computer readable storage medium storing a plurality of instructions adapted to be loaded by a processor to perform the steps of the message processing method of any of claims 1 to 9.