CN116662022B

CN116662022B - Distributed message processing method, system, device, communication equipment and storage medium

Info

Publication number: CN116662022B
Application number: CN202310961369.8A
Authority: CN
Inventors: 武鹏
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2023-08-02
Filing date: 2023-08-02
Publication date: 2024-02-20
Anticipated expiration: 2043-08-02
Also published as: CN116662022A

Abstract

The embodiment of the application provides a distributed message processing method, a system, a device, communication equipment and a storage medium, comprising the following steps: receiving a first message sent by a production end and an allocation request corresponding to the first message; in response to the allocation request, querying a registry for a list of message services and a list of message volumes in a non-operational state; and determining a message service node corresponding to the first message according to the message service list, so that the message service node receives the first message and sends the first message to all message volume nodes in the message volume list, so that all message volume nodes store the first message. According to the embodiment of the application, by introducing the independent message volume node and only responsible for storing the messages, the transverse expansion or contraction can be independently carried out, and the response of the load balancing node to the distribution request of the production end can be used for forwarding the request of the user (the first message produced by the production end) to one message service node in sequence for processing.

Description

Distributed message processing method, system, device, communication equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a method, a system, an apparatus, a communication device, and a storage medium for processing a distributed message.

Background

Message middleware (message platform) is a supportive software system that provides synchronous or asynchronous, reliable message transmission for application systems in a network environment based on queue and message passing techniques. In short, the message middleware focuses on the sending and receiving of data, and utilizes an efficient and reliable information transfer mechanism to integrate all subsystems in the distributed system.

The current message middleware can meet the basic message transfer function and can also perform main-standby switching under the condition of abnormal main node, but the message service node heavily depends on a local file system for storing the message, and when the message service node receives the message, the message is additionally written on a local disk, so that the storage of the message cannot be independently and transversely expanded, and the storage capacity of the message is limited.

Disclosure of Invention

An embodiment of the present application is to provide a method, a system, an apparatus, a communication device, and a storage medium for processing distributed messages, so as to solve the technical problem that in the prior art, the storage of the messages cannot be laterally expanded independently, and the storage capacity of the messages is limited. The specific technical scheme is as follows:

In a first aspect of the present application, there is first provided a distributed message processing method applied to a load balancing node, where the distributed message processing method includes:

receiving an allocation request corresponding to a first message sent by a production end;

querying a message service list and a message volume list in a non-working state in a registry in response to the allocation request;

and determining a message service node corresponding to the first message according to the message service list so that the message service node receives the first message and sends the first message to all message volume nodes in the message volume list so that all the message volume nodes store the first message.

Optionally, the determining, according to the message service list, the message service node corresponding to the first message includes:

and determining a message service node corresponding to the first message in the message service list based on a polling access mode.

Optionally, the distributed message processing method further includes:

receiving a message reading request corresponding to a second message sent by a consumer;

responding to the message reading request, and inquiring a message service list and a message volume list in a non-working state in a registry;

And determining a message service node and a message volume node corresponding to the second message according to the message service list and the message volume list, wherein the message volume node comprises the second message corresponding to the message reading request, and the message service node is used for sending the second message to the consumption terminal.

In yet another aspect of the present application, there is further provided a distributed message processing method applied to a message service node, where the distributed message processing method includes:

receiving a first message, wherein the first message is sent to the load balancing node by the production end, the load balancing node queries a message service list and a message volume list which are in a non-working state in a registry according to an allocation request sent by the production end, and determines a message service node corresponding to the first message according to the message service list;

and sending the first message to all message volume nodes in the message volume list so that all the message volume nodes store the first message.

Optionally, after the step of sending the first message to all message volume nodes in the message volume list, so that all the message volume nodes store the first message, the method includes:

Obtaining a target message, wherein the target message is generated by a data stream calculation module according to a function code uploaded by a user to generate a calculation function and carrying out data stream processing on the first message according to the calculation function;

and sending the target message to a consumer.

Optionally, the computation function includes computation rules corresponding to the first message, and the target message is the data flow computation module

And generating a calculation task according to the calculation rule, and starting a thread pool to execute the calculation task, wherein the calculation rule comprises a message service node corresponding to the first message, a message service node corresponding to the target message, and frequency and times corresponding to the calculation task.

In yet another aspect of the present application, there is also provided a distributed message processing system, the system comprising: a message flow cluster comprising a load balancing node, at least one message service node, and at least one message volume node;

the load balancing node is used for receiving an allocation request corresponding to a first message sent by a production end; querying a message service list and a message volume list in a non-working state in a registry in response to the allocation request; determining a message service node corresponding to the first message according to the message service list, so that the message service node receives the first message and sends the first message to all message volume nodes in the message volume list, so that all the message volume nodes store the first message;

The message service node is used for receiving a first message; and sending the first message to all message volume nodes in the message volume list so that all the message volume nodes store the first message.

Optionally, the system further comprises: a production end;

the production end is used for sending a first message to the load balancing nodes in the message flow cluster and an allocation request corresponding to the first message.

Optionally, the system further comprises: a consumer end;

the consumer is configured to send a message read request for a second message to the message flow cluster.

Optionally, the message volume node is configured to store the first message, and the second message.

Optionally, the message volume node includes a disk usage alarm, where the disk usage alarm is configured to monitor usage of a local disk on the node in real time, and report the usage to a registry.

Optionally, the message volume node is further configured to send alarm information to the registry when detecting that the usage rate of the local disk is greater than a preset threshold, so that the registry sends message volume node capacity expansion information to a user according to the alarm information.

Optionally, the message flow cluster further comprises a data flow calculation module;

the data flow calculation module is in communication connection with the message service node;

the data flow calculation module is used for generating a calculation function according to a function code uploaded by a user, acquiring a first message in the message service node, carrying out data flow processing on the first message according to the calculation function, generating a target message, and sending the target message to a consumer through the message service node.

Optionally, the message flow cluster further comprises a registry;

the registry is configured to maintain a message service list and a message volume list, where the message service list includes all message service nodes and states corresponding to the message service nodes, and the message volume list includes all message volume nodes, states corresponding to the message volume nodes, and disk usage.

Optionally, the state includes at least one of: a non-operating state, and an operating state.

Optionally, one of the message service nodes is communicatively connected to a plurality of the message volume nodes.

In yet another aspect of the present application, there is further provided a distributed message processing apparatus, applied to a load balancing node, the apparatus including:

The receiving module is used for receiving an allocation request corresponding to the first message sent by the production end;

the inquiring module is used for responding to the allocation request and inquiring a message service list and a message volume list which are in a non-working state in a registry;

and the determining module is used for determining the message service node corresponding to the first message according to the message service list so that the message service node receives the first message and sends the first message to all message volume nodes in the message volume list so that all the message volume nodes store the first message.

In yet another aspect of the present application, there is also provided a distributed message processing apparatus, applied to a message service node, the apparatus including:

the receiving module is used for receiving a first message, wherein the first message is sent to the load balancing node by the production end, the load balancing node inquires a message service list and a message volume list which are in a non-working state in a registry according to an allocation request sent by the production end, and the message service node corresponding to the first message is determined according to the message service list;

And the sending module is used for sending the first message to all message volume nodes in the message volume list so as to enable all the message volume nodes to store the first message.

In yet another aspect of the present application, there is provided a communication device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

a memory for storing a computer program;

and the processor is used for realizing any one of the distributed message processing methods when executing the programs stored in the memory.

In yet another aspect of the application implementation, there is also provided a computer readable storage medium having instructions stored therein, which when run on a computer, cause the computer to perform any of the above-described distributed message processing methods.

According to the distributed message processing method, an allocation request corresponding to a first message sent by a production end is received; querying a message service list and a message volume list in a non-working state in a registry in response to the allocation request; and determining a message service node corresponding to the first message according to the message service list so that the message service node receives the first message and sends the first message to all message volume nodes in the message volume list so that all the message volume nodes store the first message. According to the embodiment of the application, by introducing the independent message volume node, only the storage of the message is responsible, the transverse expansion or contraction can be independently carried out, and further the calculation and the storage of the message are separated, namely the processing and the storage of the message are not bound in the same 1 node but are carried out in 2 nodes (the message service node and the message volume node), the message service node is used for processing the message, the message volume nodes are used for storing the message and mutually independent, the transverse expansion or contraction can be independently carried out without mutual influence, and the request of a user (the first message produced by the production end) can be sequentially forwarded to one message service node for processing through the response of the load balancing node to the allocation request of the production end.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

FIG. 1 is a flowchart showing steps of a method for processing a distributed message according to an embodiment of the present application;

FIG. 2 shows a second flowchart of steps of a method for processing a distributed message according to an embodiment of the present application;

FIG. 3 shows a third flowchart of steps of a distributed message processing method provided by an embodiment of the present application;

FIG. 4 shows a fourth flowchart of steps of a distributed message processing method provided in an embodiment of the present application;

FIG. 5 shows a schematic diagram of a distributed message processing system provided by an embodiment of the present application;

FIG. 6 shows a device block diagram of a distributed message processing device provided by an embodiment of the present application;

FIG. 7 illustrates an apparatus block diagram of another distributed message processing apparatus provided by an embodiment of the present application;

fig. 8 shows a block diagram of a communication device according to an embodiment of the present application;

fig. 9 is a schematic diagram of a message middleware operation principle in the related art according to an embodiment of the present application;

fig. 10 shows a schematic diagram of message writing in a distributed message processing method according to an embodiment of the present application;

Fig. 11 shows a schematic diagram of data flow calculation in a distributed message processing method according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the following detailed description of each embodiment of the present application will be given with reference to the accompanying drawings. However, those of ordinary skill in the art will understand that in various embodiments of the present application, numerous technical details have been set forth in order to provide a better understanding of the present application. However, the technical solutions claimed in the present application can be implemented without these technical details and with various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not be construed as limiting the specific implementation of the present application, and the embodiments may be mutually combined and referred to without contradiction.

As shown in fig. 9, fig. 9 illustrates the principle of operation of a common message middleware. Message middleware is generally divided into 3 parts: producer (Producer) consumers (Consumer) and message services (Broker). The producer generates a message and sends it to the messaging service. The message service is generally deployed on a plurality of hosts in a master-slave cluster mode, writing and reading of the message are generally completed through a master node, a slave node is used as a standby node to switch when the master node is down, and the master node can synchronize to the slave node after writing data. Messages are stored on the local disk of the host where the message service resides, and the messages belong to a Topic (Topic) for categorizing the messages. The consumer reads the message from the message service for subsequent business processing. The general message middleware can only send and receive the message, cannot perform real-time streaming calculation when receiving the message, and can also independently deploy other systems if necessary. The current message middleware can meet the basic message transfer function and also can perform active-standby switching under the condition of abnormal master nodes, but has the following problems: the message service Broker is heavily dependent on the local file system for storing messages, and when the Broker receives a message, the Broker additionally writes the message to the local disk. The architecture determines that the storage of the message cannot be subjected to independent transverse expansion, and the storage capacity of the message can be increased only by adding a Broker node at the same time; the single-point bottleneck of reading and writing exists, the reading and writing are carried out through the master node, and the pressure of the master node is high; the real-time streaming calculation of the message cannot be performed in the Broker, and if necessary, additional systems are deployed, thereby increasing additional overhead.

Note that, the message middleware generally includes: a Producer Consumer (Consumer) and a message service (Broker), in which in the embodiment of the present application, the message service corresponds to a message service node in a message flow cluster, the Producer corresponds to a Producer side, and the Consumer corresponds to a Consumer side.

Referring to fig. 1, a first step flowchart of a distributed message processing method provided in an embodiment of the present application is shown, where the method may include:

step 101, receiving an allocation request corresponding to a first message sent by a production end.

It should be noted that, the embodiment of the present application is applied to a load balancing node, where a load balancing software component is carried on the node and may be represented as a load balancing, where a production end is a server or a terminal where a producer is located in a message middleware, and a first message is a message sent by the production end to a message flow cluster in the present application.

The load balancing node is one of the modules in the message flow cluster, which may include a message service node, a message volume node, a load balancing node, a registry, and a data flow calculation module.

The load balancing node is an exchange interface for external (producer and consumer), and after receiving a request for sending or obtaining a message, the load balancing node forwards the request to a certain 1 surviving Broker in the cluster for processing, so in the embodiment of the application, the load balancing node receives an allocation request corresponding to a first message sent by a production end, and the load balancing node determines a message service node of the received first message based on the allocation request.

And step 102, responding to the allocation request, and inquiring a message service list and a message volume list in a non-working state in a registry.

It should be noted that, the load balancing node is a switching interface for external (producer and consumer), and after receiving a request for sending or obtaining a message, the load balancing node forwards the request to a certain 1 surviving Broker in the cluster for processing, where the surviving Broker node is determined according to real-time heartbeat detection.

Specifically, two lists, namely a message volume list and a message service list, are recorded in the registry, the Broker list stores the currently surviving Broker node, and the Broker and the registry do real-time heartbeat detection and update the list. The message volume list records the current surviving message volume nodes and disk usage of the nodes, and the message volume nodes update the heartbeat (surviving) and usage in real time.

Step 103, determining a message service node corresponding to the first message according to the message service list, so that the message service node receives the first message, and sending the first message to all message volume nodes in the message volume list, so that all the message volume nodes store the first message.

Further, the determining, according to the message service list, the message service node corresponding to the first message includes: and determining a message service node corresponding to the first message in the message service list based on a polling access mode.

It should be noted that, in this embodiment of the present application, as shown in fig. 10, fig. 10 shows a schematic message writing diagram in a distributed message processing method provided in this embodiment of the present application, a producer sends a message to a message flow platform, after receiving a request, a load balancer queries a registration center for a surviving Broker list and a surviving message volume list, 1 is selected in a polling manner, forwards the request to the Broker, and the Broker sends the message to all surviving message volume nodes under a default request, so as to save the message in a multi-copy redundancy manner.

Thus, the load balancing node, after determining the message service node for the received first message, informs the message service node that the first message is stored to all surviving message volume nodes after the message service node receives the first message.

It should be noted that, in the embodiment of the present application, the number of message service nodes and the number of message volume nodes are not limited, and the distributed message architecture in the present application adopts a architecture design of separating computation and storage, and the Broker node is only responsible for receiving and sending the message, and is no longer responsible for storing the message, so that the message volume node is used for storing the message, and the message volume node and the Broker node are independent from each other, and can be respectively and laterally expanded without influencing each other.

Referring to fig. 2, a second step flowchart of a distributed message processing method provided in an embodiment of the present application is shown, where the method may include:

step 201, receiving an allocation request corresponding to a first message sent by a production end.

Step 202, in response to the allocation request, querying a registry for a list of message services and a list of message volumes in a non-operational state.

Step 203, determining a message service node corresponding to the first message according to the message service list, so that the message service node receives the first message, and sending the first message to all message volume nodes in the message volume list, so that all message volume nodes store the first message.

It should be noted that, in the embodiment of the present application, the steps 201 to 203 are discussed with reference to the foregoing, and are not repeated.

Step 204, receiving a message reading request corresponding to the second message sent by the consumer.

In response to the message reading request, the registry is queried for a list of message services and a list of message volumes in a non-operational state, step 205.

And step 206, determining a message service node and a message volume node corresponding to the second message according to the message service list and the message volume list, wherein the message volume node comprises the second message corresponding to the message reading request, and the message service node is used for sending the second message to the consumer.

It should be noted that, in the embodiment of the present application, the steps 204 to 206 are a process that the consumer obtains the second message from the message flow cluster, where the consumer side is a server or a terminal where the consumer is in the message middleware, and the second message is a message that the consumer side requests to obtain from the message flow cluster in the present application.

Specifically, the flow of the consumer reading the message is similar to the process of writing the message by the producer, the consumer sends a request to the load balancer, the load balancer queries the surviving Broker list and the surviving message volume list, 1 is selected in a polling mode, and forwards the request to the Broker, the Broker finds the 1 st volume node from the surviving message volume list and queries the message, and generally, the result can be queried, and if the result is not queried, the subsequent nodes are queried in turn.

According to the embodiment of the application, by introducing the independent message volume node, only the storage of the message is responsible, the transverse expansion or contraction can be independently carried out, and further the calculation and the storage of the message are separated, namely the processing and the storage of the message are not bound in the same 1 node but are carried out in 2 nodes (the message service node and the message volume node), the message service node is used for processing the message, the message volume nodes are used for storing the message and are mutually independent, and the transverse expansion or contraction can be independently carried out without mutual influence.

In addition, through the embodiment of the application, the producer can send and store the message to the message flow cluster, the consumer can acquire the message from the message flow cluster, and through the response of the load balancing node to the allocation request of the production end, the request of the user (the first message produced by the production end) can be forwarded to one message service node in sequence for processing.

Referring to fig. 3, a step flowchart three of a distributed message processing method provided in an embodiment of the present application is shown, where the method may include:

step 301, receiving a first message, where the first message is sent by the production end to the load balancing node, and the load balancing node queries a message service list and a message volume list in a non-working state in a registry according to an allocation request sent by the production end, and determines a message service node corresponding to the first message according to the message service list.

Step 302, sending the first message to all message volume nodes in the message volume list, so that all the message volume nodes store the first message.

It should be noted that, the steps 301-302 are applied to the message service node, and specific reference may be made to the foregoing introduction, and detailed description is omitted herein.

According to the embodiment of the application, by introducing the independent message volume node, only the storage of the message is responsible, the transverse expansion or contraction can be independently carried out, and further the calculation and the storage of the message are separated, namely the processing and the storage of the message are not bound in the same 1 node but are carried out in 2 nodes (the message service node and the message volume node), the message service node is used for processing the message, the message volume nodes are used for storing the message and mutually independent, the transverse expansion or contraction can be independently carried out without mutual influence, and in the application, the response of the load balancing node to the distribution request of the production end can be realized, and the request of a user (the first message produced by the production end) is sequentially forwarded to one message service node for processing.

Referring to fig. 4, a fourth step flowchart of a distributed message processing method provided in an embodiment of the present application is shown, where the method may include:

step 401, receiving a first message, where the first message is sent by the production end to the load balancing node, and the load balancing node queries a message service list and a message volume list in a non-working state in a registry according to an allocation request sent by the production end, and determines a message service node corresponding to the first message according to the message service list.

Step 402, sending the first message to all message volume nodes in the message volume list, so that all the message volume nodes store the first message.

It should be noted that, the steps 401-402 are discussed with reference to the foregoing, and are not repeated herein.

Step 403, obtaining a target message, where the target message is generated by a data flow computing module according to a function code uploaded by a user to generate a computing function, and performing data flow processing on the first message according to the computing function.

Further, the computing function includes a computing rule corresponding to the first message, the target message is generated by the data flow computing module according to the computing rule, and a thread pool is started to execute the computing task, the computing rule includes a message service node corresponding to the first message, a message service node corresponding to the target message, and a frequency and a number of times corresponding to the computing task.

Step 404, the target message is sent to the consumer.

It should be noted that, in the above steps 403-404, the message flow cluster includes a data flow calculation module, which may receive the function code uploaded by the user, generate a calculation task according to the logic set in the code, run the task in the thread pool to calculate the message of the designated Topic, and generate the message to the new Topic.

Wherein the message is stored on a local disk of a host where the message service is located, the message belongs to a Topic (Topic) for classifying the message.

The user self-defines a plurality of function codes, wherein the function codes comprise the calculation logic of the message (which Topic reads the message from which Topic the generated message belongs to, and the frequency and the times of the calculation tasks) and the message is uploaded to a message flow platform or is deployed along with the message flow platform, and after the function codes are received, a data flow calculator generates the calculation tasks according to the calculation logic and starts a thread pool to execute the calculation tasks.

Further, in the embodiment of the present application, in a form of calculation supporting streaming, as shown in fig. 11, fig. 11 shows a schematic diagram of data flow calculation in the distributed message processing method provided in the embodiment of the present application, that is, the last output message may be used as input to participate in the next calculation, 2 functions are defined by the user, 2 calculation tasks are generated, the task 1 reads the messages of the Topic1 and the Topic2, a new Topic3 message is generated after processing according to the logic calculation of the user, and the message and the original message of the Topic2 participate in the calculation of the task 2 again, so as to generate a new Topic4 message again.

According to the embodiment of the application, by introducing the independent message volume node, only the storage of the message is responsible, the transverse expansion or contraction can be independently carried out, and further the calculation and the storage of the message are separated, namely the processing and the storage of the message are not bound in the same 1 node but are carried out in 2 nodes (the message service node and the message volume node), the message service node is used for processing the message, the message volume nodes are used for storing the message and mutually independent, the transverse expansion or contraction can be independently carried out without mutual influence, and the request of a user (the first message produced by the production end) can be sequentially forwarded to one message service node for processing through the response of the load balancing node to the allocation request of the production end.

In addition, by introducing a data flow calculator, the cluster not only has the function of transmitting the message, but also has the function of calculating the flow of the message, and a set of system is deployed, so that the transmission and calculation of the message can be completed at the same time.

Referring to fig. 5, fig. 5 shows a schematic diagram of a distributed message processing system according to an embodiment of the present application, where the system includes: a message flow cluster comprising a load balancing node, at least one message service node, and at least one message volume node;

It should be noted that, in this embodiment of the present application, as shown in fig. 5, the distributed message processing system includes a production end, a consumption end, and a message flow cluster, where the message flow cluster includes a load balancing node, at least one message service node, at least one message volume node, a data flow calculation module, and a registry.

Further, one of the message service nodes is communicatively coupled to a plurality of the message volume nodes.

The architecture design of separating calculation and storage is adopted, the Broker node is only responsible for receiving and sending the message, is not responsible for storing the message any more, and the message volume node is introduced for storing the message, and is mutually independent, can be respectively and transversely expanded and has no influence on each other, such as: the message read-write pressure is smaller, but the data volume is larger, the data volume node can be increased independently, and the Broker node is kept unchanged. The master-slave architecture design of a Broker is canceled, a centerless cluster design is changed, a load balancer is introduced, a producer sends a message to the load balancer, the balancer forwards the message to a certain Broker, the Broker sends the message to each message volume node for storage, a data stream calculator is introduced, the messages of certain Topic can be read according to a user-defined calculation function and rule, and new Topic messages are produced after calculation and stored in the message volume nodes.

Further, the system further comprises: a production end;

Further, the system further comprises: a consumer end;

Further, the message volume node is configured to store the first message, and a second message.

The message volume node comprises a disk usage alarm which is used for monitoring the usage of the local disk on the node in real time and reporting the usage to a registry.

Further, the message volume node is further configured to send alarm information to the registry when it is detected that the usage rate of the local disk is greater than a preset threshold, so that the registry sends message volume node capacity expansion information to a user according to the alarm information.

It should be noted that, in the embodiment of the present application, the message volume node is a node that is only used to store the message, and uses the local disk to store the message, and is independent from the Broker node. Each message volume node is internally provided with a disk utilization rate alarm, the utilization rate of a local disk can be monitored in real time and reported to a registry, when the utilization rate exceeds a threshold value, an alarm is sent to the registry, and a user can expand the volume of the message volume node and transfer data according to the alarm.

Specifically, when a disk with higher usage rate is detected, an alarm is given to the registry, and the user can check the usage rate of the disk in each node currently according to the registry, for example, when the writing information of the node 2 in the message volume nodes 1, 2 and 3 is too much, the user can expand the capacity of the node 2 after sensing, or increase the message volume node 4, and redistribute the data stored in the original node 2.

Further, the message flow cluster further comprises a data flow calculation module;

It should be noted that, the message flow cluster includes a data flow calculation module, which can receive the function code uploaded by the user, generate a calculation task according to the logic set in the code, run the task in the thread pool to calculate the message of the designated Topic, and generate the message to the new Topic.

Further, the data stream calculation module stores a plurality of theme files and calculation tasks;

when the computing tasks comprise a first computing task and a second computing task, the data stream computing module comprises a first theme file, a second theme file, a third theme file and a fourth theme file, wherein the first theme file comprises commodity order information sent by a production end, the second theme file comprises payment order information sent by the production end, the third theme file comprises customer point information, and the fourth theme file comprises coupon information.

The customer point information is generated by the data flow calculation module through a first calculation task according to the commodity order information and the payment order information;

the coupon message is generated by the data flow computing module performing a second computing task based on the customer loyalty message.

It should be noted that, the theme file is Topic, and the calculation task is generated by analyzing the calculation rule according to the code uploaded by the user.

Therefore, the data stream computing module may store a plurality of theme files and computing tasks, and when the computing tasks include the first computing task and the second computing task, the data stream computing module may include Topic1, topic2, topic3, and Topic4.

Therefore, the data flow calculation module can be used for receiving commodity order information sent by the production end and payment order information corresponding to the commodity order information; judging whether the payment amount is larger than a target payment threshold value according to the payment order information; if the payment amount is larger than the target point, determining a target point according to the payment amount; determining a client number according to the payment serial number in the payment order message; generating a client point message according to the client number, the target point and the payment serial number, and storing the client point message in a third theme file in a data stream calculation module; and generating a coupon message according to the client point message and the payment order message, and storing the coupon message and a fourth theme file in the data stream calculation module.

Specifically, the user uploads 2 real-time computing functions to a data stream computing module in the distributed message processing system, the data stream computing module generates 2 real-time computing tasks according to the function content, the first computing task can be represented as task 1, task 1 is used for computing the integral of the client, the second computing task can be represented as task 2, and task 2 is used for computing the coupon of the client.

The production end generates commodity order information (Topic 1) and corresponding payment order information (Topic 2) and sends the commodity order information and the payment order information to the information flow platform, the task 1 reads the order information and the payment information in real time and carries out real-time calculation, and the calculation rule corresponding to the first calculation task comprises: for each payment message, if the payment is successful and the payment amount is greater than 10, the user's credit is increased (e.g., credit pts equals the amount divided by 10), then customer Id (cusId) is found from the payment serial number (payId) traversal order message, and the customer credit message is generated along with the credit (pts) and the payment serial number (payId) and saved to Topic3. Task 2 reads the customer loyalty message (Topic 3) and the payment order message (Topic 2) in real time, calculates and generates a coupon message, and saves the coupon message to Topic4. The calculation function (function 2) is as follows: for each credit message, if the single credit exceeds 100, a payment channel (channel) is found by traversing the payment message according to a payment serial number (payId), if the channel is a 'stored value number', coupons are added by dividing the credit by 100 (1 coupon is added per 100 credits), and the amount of each coupon is fixed to 10.

Further, the message flow cluster also includes a registry;

The state includes at least one of: a non-operating state, and an operating state.

It should be noted that, in the embodiment of the present application, 2 lists are maintained in the registry, and the Broker list stores the currently surviving Broker nodes, and the Broker and the registry do real-time heartbeat detection and update the list. The message volume list records the current surviving message volume nodes and disk usage of the nodes, and the message volume nodes update the heartbeat (surviving) and usage in real time.

The node is in a non-working state, namely a node which is alive at present, and the node is in a working state, namely a node which is occupied at present or the disk utilization rate exceeds a preset threshold value.

In addition, the distributed message processing system is arranged, namely, a framework with separated computation and storage is designed, the computation and the storage are mutually independent, and can be independently expanded or contracted transversely without mutual influence, so that the single-point pressure problem of the original master Broker node is solved. The streaming computation can perform the computation processing again on the original message, and a set of platform is deployed, so that the real-time transmission and streaming computation of the message can be completed simultaneously.

Referring to fig. 6, fig. 6 shows a schematic diagram of a distributed message processing system provided in an embodiment of the present application, where the distributed message processing system is applied to a load balancing node, and the apparatus includes:

a receiving module 601, configured to receive an allocation request corresponding to a first message sent by a production end;

a query module 602, configured to query a registry for a list of message services and a list of message volumes in a non-working state in response to the allocation request;

and the determining module 603 is configured to determine, according to the message service list, a message service node corresponding to the first message, so that the message service node receives the first message, and send the first message to all message volume nodes in the message volume list, so that all message volume nodes store the first message.

Referring to fig. 7, fig. 7 shows a schematic diagram of another distributed message processing system provided in an embodiment of the present application, applied to a message service node, where the apparatus includes:

a receiving module 701, configured to receive a first message, where the first message is sent by the production end to the load balancing node, and the load balancing node queries a message service list and a message volume list in a non-working state in a registry according to an allocation request sent by the production end, and determines a message service node corresponding to the first message according to the message service list;

A sending module 702, configured to send the first message to all message volume nodes in the message volume list, so that all the message volume nodes store the first message.

The embodiment of the present application also provides a communication device, as shown in fig. 8, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804, where the processor 801, the communication interface 802, and the memory 803 complete communication with each other through the communication bus 804,

A memory 803 for storing a computer program;

the processor 801, when executing the program stored in the memory 803, may implement the following steps:

Or, receiving a first message, wherein the first message is sent to the load balancing node by the production end, and the load balancing node queries a message service list and a message volume list in a non-working state in a registry according to an allocation request sent by the production end, and determines a message service node corresponding to the first message according to the message service list;

The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the terminal and other devices.

The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment provided herein, there is also provided a computer readable storage medium having instructions stored therein that, when run on a computer, cause the computer to perform congestion control as described in any of the above embodiments.

In yet another embodiment provided herein, there is also provided a computer program product containing instructions that, when run on a computer, cause the computer to perform congestion control as described in any of the above embodiments.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or third database to another website, computer, server, or third database by a wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more servers, third databases, etc. that can be integrated with the available medium. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and principles of the present application are intended to be included within the scope of the present application.

Claims

1. A distributed message processing method, applied to a load balancing node, the distributed message processing method comprising:

responding to the allocation request, inquiring a message service list and a message volume list in a non-working state in a registry, wherein the registry comprises the message volume list and the message service list, the message service list stores a current surviving message service node, the message volume list records the current surviving message volume node and the disk utilization rate corresponding to the message volume node, the message service node and the registry update the message service list based on real-time heartbeat detection, and the message volume node updates the message volume list based on real-time heartbeat detection and utilization rate;

determining a message service node corresponding to the first message according to the message service list, so that the message service node receives the first message, and sends the first message to all message volume nodes in the message volume list, so that all the message volume nodes store the first message, one message service node is in communication connection with a plurality of message volume nodes, the message service node is used for receiving the first message, and sends a second message to a consumption terminal, and the message volume nodes are used for storing the first message and the second message.

2. The method according to claim 1, wherein determining a message service node corresponding to the first message according to the message service list comprises:

3. The distributed message processing method of claim 1, further comprising:

4. A distributed message processing method, applied to a message service node, the distributed message processing method comprising:

receiving a first message, wherein the first message is sent to a load balancing node by a production end, the load balancing node queries a message service list and a message volume list which are in a non-working state in a registry according to an allocation request sent by the production end, and determines a message service node corresponding to the first message according to the message service list;

Sending the first message to all message volume nodes in the message volume list, so that all message volume nodes store the first message, one message service node is in communication connection with a plurality of message volume nodes, the message service node is used for receiving the first message, and sending a second message to a consumption end, and the message volume nodes are used for storing the first message and the second message;

the registration center comprises a message volume list and a message service list, wherein the message service list stores the current surviving message service node, the message volume list records the current surviving message volume node and the disk utilization rate corresponding to the message volume node, the message service node and the registration center update the message service list based on real-time heartbeat detection, and the message volume node updates the message volume list based on real-time heartbeat detection and utilization rate.

5. The method of distributed message processing according to claim 4, wherein after the step of sending the first message to all of the message volume nodes in the message volume list to cause all of the message volume nodes to store the first message, the method comprises:

and sending the target message to a consumer.

6. The method of claim 5, wherein the computation function includes computation rules corresponding to the first message, and the target message is the data flow computation module

7. A distributed message processing system, the system comprising: a message flow cluster comprising a load balancing node, at least one message service node, and at least one message volume node;

The message service node is used for receiving a first message; sending the first message to all message volume nodes in the message volume list, so that all message volume nodes store the first message, one message service node is in communication connection with a plurality of message volume nodes, the message service node is used for receiving the first message, and sending a second message to a consumption end, and the message volume nodes are used for storing the first message and the second message;

8. The distributed message processing system of claim 7, wherein the system further comprises: a production end;

9. The distributed message processing system of claim 7, wherein the system further comprises: a consumer end;

10. The distributed message processing system of claim 7, wherein the message volume node is configured to store the first message and a second message.

11. The distributed message processing system of claim 10, wherein the message volume node includes a disk usage alert for monitoring usage of a local disk on the node in real time and reporting the usage to a registry.

12. The distributed message processing system of claim 11, wherein the message volume node is further configured to send alert information to the registry upon detecting that the local disk usage is greater than a preset threshold, such that the registry sends message volume node expansion information to a user based on the alert information.

13. The distributed message processing system of claim 7, wherein the message flow cluster further comprises a data flow calculation module;

14. The distributed message processing system of claim 13, wherein the data stream computing module has a plurality of theme files and computing tasks stored therein;

15. The distributed message processing system of claim 14, wherein the customer loyalty message is generated by the data flow calculation module performing a first calculation task from the merchandise order message and the payment order message;

16. The distributed message processing system of claim 7, wherein the message flow cluster further comprises a registry;

17. A distributed message processing apparatus for use with a load balancing node, the apparatus comprising:

the inquiring module is used for responding to the allocation request, inquiring a message service list and a message volume list which are in a non-working state in a registry, wherein the registry comprises the message volume list and the message service list, the message service list stores the current surviving message service nodes, the message volume list records the current surviving message volume nodes and the disk utilization rate corresponding to the message volume nodes, the message service nodes and the registry update the message service list based on real-time heartbeat detection, and the message volume nodes update the message volume list based on real-time heartbeat detection and utilization rate;

The determining module is configured to determine, according to the message service list, a message service node corresponding to the first message, so that the message service node receives the first message, and sends the first message to all message volume nodes in the message volume list, so that all message volume nodes store the first message, one message service node is communicatively connected to a plurality of message volume nodes, and the message service node is configured to receive the first message, and send a second message to a consumer, where the message volume nodes are configured to store the first message and the second message.

18. A distributed message processing apparatus for use with a message service node, the apparatus comprising:

the receiving module is used for receiving a first message, wherein the first message is sent to a load balancing node by a production end, the load balancing node inquires a message service list and a message volume list which are in a non-working state in a registry according to an allocation request sent by the production end, and the message service node corresponding to the first message is determined according to the message service list;

a sending module, configured to send the first message to all message volume nodes in the message volume list, so that all message volume nodes store the first message, one message service node is communicatively connected to a plurality of message volume nodes, and the message service node is configured to receive the first message, and send a second message to a consumer, where the message volume node is configured to store the first message and the second message;

19. A communication device, comprising: a transceiver, a memory, a processor, and a program stored on the memory and executable on the processor;

the processor is configured to read a program in the memory to implement the distributed message processing method according to any one of claims 1 to 3 or to implement the distributed message processing method according to any one of claims 4 to 6.

20. A readable storage medium storing a program, wherein the program when executed by a processor implements the distributed message processing method according to any one of claims 1-3 or the distributed message processing method according to any one of claims 4-6.