CN116055401A - Message processing method, device, equipment and storage medium - Google Patents

Abstract

The present application discloses a message processing method, device, device, and storage medium, wherein the method includes: acquiring the real-time consumption backlog value of the publish-subscribe message system; and determining the consumption backlog real-time value of the publish-subscribe message system Message congestion occurs; adjust system parameters of the publish-subscribe message system; complete message processing based on the adjusted system parameters.

Description

A message processing method, device, equipment and storage medium

technical field

The embodiment of the present application relates to the field of big data processing, and relates to but not limited to a message processing method, device, equipment and storage medium.

Background technique

Distributed messaging systems are widely used in high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. In the distributed message system, there are scenarios where the volume of business data increases or a large number of messages burst out. In this scenario, the channel and consumption capacity reach the upper limit, and the message system capacity is concentrated on data forwarding and message buffering. The congestion status is not fed back to the producer and consumer, resulting in the inability to control the congestion in real time, which will cause data congestion in the production environment. pressure, resulting in delays in business processing.

Contents of the invention

In view of this, embodiments of the present application provide a message processing method, device, device, and storage medium.

The technical scheme of the embodiment of the application is realized in this way:

In the first aspect, the embodiment of the present application provides a message processing method, the method including:

Obtain the real-time value of the consumption backlog of the publish-subscribe message system;

Based on the real-time value of the consumption backlog, it is determined that message congestion occurs in the publish-subscribe message system;

Adjust system parameters of the publish-subscribe message system;

Complete message processing based on the adjusted system parameters.

In a second aspect, an embodiment of the present application provides a message processing device, the device comprising:

An acquisition module, configured to acquire the real-time value of the consumption backlog of the publish-subscribe message system;

A determining module, configured to determine that message congestion occurs in the publish-subscribe message system based on the real-time value of the consumption backlog;

An adjustment module, configured to adjust system parameters of the publish-subscribe message system;

The message processing module is configured to complete message processing based on the adjusted system parameters.

In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the above method when executing the program.

In a fourth aspect, the embodiment of the present application provides a storage medium storing executable instructions for implementing the above method when executed by a processor.

In the embodiment of the present application, first obtain the consumption backlog real-time value of the publish-subscribe message system; then based on the consumption backlog real-time value, determine that the publish-subscribe message system has message congestion; adjust the system parameters of the publish-subscribe message system; finally Complete message processing based on the adjusted system parameters. In this way, the congestion control capability can be increased. Through dynamic detection and congestion feedback, after determining the backlog trend of the message channel, adjust the system parameters of the publish-subscribe message system, effectively control the production and consumption of messages, and alleviate message congestion.

Description of drawings

FIG. 1A is a schematic diagram of a publish-subscribe messaging system provided by an embodiment of the present application;

FIG. 1B is a schematic diagram of an implementation flow of a message processing method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of an implementation flow of a method for determining message congestion provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of an implementation flow of a method for determining message congestion provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a solution architecture for processing messages provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a message processing device provided in an embodiment of the present application;

FIG. 6 is a schematic diagram of a hardware entity of an electronic device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the application clearer, the specific technical solutions of the embodiments of the application will be further described in detail below in conjunction with the drawings in the embodiments of the application. The following examples are used to illustrate the present application, but not to limit the scope of the present application.

In the following description, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.

In the following description, the term "first\second\third" is only used to distinguish similar objects, and does not represent a specific ordering of objects. Understandably, "first\second\third" Where permitted, the specific order or sequencing may be interchanged such that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

FIG. 1A is a schematic diagram of a publish-subscribe message system provided by an embodiment of the present application. As shown in FIG. 1A, the schematic diagram includes a producer 11, a message channel, a message partition 12, and a consumer 13, wherein,

In the publish-subscribe message system, the producer (Producer) 11 of the message is called the publisher, and the consumer (Consumer) 13 is called the subscriber. The message is persisted in the message channel (topic) and the message partition (Partition) 12. Consumers can subscribe to one or more topics, consumer 11 can consume all the data in the topic, the same piece of data can be consumed by multiple consumers, and the data will not be deleted immediately after consumption.

In the distributed message system, the producers and consumers themselves do not know each other's situation. The producer does not know that the consumer's consumption is blocked, and cannot reduce the frequency of data transmission; the consumer does not know that the consumption cannot keep up with the production, and cannot flexibly adjust the consumption capacity.

The embodiment of the present application provides a message processing method, as shown in Figure 1B, the method includes:

Step S110, obtaining the real-time value of the consumption backlog of the publish-subscribe message system;

Here, publish-subscribe is a message paradigm, and the producer of the message will not send the message directly to a specific consumer. Instead, the published messages are grouped into different categories without having to determine which subscribers might exist. Likewise, Subscribers can express interest in one or more categories and only receive messages of interest without having to determine which Publishers exist. For example, Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action stream data of consumers in the website.

Consumption backlog value (LAG), the difference between the messages retained in the message middleware server and the consumed messages, that is, the amount of message accumulation is also called the consumption lag. The value of Lag has positive and negative numbers and 0. A positive Lag value means that Kafka data is backlogged, and the speed at which data is fed to Kafka is greater than the speed at which this data is consumed, indicating that the supply exceeds demand. Lag takes a negative value, indicating that supply is less than demand. The Lag value is 0, indicating that the producer and consumer rates are basically the same. In a busy message system, LAG is generally greater than 0 and remains at a relatively constant value, indicating that production and consumption have reached a balance.

During the implementation process, the consumption backlog value of the publish-subscribe message system can be obtained in real time, that is, the consumption backlog value currently used to represent the difference between the production message and the consumption message can be obtained.

Step S120, based on the real-time value of the consumption backlog, determine that message congestion occurs in the publish-subscribe message system;

Here, since the real-time value of the consumption backlog can be used to represent the difference between the production message and the consumption message, when the real-time value of the consumption backlog is determined to increase, it can be determined that the number of produced messages cannot be consumed in time, that is, the publish-subscribe message The system is congested with messages.

Step S130, adjusting the system parameters of the publish-subscribe message system;

Here, the system parameters of the publish-subscribe message system include at least the number of consumers, the number of partitions and the number of producers. During the implementation process, at least one of the above system parameters may be adjusted to alleviate message congestion when it is determined that message congestion occurs in the publish-subscribe message system. For example, producers and consumers can adjust data sending and data consumption capabilities according to congestion conditions to resolve congestion.

Step S140, complete message processing based on the adjusted system parameters.

During the implementation process, the message processing is completed based on the adjusted system parameters, which can effectively alleviate the message congestion of the publish-subscribe message system.

In the embodiment of the present application, first obtain the consumption backlog real-time value of the publish-subscribe message system; then based on the consumption backlog real-time value, determine that the publish-subscribe message system has message congestion; adjust the system parameters of the publish-subscribe message system; finally Complete message processing based on the adjusted system parameters. In this way, the congestion control capability can be increased. Through dynamic detection and congestion feedback, after determining the backlog trend of the message channel, adjust the system parameters of the publish-subscribe message system, effectively control the production and consumption of messages, and alleviate message congestion.

In some embodiments, as shown in FIG. 2, the above step S120 of "determining that message congestion occurs in the publish-subscribe message system based on the real-time value of the consumption backlog" may be implemented through the following steps:

Step S210, acquiring a preset consumption backlog threshold;

Here, the consumption backlog threshold can be determined according to the message processing capability of the publish-subscribe message system.

Step S220, determining that the real-time value of the consumption backlog is greater than the consumption backlog threshold;

During implementation, the consumption backlog value obtained in real time may be compared with the message backlog threshold to determine whether the real-time fee backlog value is greater than the consumption backlog threshold. When it is determined that the consumption backlog real-time value is greater than the consumption backlog threshold, it can be determined that the message processing capability of the publish-subscribe message system cannot meet the current message production. For example, in the scenario where the business data volume increases or a large number of messages burst Under normal circumstances, the real-time value of the consumption backlog is greater than the consumption backlog threshold, and the channel and consumption capacity reach the upper limit, which will cause data squeeze in the production environment and cause business processing delays.

Step S230, determining that message congestion occurs in the publish-subscribe message system.

In the embodiment of the present application, first obtain the preset consumption backlog threshold; then determine that the consumption backlog real-time value is greater than the consumption backlog threshold; determine that the publish-subscribe message system has message congestion. In this way, based on the consumption backlog threshold, it can be determined that the consumption backlog real-time value is greater than the consumption backlog threshold, and further effectively determine that message congestion occurs in the publish-subscribe message system.

In some embodiments, the above step S210 of "obtaining a preset consumption backlog threshold" may be implemented through the following steps:

Step 211, obtaining the number of production messages of the publish-subscribe message system per unit time;

Here, the number of production messages per unit time (n minutes) can be obtained.

Step 212. Obtain a ratio threshold corresponding to the number of production messages;

Here, a ratio threshold (m%) corresponding to the number of production messages per unit time (n minutes) may be obtained.

In the implementation process, step 211 can be performed first and then step 212 can be performed, step 212 can also be performed first and then step 211 can be performed, or step 211 and step 212 can be performed simultaneously, that is, there is no requirement for the order of execution of step 211 and step 212 .

Step 213: Multiply the production message quantity by the ratio threshold to obtain the consumption backlog threshold.

Here, the consumption backlog threshold can be obtained by multiplying the production message quantity by the ratio threshold.

During the implementation process, the following formula (1) can be used to determine the occurrence of message system congestion:

In this embodiment of the application, first obtain the production message quantity of the publish-subscribe message system per unit time; obtain the ratio threshold corresponding to the production message quantity; then multiply the production message quantity by the ratio threshold to obtain the The above consumption backlog threshold. In this way, an effective consumption backlog threshold can be determined based on the number of production messages of the publish-subscribe message system per unit time.

In some embodiments, the above step S210 of "obtaining a preset consumption backlog threshold" may be implemented through the following process:

The consumption backlog threshold is determined based on the message processing capability of the publish-subscribe message system.

Here, it may be determined that the message system is congested when it is determined that the LAG value is higher than the configured value (N).

During the implementation process, the following formula (2) can be used to determine the occurrence of message system congestion:

LAG > N (2);

Among them, N is the configuration value, that is, the consumption backlog threshold.

In the embodiment of the present application, the consumption backlog threshold can be effectively determined based on the message processing capability of the publish-subscribe message system.

In some embodiments, as shown in FIG. 3 , before step S220 "determining that message congestion occurs in the publish-subscribe messaging system", the method for determining that message congestion occurs in the publish-subscribe messaging system further includes:

Step S310, acquiring the first consumption backlog value and the second consumption backlog value respectively at intervals of unit time;

Here, the unit time can be set according to the actual situation, for example, the unit time can be set as one minute, or the unit time can be set as one second.

In the implementation process, LAG _x -1 can be obtained at time x-1, and LAG _x can be obtained at time x, that is, the first consumption backlog value LAG _x-1 and the second consumption backlog value LAG _x are respectively obtained at a unit time interval.

Step S320, subtracting the second consumption backlog value from the first consumption backlog value to obtain a consumption backlog difference;

During implementation, LAG _x-1 can be subtracted from LAG _x to obtain the consumption backlog difference.

Step S330, dividing the consumption backlog difference by the unit time to obtain a consumption backlog slope value;

During the implementation process, the consumption backlog slope value can be obtained based on the following formula (3), and the real-time slope P is:

Wherein, LAG _x is the LAG at time x, LAG _x-1 is the LAG at time x-1, T _x is the time value at time x, and T _x-1 is the time value at time x-1.

Step S340, acquiring the consumption backlog slope threshold;

During implementation, the consumption backlog slope threshold may be determined based on the message processing capability of the subscription message system.

Step S350, determining that at least two consecutive consumption backlog slope values are greater than the consumption backlog slope threshold.

In some embodiments, it may be determined that the message system is congested when it is determined that at least two consecutive values of P are higher than the consumption backlog slope threshold.

In the embodiment of the present application, firstly, the first consumption backlog value and the second consumption backlog value are respectively obtained at intervals of unit time; then the first consumption backlog value is subtracted from the second consumption backlog value to obtain the consumption backlog difference value; Divide the consumption backlog difference by the unit time to obtain the consumption backlog slope value; obtain the consumption backlog slope threshold; finally determine that at least two consecutive consumption backlog slope values are greater than the consumption backlog slope threshold. In this way, it can be determined based on the consumption backlog slope value that message congestion occurs in the publish-subscribe message system.

In some embodiments, the system parameters include at least the number of consumers, the number of partitions and the number of producers, and the above step S130 "adjusting the system parameters of the publish-subscribe message system" can be implemented by any of the following steps:

Step 131. When it is determined that the number of consumers is smaller than the number of partitions, increase the number of consumers;

Here, if the number of consumers is determined to be smaller than the number of partitions and the number of producers remains unchanged, message congestion can be alleviated by increasing the number of consumers.

Step 132, when it is determined that the number of consumers is equal to the number of partitions, increase the number of partitions and the number of consumers at the same time;

Here, if the number of consumers is determined to be equal to the number of partitions and the number of producers remains unchanged, message congestion can be alleviated by increasing the number of partitions and the number of consumers at the same time.

For example, a message queue will include at least one partition, and each partition corresponds to a working consumer (multiple partitions can share consumers, and the number of partitions determines the upper limit of working consumers), so adding partitions means expanding The upper limit of consumers is set, thus laying the foundation for the subsequent increase of consumers.

In the case of message congestion, dynamically increase the number of partitions and dynamically notify the consumer, so that the consumer can add more consumers according to the latest upper limit of the number of partitions to increase processing capacity and reduce congestion.

Step 133 , if it is determined that the number of producers, the number of consumers and the number of partitions remain unchanged, reduce the production frequency of the producers.

Here, the consumer is limited to the situation where resources cannot be increased, and the message congestion can be alleviated by reducing the production frequency of the producer.

For example, the producer can be notified of the message congestion situation. For producers who can adjust the sending capacity (such as reducing the data collection cycle), the congestion problem can be solved by sending a deceleration method, such as sending from 100 per second, reducing Send 10 per second. This applies to scenarios where data can be decelerated.

In this embodiment of the application, if it is determined that the number of consumers is less than the number of partitions, increase the number of consumers; or, if it is determined that the number of consumers is equal to the number of partitions, increase the number of The number of partitions and the number of consumers; or, in a case where it is determined that the number of producers, the number of consumers, and the number of partitions remain unchanged, reduce the production frequency of the producer. In this way, message congestion can be effectively alleviated.

In some embodiments, the above step S110 of "obtaining the real-time value of the consumption backlog of the publish-subscribe message system" may be implemented through the following steps:

Step 111, obtaining the number of produced messages and the number of read messages of the publish-subscribe message system in real time;

During implementation, the number of produced messages EndOffset and the number of read messages CommitOffset can be obtained.

Step 112: Subtract the number of read messages from the number of produced messages to obtain the real-time value of the consumption backlog.

During the implementation process, the following formula (4) can be used to obtain the real-time value of consumption pressure:

Lag=EndOffset-CommitOffset (4);

Among them, Lag indicates the consumption backlog value, EndOffset indicates the number of produced messages, and CommitOffset indicates the number of read messages.

In the embodiment of the present application, the number of produced messages and the number of read messages of the publish-subscribe message system are obtained in real time; then the number of produced messages is subtracted from the number of read messages to obtain the real-time value of the consumption backlog. In this way, the real-time value of the consumption backlog can be obtained by obtaining the number of production messages and the number of read messages obtained in real time.

Distributed messaging systems are currently widely used in high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Message sending and receiving generally includes the following steps:

Step A, message channel preparation;

In the message channel preparation stage, you can first create a message channel (topic) and a message partition (Partition).

Step B, message sending data;

In the message sending data stage, when the producer sends data, the message will be sent to the specified message partition according to the rules.

Step C, message receiving data;

In the stage of message receiving data, consumers subscribe to message partitions to realize parallel reception and processing of messages.

FIG. 4 is a schematic diagram of a solution architecture for processing messages provided by an embodiment of the present application. As shown in FIG. 4 , the solution architecture diagram includes an existing message system 41 and a congestion control service 42, wherein,

The congestion control service 42 is extended based on the existing messaging system 41 to provide congestion control capabilities and support message sending channels to form a closed loop.

Congestion control service 42 includes message channel throughput and accumulation information collection service 421 , data analysis and automatic adjustment service 422 and upper layer application service 423 .

Message channel throughput and accumulation information collection service 421 is responsible for channel monitoring and basic information collection for real-time statistical calculation of production, consumption throughput and congestion.

The data analysis and automatic adjustment service 422, after analyzing the data according to the policy, notifies the upper-layer application of the throughput and accumulation of the message channel. It can perceive and adjust the number of sending and consumption capabilities, and can also automatically adjust the number of partitions of the message channel, and notify the "upper layer application" of the policy adjustment. Supports automatic identification of multi-scenario message backlogs (the threshold is a certain percentage of the number of messages produced per unit time; the threshold is an absolute number; the accumulation trend exceeds a certain slope), and when the message backlog is determined, an automatic adjustment strategy is activated to adjust Message channel (topic) and message partition (Partition), realize the partition automatic adjustment strategy, and dynamically increase the number of message partitions.

The embodiment of this application provides three strategies for data analysis:

Strategy 1: The threshold for canceling the consumption backlog is a certain proportion of the number of production messages per unit time. For example, a certain proportion (m%) of the number of production messages per unit time (n minutes) may be acquired. If the LAG value is higher than the consumption backlog threshold, it is determined that the message system is congested.

During the implementation process, the following formula (1) can be used to determine the occurrence of message system congestion:

LAG>number of messages produced within n minutes*m%(1).

Strategy 2: Determine the consumption backlog threshold based on the message processing capability of the publish-subscribe message system, that is, determine that the message system is congested when the LAG value is determined to be higher than the configured value (N).

During the implementation process, the following formula (2) can be used to determine the occurrence of message system congestion:

LAG > N (2);

Among them, N is the configuration value.

Strategy 3: When the accumulation trend of messages exceeds a certain slope, it is determined that the message system is congested.

Obtain the consumption backlog slope threshold, and obtain the real-time slope P based on the following formula (3):

Wherein, LAG _x is the LAG at time x, LAG _x-1 is the LAG at time x-1, T _x is the time value at time x, and T _x-1 is the time value at time x-1.

In some embodiments, it may be determined that the message system is congested when it is determined that the value of P is higher than the consumption backlog slope threshold for M consecutive times.

Here, the configuration of the policy and whether it is enabled or not is configured through the threshold method, and is adjusted up by modifying the configuration. Generally, the configuration value is determined according to the business requirements. It includes determining the initial value of the partition during the business requirement analysis stage. After running, the monitoring data of the three enabled strategies can be used at the same time. If message congestion is determined under any strategy, the upper-layer application will be notified, and the number of partitions and consumers will be increased.

Whether the above three strategies take effect can be enabled through the configuration switch, that is, more than one can be enabled at the same time. For example, both policy 1 and policy 2 are enabled, and if any policy is detected to have reached the threshold during monitoring, the number of partitions will be adjusted and the upper-layer application will be notified.

The upper layer application service 423 is used to adjust the number of upper layer consumers according to the policy.

In the embodiment of this application, the channel increases the congestion control capability, and through dynamic detection and congestion feedback, the message transmission path forms a closed loop, effectively controlling the production and consumption of messages. In this way, after the message channel has a backlog trend, the channel capacity is automatically adjusted according to the policy, supporting the flexible processing of large message data. The time-consuming cycle of channel backlog perception and adjustment is increased from the original day or hour level to the minute level. There is no need to restart the upper-layer application in the whole process, no need to interrupt the online business operation, and no need for administrator intervention.

Based on the aforementioned embodiments, this embodiment of the present application provides a message processing device, the device includes each module included, each module includes each sub-module, each sub-module includes a unit, and can be implemented by a processor in an electronic device; Of course, it can also be realized by a specific logic circuit; in the process of implementation, the processor can be a central processing unit (Central Processing Unit, CPU), a microprocessor (Microprocessor Unit, MPU), a digital signal processor (DigitalSignal Process, DSP) Or Field Programmable Gate Array (Field Programmable Gate Array, FPGA), etc.

FIG. 5 is a schematic diagram of the composition and structure of the message processing device provided by the embodiment of the present application. As shown in FIG. 5, the device 500 includes:

An acquisition module 510, configured to acquire the real-time value of the consumption backlog of the publish-subscribe message system;

A determining module 520, configured to determine that message congestion occurs in the publish-subscribe message system based on the real-time value of the consumption backlog;

An adjustment module 530, configured to adjust system parameters of the publish-subscribe messaging system;

The message processing module 540 is configured to complete message processing based on the adjusted system parameters.

In some embodiments, the determination module 520 includes a first acquisition submodule, a first determination submodule and a second determination submodule, wherein the first acquisition submodule is used to acquire a preset consumption backlog threshold; The first determining submodule is configured to determine that the consumption backlog real-time value is greater than the consumption backlog threshold; the second determining submodule is configured to determine that message congestion occurs in the publish-subscribe message system.

In some embodiments, the first acquisition submodule includes a first acquisition unit, a second acquisition unit, and a multiplication unit, wherein the first acquisition unit is used to acquire the output of the publish-subscribe messaging system per unit time The number of messages; the second acquisition unit is configured to acquire a ratio threshold corresponding to the number of production messages; the multiplication unit is configured to multiply the number of production messages by the ratio threshold to obtain the consumption backlog threshold .

In some embodiments, the first obtaining submodule is further configured to determine the consumption backlog threshold based on the message processing capability of the publish-subscribe message system.

In some embodiments, the determination module 520 further includes a second acquisition submodule, a first subtraction submodule, a division submodule, a third acquisition submodule and a third determination submodule, wherein the second acquisition submodule , used to obtain the first consumption backlog value and the second consumption backlog value respectively at intervals of unit time; the first subtraction submodule is used to subtract the second consumption backlog value from the first consumption backlog value to obtain the consumption The backlog difference; the division submodule is used to divide the consumption backlog difference by the unit time to obtain the consumption backlog slope value; the third acquisition submodule is used to obtain the consumption backlog slope threshold; the The third determining submodule is configured to determine that at least two consecutive consumption backlog slope values are greater than the consumption backlog slope threshold.

In some embodiments, the adjustment module 530 includes a first increase submodule, a second increase submodule and a decrease submodule, wherein the first increase submodule is used to determine that the number of consumers is less than the In the case of the number of partitions, increase the number of consumers; the second increasing submodule is used to increase the number of partitions and the number of consumers when it is determined that the number of consumers is equal to the number of partitions Quantity; the reducing submodule is used to reduce the production frequency of the producer under the condition that the quantity of the producer, the quantity of the consumer and the quantity of the partition are determined to be unchanged.

In some embodiments, the acquisition module 510 includes a fourth acquisition submodule and a second subtraction submodule, wherein the fourth acquisition submodule is used to acquire the number of produced messages and the number of already generated messages of the publish-subscribe message system in real time The number of read messages; the second subtraction submodule is used to subtract the number of read messages from the number of produced messages to obtain the real-time value of the consumption backlog.

The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solutions of the embodiments of the present application or the part that contributes to the related technologies can be embodied in the form of software products. The computer software products are stored in a storage medium and include several instructions to make An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) executes all or part of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), magnetic disk or optical disk. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the message processing method provided in the foregoing embodiments are implemented.

Correspondingly, the embodiment of the present application provides an electronic device. FIG. 6 is a schematic diagram of a hardware entity of the electronic device provided in the embodiment of the present application. As shown in FIG. 6 , the hardware entity of the device 600 includes: a memory 601 and a processing The memory 601 stores a computer program that can run on the processor 602, and the processor 602 implements the steps in the message processing method provided in the above-mentioned embodiments when executing the program.

The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data and Video communication data) can be realized by flash memory (FLASH) or random access memory (Random Access Memory, RAM).

It should be pointed out here that: the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to those of the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be understood that reference throughout the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Thus, appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not be used in the embodiments of the present application. The implementation process constitutes any limitation. The serial numbers of the above embodiments of the present application are for description only, and do not represent the advantages and disadvantages of the embodiments.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or May be integrated into another system, or some features may be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms of.

The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be used as a single unit, or two or more units can be integrated into one unit; the above-mentioned integration The unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

Those of ordinary skill in the art can understand that all or part of the steps to realize the above method embodiments can be completed by hardware related to program instructions, and the aforementioned programs can be stored in computer-readable storage media. When the program is executed, the execution includes The steps of the foregoing method embodiments; and the foregoing storage media include: removable storage devices, read-only memory (Read Only Memory, ROM), magnetic disks or optical disks and other media that can store program codes.

Alternatively, if the above-mentioned integrated units of the present application are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solutions of the embodiments of the present application or the part that contributes to the related technologies can be embodied in the form of software products. The computer software products are stored in a storage medium and include several instructions to make An electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) executes all or part of the methods described in various embodiments of the present application. The aforementioned storage medium includes various media capable of storing program codes such as removable storage devices, ROMs, magnetic disks or optical disks.

The methods disclosed in several method embodiments provided in this application can be combined arbitrarily to obtain new method embodiments under the condition of no conflict.

The features disclosed in several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above is only the embodiment of the present application, but the scope of protection of the present application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, and should covered within the scope of protection of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (10)

1. A message processing method, said method comprising: Obtain the real-time value of the consumption backlog of the publish-subscribe message system; Based on the real-time value of the consumption backlog, it is determined that message congestion occurs in the publish-subscribe message system; Adjust system parameters of the publish-subscribe message system; Complete message processing based on the adjusted system parameters. 2. The method according to claim 1, wherein determining that message congestion occurs in the publish-subscribe message system based on the real-time value of the consumption backlog comprises: Obtain the preset consumption backlog threshold; determining that the consumption backlog real-time value is greater than the consumption backlog threshold; It is determined that message congestion occurs in the publish-subscribe message system. 3. The method according to claim 2, said obtaining a preset consumption backlog threshold, comprising: Obtain the number of production messages of the publish-subscribe message system described in the unit time; Obtain a ratio threshold corresponding to the number of production messages; The consumption backlog threshold is obtained by multiplying the production message quantity by the ratio threshold. 4. The method according to claim 2, said acquiring a preset consumption backlog threshold comprises: The consumption backlog threshold is determined based on the message processing capability of the publish-subscribe message system. 5. The method according to claim 2, before determining that message congestion occurs in the publish-subscribe message system, the method further comprises: Acquiring the first consumption backlog value and the second consumption backlog value respectively at intervals of unit time; subtracting the second consumption backlog value from the first consumption backlog value to obtain a consumption backlog difference; Divide the consumption backlog difference by the unit time to obtain the consumption backlog slope value; Obtain the consumption backlog slope threshold; It is determined that at least two consecutive values of the consumption backlog slope are greater than the consumption backlog slope threshold. 6. The method according to claim 1, wherein the system parameters at least include the number of consumers, the number of partitions and the number of producers, and the adjustment of the system parameters of the publish-subscribe message system includes: If it is determined that the number of consumers is less than the number of partitions, increasing the number of consumers; or, In the case of determining that the number of consumers is equal to the number of partitions, simultaneously increasing the number of partitions and the number of consumers; or, When it is determined that the number of producers, the number of consumers and the number of partitions remain unchanged, the production frequency of the producers is reduced. 7. The method according to claim 1, said obtaining the consumption backlog real-time value of said publish-subscribe messaging system, comprising: Obtaining the number of produced messages and the number of read messages of the publish-subscribe message system in real time; The real-time value of the consumption backlog is obtained by subtracting the number of read messages from the number of produced messages. 8. A message processing device, said device comprising: An acquisition module, configured to acquire the real-time value of the consumption backlog of the publish-subscribe message system; A determining module, configured to determine that message congestion occurs in the publish-subscribe message system based on the real-time value of the consumption backlog; An adjustment module, configured to adjust system parameters of the publish-subscribe messaging system; The message processing module is configured to complete message processing based on the adjusted system parameters. 9. An electronic device, comprising a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the method according to any one of claims 1 to 7 when executing the program A step of. 10. A storage medium, storing executable instructions for causing a processor to implement the steps in the method according to any one of claims 1 to 7 when executed.

Images