CN116055401A

CN116055401A - Message processing method, device, equipment and storage medium

Info

Publication number: CN116055401A
Application number: CN202310193575.9A
Authority: CN
Inventors: 李志刚; 高锋
Original assignee: Lenovo Beijing Ltd
Current assignee: Lenovo Beijing Ltd
Priority date: 2023-02-28
Filing date: 2023-02-28
Publication date: 2023-05-02

Abstract

The application discloses a message processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring a consumption product compaction value of a publish-subscribe message system; determining that the publish-subscribe message system is subject to message congestion based on the consumption product compaction value; adjusting system parameters of the publish-subscribe message system; and finishing message processing based on the adjusted system parameters.

Description

Message processing method, device, equipment and storage medium

Technical Field

Embodiments of the present application relate to the field of big data processing, and relate to, but are not limited to, a method, an apparatus, a device, and a storage medium for processing a message.

Background

Distributed messaging systems are widely used for high performance data pipelines, stream analysis, data integration, and mission critical applications. There are scenarios in distributed messaging systems where traffic data volume rises or a large number of messages are bursty. Under the scene, the channel and consumption capacity reaches the upper limit, the capacity of the message system is concentrated in data forwarding and message caching, and the congestion condition is not fed back to the producer and the consumer, so that the congestion cannot be controlled in real time, the data extrusion of the production environment can be caused, and the service processing delay is caused.

Disclosure of Invention

In view of this, embodiments of the present application provide a method, an apparatus, a device, and a storage medium for processing a message.

The technical scheme of the embodiment of the application is realized as follows:

in a first aspect, an embodiment of the present application provides a message processing method, where the method includes:

acquiring a consumption product compaction value of a publish-subscribe message system;

determining that the publish-subscribe message system is subject to message congestion based on the consumption product compaction value;

adjusting system parameters of the publish-subscribe message system;

and finishing message processing based on the adjusted system parameters.

In a second aspect, an embodiment of the present application provides a message processing apparatus, including:

the acquisition module is used for acquiring a consumption product compaction value of the publish-subscribe message system;

a determining module, configured to determine, based on the consumption product compaction value, that a message congestion occurs in the publish-subscribe message system;

the adjusting module is used for adjusting the system parameters of the publishing and subscribing message system;

and the message processing module is used for completing message processing based on the adjusted system parameters.

In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program executable on the processor, and where the processor implements the method described above when executing the program.

In a fourth aspect, embodiments of the present application provide a storage medium storing executable instructions for implementing the above method when executed by a processor.

In the embodiment of the application, firstly, a consumption product compaction value of a publish-subscribe message system is obtained; then, based on the consumption product compaction value, determining that the message congestion occurs in the publish-subscribe message system; adjusting system parameters of the publish-subscribe message system; and finally, finishing message processing based on the adjusted system parameters. Therefore, the congestion control capability can be increased, and after the backlog trend generated by the message channel is determined through dynamic detection and congestion feedback, the system parameters of the message publishing and subscribing system are adjusted, so that the production and consumption conditions of the messages are effectively controlled, and the message congestion condition is relieved.

Drawings

FIG. 1A is a schematic diagram of a publish-subscribe messaging system according to an embodiment of the present application;

fig. 1B is a schematic implementation flow diagram of a message processing method according to an embodiment of the present application;

fig. 2 is a schematic implementation flow chart of a method for determining message congestion according to an embodiment of the present application;

fig. 3 is a schematic implementation flow chart of a method for determining message congestion according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a solution architecture for processing messages according to an embodiment of the present application;

fig. 5 is a schematic diagram of a composition structure of a message processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application.

Detailed Description

For the purposes, technical solutions and advantages of the embodiments of the present application to be more apparent, the specific technical solutions of the embodiments of the present application will be further described in detail below with reference to the accompanying drawings in the embodiments of the present application. The following examples are illustrative of the present application, but are not intended to limit the scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.

In the following description, the terms "first", "second", "third" and the like are merely used to distinguish similar objects and do not represent a specific ordering of the objects, it being understood that the "first", "second", "third" may be interchanged with a specific order or sequence, as permitted, to enable embodiments of the application described herein to be practiced otherwise than as illustrated or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

Fig. 1A is a schematic diagram of a publish-subscribe message system according to an embodiment of the present application, as shown in fig. 1A, which includes a producer 11, a message channel and message partition 12, and a consumer 13, wherein,

in a publish-subscribe messaging system, the Producer (Producer) 11 of a message is referred to as a publisher and the Consumer (Consumer) 13 is referred to as a subscriber. The message is persisted into a message channel (topic) and a message Partition (Partition) 12. Consumers can subscribe to one or more topics, consumers 11 can consume all data in the topics, the same piece of data can be consumed by multiple consumers, and the data cannot be deleted immediately after being consumed.

In a distributed messaging system, the producer and consumer themselves are unaware of each other. The producer does not know that consumer consumption is blocked, and the sending frequency of data cannot be reduced; consumers do not know that the consumption is not keeping up with production, and cannot flexibly adjust the consumption capacity.

An embodiment of the present application provides a message processing method, as shown in fig. 1B, including:

step S110, obtaining a consumption product compaction value of a publish-subscribe message system;

here, a publish-subscribe is a paradigm of messages that a producer of a message does not send directly to a particular consumer. Rather, published messages are divided into different categories without determining which subscribers may exist. Likewise, subscribers may express interest in one or more categories, receiving only messages of interest, without determining which publishers exist. For example, kafka is a high throughput distributed publish-subscribe messaging system that can handle all action flow data for consumers in a web site.

Consumption backlog (LAG), the difference between the messages retained in the message middleware server and the consumed messages, i.e., the message accumulation, is also referred to as consumption hysteresis. Lag has a positive and negative number and 0. The positive Lag number represents the backlog of kafka data, and the speed of feeding the data to the kafka is greater than the speed at which the data is consumed and is greater than the demand. Lag takes a negative number, which indicates that supply is smaller than demand. Lag takes a value of 0, indicating that the producer and consumer rates are substantially comparable. LAG in busy messaging systems is typically greater than 0 and remains at a relatively constant value, indicating that production and consumption are in balance.

In the implementation process, the consumption backlog value of the publish-subscribe message system can be obtained in real time, namely, the consumption backlog value which is currently used for representing the difference value between the production message and the consumption message is obtained.

Step S120, determining that the message congestion occurs in the publish-subscribe message system based on the consumption product compaction value;

here, since the consumption accumulation compaction value can be used to represent the difference between the production message and the consumption message, in the case that it is determined that the consumption accumulation real-time value is increased, it can be determined that the number of produced messages cannot be consumed in time, that is, the message congestion occurs in the publish-subscribe message system.

Step S130, adjusting system parameters of the publish-subscribe message system;

here, the system parameters of the publish-subscribe message system include at least the number of consumers, the number of partitions, and the number of producers. In the implementation process, under the condition that the message congestion of the publish-subscribe message system is determined, at least one system parameter can be adjusted so as to relieve the message congestion. For example, producers and consumers may adjust the ability of data to be sent and consumed based on congestion conditions to address congestion.

And step 140, completing message processing based on the adjusted system parameters.

In the implementation process, the message processing is completed based on the adjusted system parameters, so that the message congestion condition of the publish-subscribe message system can be effectively relieved.

In some embodiments, as shown in fig. 2, the above step S120 "determining that the publish-subscribe messaging system is congested based on the consumption product compaction value" may be implemented by:

step S210, acquiring a preset consumption backlog threshold;

here, the consumption backlog threshold may be determined based on the message processing capabilities of the publish-subscribe message system.

Step S220, determining that the consumption backlog compaction value is greater than the consumption backlog threshold;

in the implementation process, the consumption backlog value obtained in real time can be compared with the message backlog threshold value to determine whether the consumption backlog real-time value is larger than the consumption backlog threshold value. In the case that the consumption accumulation compaction value is determined to be greater than the consumption accumulation threshold, it may be determined that the message processing capability of the publish-subscribe message system cannot meet the current message production, for example, in the scenario that the traffic data volume rises or a large number of messages are bursty, the consumption accumulation compaction value is greater than the consumption accumulation threshold, and the channel and the consumption capability reach the upper limit, which may cause data extrusion of the production environment, resulting in traffic processing delay.

Step S230, determining that the publishing and subscribing information system has information congestion.

In the embodiment of the application, a preset consumption backlog threshold is firstly obtained; then determining that the consumption backlog value is greater than the consumption backlog threshold; and determining that the publishing and subscribing message system has message congestion. Therefore, based on the consumption backlog threshold, the consumption backlog value is determined to be larger than the consumption backlog threshold, and the message congestion of the publishing and subscribing message system is further effectively determined.

In some embodiments, the step S210 "obtaining the preset consumption backlog threshold" above may be implemented by the following steps:

step 211, obtaining the number of production messages of the publish-subscribe message system in unit time;

here, the number of production messages per unit time (n minutes) can be acquired.

Step 212, obtaining a proportional threshold corresponding to the number of the production messages;

here, a proportional threshold (m%) corresponding to the number of production messages per unit time (n minutes) may be acquired.

In the implementation process, step 211 may be performed first and then step 212 may be performed, step 212 may be performed first and then step 211 may be performed, or step 211 and step 212 may be performed simultaneously, that is, the execution sequence of step 211 and step 212 is not required.

And step 213, multiplying the number of the production messages by the proportion threshold value to obtain the consumption backlog threshold value.

Here, the consumption backlog threshold may be obtained by multiplying the number of production messages by a proportional threshold.

In practice, the occurrence of message system congestion may be determined using the following equation (1):

in the embodiment of the application, firstly, the number of production messages of the publish-subscribe message system in unit time is obtained; acquiring a proportional threshold corresponding to the number of the production messages; and then multiplying the number of the production messages by the proportion threshold value to obtain the consumption backlog threshold value. In this way, an effective consumption backlog threshold may be determined based on the number of production messages per unit time of the publish-subscribe message system.

In some embodiments, the step S210 "obtaining the preset consumption backlog threshold" above may be implemented by the following procedure:

the consumption backlog threshold is determined based on a message processing capability of a publish-subscribe message system.

Here, in case that the LAG value is determined to be higher than the configuration value (N), the message system congestion may be determined.

In practice, the occurrence of message system congestion may be determined using the following equation (2):

LAG > N (2)；

wherein N is a configuration value, namely a consumption backlog threshold.

In the embodiment of the application, the consumption backlog threshold value can be effectively determined based on the message processing capacity of the publish-subscribe message system.

In some embodiments, as shown in fig. 3, before "determining that the publish-subscribe message system has message congestion" in step S220, the method for determining that the publish-subscribe message system has message congestion further includes:

step S310, respectively obtaining a first consumption backlog value and a second consumption backlog value at intervals of unit time;

here, the unit time may be set according to the actual situation, and for example, the unit time may be set to one minute or one second.

In practice, LAG may be acquired at time x-1 _x-1 Acquiring LAG at time x _x I.e. obtaining the first consumption backlog LAG separately in unit time _x-1 And a second consumption backlog value LAG _x 。

Step S320, subtracting the second consumption backlog value from the first consumption backlog value to obtain a consumption backlog difference value;

in practice, the LAG may be applied _x Subtracting LAG _x-1 And obtaining the consumption backlog difference value.

Step S330, dividing the consumption backlog difference value by the unit time to obtain a consumption backlog slope value;

in the implementation process, the consumption backlog slope value can be obtained based on the following formula (3), and the real-time slope P is:

wherein, LAG _x LAG, LAG at time x _x-1 LAG, T at time x-1 _x Time value of x time, T _x-1 Is the time value of the x-1 moment.

Step S340, obtaining a consumption backlog slope threshold;

in implementations, the consumption backlog slope threshold may be determined based on the message processing capabilities of the subscription messaging system.

Step S350, determining that at least two continuous consumption backlog slope values are larger than the consumption backlog slope threshold.

In some embodiments, message system congestion may be determined if it is determined that the value of at least two consecutive P's is above the consumption backlog slope threshold.

In the embodiment of the application, a first consumption backlog value and a second consumption backlog value are respectively acquired at intervals; then subtracting the second consumption backlog value from the first consumption backlog value to obtain a consumption backlog difference value; dividing the consumption backlog difference value by the unit time to obtain a consumption backlog slope value; acquiring a consumption backlog slope threshold; and finally, determining that at least two continuous consumption backlog slope values are larger than the consumption backlog slope threshold value. In this way, it may be determined that the publish-subscribe message system is experiencing message congestion based on the consumption backlog slope value.

In some embodiments, the system parameters include at least the number of consumers, the number of partitions, and the number of producers, and the step S130 "adjusting the system parameters of the publish-subscribe message system" may be implemented by any of the following steps:

step 131, increasing the number of consumers in the case that the number of consumers is determined to be smaller than the number of partitions;

here, it is determined that the number of consumers is smaller than the number of partitions, and the number of producers is unchanged, the message congestion can be alleviated by increasing the number of consumers.

Step 132, in the case that the number of consumers is determined to be equal to the number of partitions, increasing the number of partitions and the number of consumers simultaneously;

here, it is determined that the number of consumers is equal to the number of partitions, and the number of producers is unchanged, the message congestion can be alleviated by increasing the number of partitions and the number of consumers at the same time.

For example, a message queue may include at least one partition, each partition corresponding to a working consumer (multiple partitions may share a consumer, and the number of partitions determines the upper limit of the working consumer), so increasing the partitions means expanding the upper limit of the consumer, thereby laying a foundation for subsequently increasing the consumer.

Under the condition of message congestion, the partition number is dynamically increased, and the consumers are dynamically informed, so that the consumers can increase the processing capacity and reduce the congestion by increasing more consumers according to the latest upper limit of the partition number.

Step 133, reducing the production frequency of the producer under the condition that the number of producers, the number of consumers and the number of partitions are unchanged.

Here, in the case where the consumer is limited to the resource cannot be increased any more, the message congestion can be relieved by reducing the production frequency of the producer.

For example, a producer may be notified of a message congestion condition, and for a producer that may adjust the transmission capacity (e.g., reduce the data acquisition period), the congestion problem may be addressed by way of transmission slowdown, such as from 100 transmissions per second, reducing 10 transmissions per second. This applies to scenarios where data may be sent at a reduced rate.

In the embodiment of the application, the number of consumers is increased under the condition that the number of consumers is determined to be smaller than the number of partitions; or, in the case where it is determined that the number of consumers is equal to the number of partitions, increasing the number of partitions and the number of consumers simultaneously; or, in the case where it is determined that the number of producers, the number of consumers, and the number of partitions are unchanged, reducing the production frequency of the producers. Thus, message congestion can be effectively relieved.

In some embodiments, the step S110 "obtaining the consumption product compaction value of the publish-subscribe message system" above may be implemented by:

step 111, acquiring the number of production messages and the number of read messages of the publish-subscribe message system in real time;

in an implementation, the production message number EndOffset and the already read message number CommitOffset may be obtained.

Step 112, subtracting the number of read messages from the number of production messages to obtain the consumption product compaction value.

In practice, the consumer compaction values may be obtained using the following equation (4):

Lag＝EndOffset-CommitOffset (4)；

where Lag represents the consumption backlog value, endOffset represents the number of production messages, commitOffset represents the number of messages that have been read.

In the embodiment of the application, firstly, the number of production messages and the number of read messages of the publish-subscribe message system are obtained in real time; and then subtracting the read message number from the production message number to obtain the consumption product compaction value. Thus, the consumption product compaction value can be obtained by the number of production messages and the number of read messages acquired in real time.

Distributed messaging systems are currently widely used for high performance data pipeline, stream analysis, data integration, and mission critical applications. Message transmission and reception generally includes the steps of:

step A, preparing a message channel;

in the message channel preparation phase, a message channel (partial) and a message Partition (Partition) may be created first.

Step B, sending data by the message;

in the message sending data stage, when the producer sends data, the message will send the message into the designated message partition according to the rule.

Step C, receiving data by the message;

in the message receiving data stage, the consumer subscribes to the message partition, and parallel receiving and processing of the message are realized.

Fig. 4 is a schematic diagram of a solution architecture for processing messages according to an embodiment of the present application, as shown in fig. 4, which includes an existing messaging system 41 and a congestion control service 42, wherein,

the congestion control service 42 extends based on the existing messaging system 41, providing congestion control capabilities, supporting the messaging path to form a closed loop.

Congestion control services 42 include message channel throughput and accumulation information collection service 421, data analysis and auto-adjustment service 422, and upper layer application service 423.

The message channel throughput and accumulation information collection service 421 is responsible for channel monitoring and basic information collection and is used for counting and measuring production, consumption throughput and congestion conditions in real time.

Data analysis and automatic adjustment service 422 notifies upper applications of the throughput and accumulation of message channels after data analysis according to policies. The sending quantity and the consumption capacity can be perceived and regulated, the partition quantity of the message channel can be automatically regulated, and the strategy regulation condition is notified to the upper layer application. The method supports automatic identification of multi-scene message backlog (a certain proportion of the number of messages is produced in unit time according to a threshold value, the number is absolute according to the threshold value, and the accumulation trend exceeds a certain slope), and under the condition that the message backlog is determined, an automatic adjustment strategy is started to adjust message channels (topic) and message partitions (Partition), so that the automatic adjustment strategy of the partitions is realized, and the number of the message partitions is dynamically increased.

The embodiment of the application provides three strategies for data analysis:

strategy 1: and taking the consumption backlog threshold value as a certain proportion of the number of the messages produced in unit time. For example, a certain proportion (m%) of the number of production messages per unit time (n minutes) may be obtained. And if the LAG value is higher than the consumption backlog threshold value, determining that the message system is congested.

LAG > n minutes the number of messages produced is m% (1).

Strategy 2: the consumption backlog threshold is determined based on the message handling capability of the publish-subscribe message system, i.e. in case it is determined that the LAG value is higher than the configuration value (N), a message system congestion is determined.

LAG > N (2)；

wherein N is a configuration value.

Strategy 3: in the event that the per-stack trend of the message exceeds a certain slope, a message system congestion is determined.

Acquiring a consumption backlog slope threshold value, and obtaining a real-time slope P based on the following formula (3):

In some embodiments, message system congestion may be determined if the value of P is determined to be higher than the consumption backlog slope threshold M consecutive times.

Here, the configuration and opening of the policy are configured in a threshold mode, and the configuration value is generally determined according to the service requirement by modifying the configuration mode to be up-regulated. The method comprises the steps of determining initial values of partitions in a business demand analysis stage, using monitoring data of three open strategies after operation, determining message congestion under any strategy, informing an upper layer application, and increasing the number of the partitions and the number of consumers.

Whether the three strategies are effective or not can be started through the configuration switch, and a plurality of strategies can be started simultaneously. For example, both policy 1 and policy 2 are on, and when any policy detected in the monitoring reaches the threshold, it will adjust the partition number and notify the upper layer application.

The upper layer application service 423 is configured to adjust the upper layer consumer number according to the policy.

In the embodiment of the application, the channel increases the congestion control capability, and the message sending channel forms a closed loop through dynamic detection and congestion feedback, so that the production and consumption conditions of the message are effectively controlled. Thus, after the message channel generates backlog trend, the channel capacity is automatically adjusted according to the strategy, and the elastic processing of large message data is supported. The time-consuming period for channel backlog sensing and adjustment is increased from the original day or hour level to the minute level. The whole process does not need to restart the upper layer application, interrupt the online service operation and intervening by an administrator.

Based on the foregoing embodiments, the embodiments of the present application provide a message processing apparatus, where the apparatus includes each module including each sub-module, and each sub-module includes a unit, and may be implemented by a processor in an electronic device; of course, the method can also be realized by a specific logic circuit; in practice, the processor may be a central processing unit (Central Processing Unit, CPU), microprocessor (Microprocessor Unit, MPU), digital signal processor (Digital Signal Process, DSP) or field programmable gate array (Field Programmable Gate Array, FPGA), etc.

Fig. 5 is a schematic diagram of a composition structure of a message processing apparatus according to an embodiment of the present application, as shown in fig. 5, the apparatus 500 includes:

an obtaining module 510, configured to obtain a consumption product compaction value of the publish-subscribe message system;

a determining module 520, configured to determine that the publish-subscribe messaging system has message congestion based on the consumption product compaction value;

an adjustment module 530, configured to adjust system parameters of the publish-subscribe message system;

message processing module 540 is configured to complete message processing based on the adjusted system parameters.

In some embodiments, the determining module 520 includes a first obtaining sub-module, a first determining sub-module, and a second determining sub-module, where the first obtaining sub-module is configured to obtain a preset consumption backlog threshold; the first determining submodule is used for determining that the consumption backlog compaction value is larger than the consumption backlog threshold value; and the second determining submodule is used for determining that the message congestion occurs in the publish-subscribe message system.

In some embodiments, the first obtaining submodule includes a first obtaining unit, a second obtaining unit and a multiplying unit, where the first obtaining unit is configured to obtain a number of production messages of the publish-subscribe message system in a unit time; the second obtaining unit is used for obtaining a proportional threshold corresponding to the number of the production messages; and the multiplication unit is used for multiplying the number of the production messages by the proportion threshold value to obtain the consumption backlog threshold value.

In some embodiments, the first acquisition sub-module is further configured to determine the consumption backlog threshold based on a message processing capability of the published subscription message system.

In some embodiments, the determining module 520 further includes a second obtaining sub-module, a first subtracting sub-module, a removing sub-module, a third obtaining sub-module, and a third determining sub-module, where the second obtaining sub-module is configured to obtain a first consumption backlog value and a second consumption backlog value separately in a unit time; the first subtracting submodule is used for subtracting the second consumption backlog value from the first consumption backlog value to obtain a consumption backlog difference value; the division submodule is used for dividing the consumption backlog difference value by the unit time to obtain a consumption backlog slope value; the third acquisition submodule is used for acquiring a consumption backlog slope threshold value; the third determination submodule is used for determining that at least two continuous consumption backlog slope values are larger than the consumption backlog slope threshold value.

In some embodiments, the adjustment module 530 includes a first increasing sub-module, a second increasing sub-module, and a decreasing sub-module, where the first increasing sub-module is configured to increase the number of consumers if the number of consumers is determined to be less than the number of partitions; the second adding sub-module is used for simultaneously adding the partition number and the consumer number under the condition that the consumer number is determined to be equal to the partition number; the reduction submodule is used for reducing the production frequency of the producer under the condition that the number of the producer, the number of the consumers and the number of the partitions are unchanged.

In some embodiments, the obtaining module 510 includes a fourth obtaining sub-module and a second subtracting sub-module, where the fourth obtaining sub-module is configured to obtain, in real time, a number of production messages and a number of read messages of the publish-subscribe message system; the second subtracting submodule is used for subtracting the read message number from the production message number to obtain the consumption product compaction value.

The description of the apparatus embodiments above is similar to that of the method embodiments above, with similar advantageous effects as the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be noted that, in the embodiment of the present application, if the method is implemented in the form of a software functional module, and sold or used as a separate product, the method may also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied essentially or in a part contributing to the related art in the form of a software product stored in a storage medium, including several instructions for causing an electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.

Accordingly, embodiments of the present application provide a storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the message processing method provided in the above embodiments.

Correspondingly, an electronic device is provided in the embodiment of the present application, fig. 6 is a schematic diagram of a hardware entity of the electronic device provided in the embodiment of the present application, and as shown in fig. 6, the hardware entity of the device 600 includes: comprising a memory 601 and a processor 602, said memory 601 storing a computer program executable on the processor 602, said processor 602 implementing the steps of the message processing method provided in the above embodiments when said program is executed.

The memory 601 is configured to store instructions and applications executable by the processor 602, and may also cache data (e.g., image data, audio data, voice communication data, and video communication data) to be processed or processed by the processor 602 and the modules in the electronic device 600, which may be implemented by a FLASH memory (FLASH) or a random access memory (Random Access Memory, RAM).

It should be noted here that: the description of the storage medium and apparatus embodiments above is similar to that of the method embodiments described above, with similar benefits as the method embodiments. For technical details not disclosed in the embodiments of the storage medium and the apparatus of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application. The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above described device embodiments are only illustrative, e.g. the division of the units is only one logical function division, and there may be other divisions in practice, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units; can be located in one place or distributed to a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.

Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, where the program, when executed, performs steps including the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read Only Memory (ROM), a magnetic disk or an optical disk, or the like, which can store program codes.

Alternatively, the integrated units described above may be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied essentially or in a part contributing to the related art in the form of a software product stored in a storage medium, including several instructions for causing an electronic device (which may be a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc.) to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The methods disclosed in the several method embodiments provided in the present application may be arbitrarily combined without collision to obtain a new method embodiment.

The features disclosed in the several product embodiments provided in the present application may be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or apparatus embodiments provided in the present application may be arbitrarily combined without conflict to obtain new method embodiments or apparatus embodiments.

The foregoing is merely an embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of message processing, the method comprising:

adjusting system parameters of the publish-subscribe message system;

and finishing message processing based on the adjusted system parameters.

2. The method of claim 1, the determining that the publish-subscribe messaging system is congested based on the consumption product compaction values, comprising:

acquiring a preset consumption backlog threshold;

determining that the consumption backlog value is greater than the consumption backlog threshold;

and determining that the publishing and subscribing message system has message congestion.

3. The method of claim 2, the obtaining a preset consumption backlog threshold comprising:

acquiring the number of production messages of the publish-subscribe message system in unit time;

acquiring a proportional threshold corresponding to the number of the production messages;

and multiplying the number of the production messages by the proportion threshold value to obtain the consumption backlog threshold value.

4. The method of claim 2, the obtaining a preset consumption backlog threshold comprising:

the consumption backlog threshold is determined based on the message processing capabilities of the published subscription message system.

5. The method of claim 2, prior to said determining that the publish-subscribe message system is congested, the method further comprising:

respectively obtaining a first consumption backlog value and a second consumption backlog value at intervals of unit time;

subtracting the second consumption backlog value from the first consumption backlog value to obtain a consumption backlog difference value;

dividing the consumption backlog difference value by the unit time to obtain a consumption backlog slope value;

acquiring a consumption backlog slope threshold;

determining that at least two consecutive consumer backlog slope values are greater than the consumer backlog slope threshold.

6. The method of claim 1, wherein the system parameters include at least a number of consumers, a number of partitions, and a number of producers, the adjusting the system parameters of the publish-subscribe messaging system comprising:

increasing the number of consumers in the event that the number of consumers is determined to be less than the number of partitions;

or,

in the case that the number of consumers is determined to be equal to the number of partitions, increasing the number of partitions and the number of consumers simultaneously;

or,

and reducing the production frequency of the producer under the condition that the number of the producer, the number of the consumers and the number of the partitions are unchanged.

7. The method of claim 1, the obtaining a consumption product compaction value for the publish-subscribe messaging system comprising:

acquiring the number of production messages and the number of read messages of the publish-subscribe message system in real time;

subtracting the number of read messages from the number of production messages to obtain the consumption product compaction value.

8. A message processing apparatus, the apparatus comprising:

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor implementing the steps of the method of any one of claims 1 to 7 when the program is executed.

10. A storage medium storing executable instructions for causing a processor to perform the steps of the method of any one of claims 1 to 7.