CN113138860B - Message queue management method and device - Google Patents
Message queue management method and device Download PDFInfo
- Publication number
- CN113138860B CN113138860B CN202010053387.2A CN202010053387A CN113138860B CN 113138860 B CN113138860 B CN 113138860B CN 202010053387 A CN202010053387 A CN 202010053387A CN 113138860 B CN113138860 B CN 113138860B
- Authority
- CN
- China
- Prior art keywords
- message
- queue
- message queue
- index
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title claims description 41
- 238000012545 processing Methods 0.000 claims abstract description 89
- 238000000034 method Methods 0.000 claims abstract description 41
- 238000003860 storage Methods 0.000 claims abstract description 4
- 238000004891 communication Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 9
- 230000009467 reduction Effects 0.000 claims description 7
- 238000004519 manufacturing process Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000008602 contraction Effects 0.000 description 27
- 238000010586 diagram Methods 0.000 description 10
- 238000011156 evaluation Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000013468 resource allocation Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000013439 planning Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The embodiment of the application discloses a method, a device, electronic equipment and a storage medium for managing a message queue, which relate to the field of electronic information and comprise the following steps: dynamically acquiring the number of messages contained in each message queue, and calculating the message enqueue index and the message dequeue index of each message queue in real time; calculating the message processing time length of each message queue according to the message enqueuing index and the message dequeuing index of each message queue; judging whether each message queue needs to execute capacity adjustment according to the message processing time length of each message queue and a preset time length processing threshold value, and executing capacity adjustment processing for the message queue with the judging result being yes. Therefore, the capacity of the message queue can be dynamically adjusted in real time, hysteresis is avoided, and the method can better adapt to service requirements.
Description
Technical Field
The application relates to the field of electronic information, in particular to a method and a device for managing a message queue.
Background
The use of a message queuing mechanism in a system to handle asynchronous service requests can greatly relieve the concurrency pressure of the system. But with the consequent capacity management problem of the message queues. For example, the message queues of different topics bear different services, and the corresponding number of consumer instances is different. Also, since the arrival time of each service request is random, and the time of processing in the consumer is also random, the queuing state of the entire message queue is in dynamic change, which results in very difficult capacity management of the message queue.
In the prior art, there is often hysteresis in capacity management of a message queue, for example, the capacity expansion process is performed when the message backlog in the message queue is severe, thereby causing the traffic to be affected. Thus, the existing management mode of the message queue has hysteresis, and the capacity of the message queue cannot be dynamically adjusted.
Disclosure of Invention
In view of the foregoing, the present application has been made to provide a method and apparatus for managing a message queue that overcomes or at least partially solves the foregoing problems.
According to one aspect of the present application, there is provided a method of managing a message queue, comprising:
dynamically acquiring the number of messages contained in each message queue, and calculating the message enqueue index and the message dequeue index of each message queue in real time;
calculating the message processing time length of each message queue according to the message enqueuing index and the message dequeuing index of each message queue;
judging whether each message queue needs to execute capacity adjustment according to the message processing time length of each message queue and a preset time length processing threshold value, and executing capacity adjustment processing for the message queue with the judging result being yes.
According to another aspect of the present application, there is provided a management apparatus for a message queue, comprising:
the acquisition module is suitable for dynamically acquiring the number of the messages contained in each message queue, and calculating the message enqueue index and the message dequeue index of each message queue in real time;
the calculation module is suitable for calculating the message processing duration of each message queue according to the message enqueuing index and the message dequeuing index of each message queue;
the adjustment module is suitable for judging whether each message queue needs to execute capacity adjustment according to the message processing time length of each message queue and a preset time length processing threshold value, and executing capacity adjustment processing for the message queue with the judging result being yes.
According to still another aspect of the present application, there is provided an electronic apparatus including: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;
the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform operations corresponding to the method for managing a message queue as described above.
According to still another aspect of the present application, there is provided a computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the method for managing a message queue as described above.
In the method and the device for managing the message queues, the message enqueuing index and the message dequeuing index of each message queue can be calculated in real time, so that the message processing time length of each message queue is calculated, accordingly, whether each message queue needs to execute capacity adjustment or not is judged according to the message processing time length of each message queue and the preset time length processing threshold, and capacity adjustment processing is executed for the message queue with the judged result. Therefore, the capacity of the message queue can be dynamically adjusted in real time, hysteresis is avoided, and the method can better adapt to service requirements.
The foregoing description is only an overview of the present application, and is intended to be implemented in accordance with the teachings of the present application in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present application more readily apparent.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
fig. 1 is a flowchart of a method for managing a message queue according to a first embodiment of the present application;
FIG. 2 is a schematic diagram showing the structure of a management system for performing the method of managing message queues in the present application;
fig. 3 is a block diagram of a message queue management apparatus according to a third embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application;
fig. 5 shows a specific structural diagram of the capacity evaluator;
FIG. 6 shows a processing diagram of a message router;
fig. 7 shows the mapping relationship between each message queue and the producer and consumer.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Example 1
Fig. 1 is a flowchart of a method for managing a message queue according to an embodiment of the present application.
As shown in fig. 1, the method includes:
step S110: dynamically acquiring the number of messages contained in each message queue, and calculating the message enqueue index and the message dequeue index of each message queue in real time.
The message queue in this embodiment may be a message queue for storing various service data. For example, in one alternative implementation, the message queue is a time series message queue for storing a plurality of messages having time series dependencies. And, there is fixed mapping relation between the consuming end and the producing end of each time sequence message queue, in other words, the messages generated by the same message producing end are fixed to be consumed by the same consuming end.
When the number of messages contained in each message queue is dynamically obtained, the number of messages contained in the message queue can be obtained once at preset intervals, or the number of messages contained in the message queue can be obtained once when the number of messages changes (such as is increased or decreased). By dynamically acquiring the number of messages contained in the message queues, the message enqueue index and the message dequeue index of each message queue can be calculated.
Where the message enqueue index is used to indicate the rate at which messages in the message queue are added (i.e., enqueued), for example, the message enqueue index for each message queue is determined based on the total amount of message enqueue per unit time. The more messages enqueued per unit time, the higher the message rate of increase, which may require a capacity expansion process.
Message dequeue index is used for the rate at which messages in a message queue are reduced (i.e., message dequeued), e.g., the message dequeue index for each message queue is determined based on the total number of message dequeues per unit time and the number of consumers. The more messages dequeued per unit time, the higher the message reduction rate, which may require a scaling process. In practical situations, the message enqueuing index and the message dequeuing index need to be compared to determine whether the resource configuration of the message queue is reasonable.
Step S120: and calculating the message processing time length of each message queue according to the message enqueuing index and the message dequeuing index of each message queue.
Wherein the message processing duration of each message queue is used to reflect the message processing efficiency of the message queue. Specifically, the message processing duration includes various phases and various forms of processing durations, including, for example: average number of queued messages, average queue length, and/or average waiting time for each message queue. Accordingly, in this step, an average number of queued messages, an average queue length, and/or an average waiting time for each message queue are calculated according to the message enqueue index and the message dequeue index for each message queue.
Step S130: judging whether each message queue needs to execute capacity adjustment according to the message processing time length of each message queue and a preset time length processing threshold value, and executing capacity adjustment processing for the message queue with the judging result being yes.
Specifically, the message processing efficiency of the message queue can be determined according to the message processing duration, and then whether the current processing efficiency can meet the service requirement is judged, so that whether the capacity of the queue needs to be adjusted is determined. In implementation, the service waiting time of the message queues corresponding to different service data is considered to be different, so that the preset duration processing threshold value comprises a plurality of queue processing threshold values determined according to the service attribute of the message queues and/or the historical processing duration of the message queues. Correspondingly, for each message queue, determining a queue processing threshold corresponding to the message queue according to the service attribute and/or the historical processing time length of the message queue, and judging whether the message queue needs to execute capacity adjustment according to the message processing time length of the message queue and a comparison result between the queue processing thresholds.
The capacity adjustment configuration information is determined according to the message queue with the judging result being yes; and adjusting the corresponding message queue according to the capacity adjustment configuration information, and recording the message routing information in a message routing file so as to enable a consuming end and a production end corresponding to the message queue to route the message according to the message routing file. Wherein the capacity adjustment configuration information includes: capacity adjustment type and capacity adjustment index; the capacity adjustment type comprises a capacity expansion type and a capacity reduction type, and the capacity adjustment index comprises a capacity expansion quantity and/or a capacity reduction quantity.
Therefore, in the management method of the message queues provided by the application, the message enqueuing index and the message dequeuing index of each message queue can be calculated in real time, so that the message processing time length of each message queue is calculated, and accordingly, whether each message queue needs to execute capacity adjustment or not is judged according to the message processing time length of each message queue and a preset time length processing threshold value, and capacity adjustment processing is executed for the message queue with the judged result. Therefore, the capacity of the message queue can be dynamically adjusted in real time, hysteresis is avoided, and the method can better adapt to service requirements.
Example two
In order to facilitate understanding, the second embodiment of the present application provides a method for managing a message queue, so as to describe in detail specific implementation details of each step in the first embodiment:
the use of message queuing mechanisms in a system to handle asynchronous service requests can greatly relieve the concurrency pressure of the system. But with the problem of resource allocation of message queues. For example, message queues of different topics bear different services, and the number of corresponding consumer instances (consumers are consumer ends, refer to network terminals for consuming messages) is also different. The message queues of the same topic typically store the same type of service request. However, since the arrival time of each service request is random, and the processing time in the consumer is also random, the queuing state of the entire message queue is dynamically changed, which makes it very difficult to allocate resources to the message queue. Moreover, in some complex service scenarios, the processing of certain services is dependent on the front and back, and the corresponding message processing must also ensure strict sequence, namely a so-called time sequence relationship. For example, a user transacting a power-on service, a power-off service (re-power-off), and a power-on service (re-power-off) may ultimately produce completely different results. If the consumer is expanded at will, the message processing sequence is disordered, so that the business handling result is not in line with the expectations of the user.
Therefore, in the existing mode, after the capacity management of the message queue or the consumer is performed, the one-to-one correspondence between the message queue and the consumer in the consumer needs to be manually modified, and the message forwarding relationship from all producers to the message queue needs to be synchronously modified. The producer is the generating end of the message queue, and refers to various network devices for generating messages.
The above-mentioned manner has at least the following drawbacks: the capacity management cost is high and the efficiency is low; the management of the message queue capacity needs to be manually participated, and the capacity can be expanded only when message backlog occurs, so that the method has hysteresis. When the number of the message queues is increased in the time sequence message queues, the forwarding relation of the message queues corresponding to the producer and the mapping relation between the consumer threads and the message queues are required to be modified, and the expansion once involves a plurality of operation links such as modification, compiling, packing, code release and the like, so that the time cost is high, the capacity management is lagged, and the efficiency is low. Moreover, there is hysteresis in capacity management: when the traffic volume is in a peak, the capacity of the message queue cannot be timely expanded, so that the traffic is influenced; when the business has a valley period, the capacity of the message queue cannot be reduced, so that the resource is wasted.
In order to solve the problems, in the application, queuing characteristics of message queues with different topics are calculated by adopting a queuing model, and an optimal resource allocation mode of consumers is planned: calculating the time interval of service requests reaching the message queue and the average service rate of the messages in the queue according to the historical data, calculating the queuing condition of each queue under the current resource allocation condition according to a queuing model, and planning the optimal resource allocation mode according to the queuing condition, so as to realize the accuracy of the expansion and contraction capacity decision process. And then, the capacity management executor schedules the resource management platform to automatically allocate or recycle corresponding resources, and the producer, the message queue and the consumer can update the routing information in real time by automatically configuring the message routing information, so that the message time sequence is ensured. Therefore, the full-automatic dynamic capacity expansion and contraction can be realized, the effect of manual intervention is not needed, the labor cost is greatly reduced, and meanwhile, the service waiting or the resource waste caused by untimely capacity expansion and contraction is avoided.
In this embodiment, dynamic management of the message queue is achieved through interaction among the 4 working modules. The 4 working modules are an information statistics module, a capacity evaluator, a capacity management executor and a message router respectively. Fig. 2 shows a schematic configuration diagram of a management system for performing the management method of message queues in the present application. As shown in fig. 2, the management system specifically includes: information statistics module, capacity evaluator, capacity management executor and message router. The information statistics module is used for counting the message arrival rate and the message service rate of each message queue. The message arrival rate is the message enqueue index, and the message service rate is the message dequeue index. The capacity evaluator is used for evaluating the message processing time of each message queue, and specifically comprises an evaluation index calculator, a threshold setting module and a decision maker. The capacity management executor is used for judging whether each message queue needs to execute capacity adjustment according to the message processing time length of each message queue and the time length processing threshold value set by the threshold value setting module, determining capacity adjustment configuration information according to the message queue with the judging result being yes, and adjusting the corresponding message queue according to the capacity adjustment configuration information. The message router is used for recording message routing information in a message routing file after the message queue is adjusted, so that a consumption end and a production end corresponding to the message queue can perform message routing according to the message routing file.
Specific implementation details of each module are described in detail below:
1. information statistics module
The information statistics module is used for obtaining information quantity from the information queue, calculating information arrival rate (namely information enqueuing index) and information service rate (namely information dequeuing index), and sending the information arrival rate (namely information dequeuing index) and the information service rate to the capacity evaluator.
Specifically, the operation mode of the message queue with the opened theme is abstracted into a queuing model. For example, the message producer opens a first message queue and a second message queue with two topics being a first topic and a second topic, respectively, where the first message queue corresponds to a first consumer and the second message queue corresponds to a second consumer. In particular, message queues for various topics may be abstracted into queuing models, such as a top-up queuing model, an order queuing model, and the like. The resource allocation of the whole system is abstracted into resource competition among a plurality of queuing models and a resource planning problem of the consumer instance inside the queuing models.
Thus, the queuing model is selected to calculate the message queue queuing characteristics for each topic. Firstly, according to the basic index of the historical data statistics queuing model:
2. capacity evaluator
The capacity evaluator is used for evaluating whether the current capacity of the message queue can meet the requirement or not and whether the capacity expansion and contraction are needed or not. Fig. 5 shows a specific structural diagram of the capacity evaluator. Specifically, the capacity evaluator further includes: an evaluation index calculator, a threshold setting module and a decision maker. Wherein, the evaluation index calculator is used for confirming each evaluation index, and the evaluation index includes: average number of queued messages, average queue length, average waiting time, and average stay time. The threshold setting module is used for determining the tolerable longest residence time according to the business attribute and experience statistics. The decision maker is used for determining whether to expand and contract the capacity and how to expand and contract the capacity.
Taking the recharging message queue as an example, the topic queue is set to correspond to c consumer instances, and as all consumer instances are deployed on the same configured host by the same code, all consumers can be considered to be mutually independent and have the same average service rate mu. According to the queuing model of the multiple service desks, when the number of messages in the system is n under the balance condition:
the service rate of the queue is as follows
The service strength of the queue is ρ=λ/cμ; only when ρ < 1 is not queued, the present application discusses the case when ρ < 1.
QueuesThe probability of idle is:wherein k is a variable, and the values are sequentially 0 to c-1.
Average number of queued messages in queue:
average queue length in the system (including the number of messages being serviced and waiting for service in the queue):
the average waiting time (time to queue in the queue) and the residence time (time from enqueue to completion of processing by the consumer) of the message are found by the Little formula:
thus, the average waiting time and residence time of the service request of each topic queue under the current resource allocation can be obtained.
The threshold setting module is used for setting a duration processing threshold. Message queues of different topics carry different types of service requests, and thus the stay time requirements for the different queues are also different. The queue stay time requirements are divided into two steps, 0-0.5 seconds and 0.5-4 seconds, according to the statistics in the industry and the internal characteristics of the service. Queues with high service importance levels are required to meet the first-gear requirements, and queues with low service importance levels are required to meet the second-gear requirements.
In addition, the decision maker is used for expanding the queue according to the queue queuing characteristic and the stay time requirement when the current stay time of the queue exceeds the specified time requirement. In addition, how consumer instances should be expanded is calculated for the purpose of maximizing overall traffic.
For queuing model systems, in steady state conditions, the expected value of total cost per unit time is z=c s ×c+c w ×L (1)
Where c is the number of consumer instances, is an unknown quantity, c s Is the cost per unit time per consumer instance, c w The cost per unit of system residence time for each service request can be estimated from empirical values. L is the average number of messages in the system or the average number of messages waiting in the queue (all of which vary with the value of c). So z is a function of c, z (c), now solving for the optimal solution c * So that z (c) * ) Minimum. Since c is an integer only and z (c) is not a continuous function, a marginal analysis method is employed.
According to z (c) * ) Is the smallest feature, has
Can be obtained according to the formulas (1) (2)
The values of L at c=1, 2,3 were determined in sequence and the difference between two adjacent values of L was made becauseIs a known number, and c can be determined according to which inequality interval the number falls within * 。
3. Capacity management executor
The capacity management executor is responsible for receiving the expansion and contraction command (i.e. capacity adjustment configuration information) output by the capacity evaluator, and the expansion and contraction command contains detailed information such as the expansion and contraction of a consumer, the expansion and contraction of a message queue, the number and the theme of the specific operation object, and the expansion and contraction form is as shown in table 1:
TABLE 1
| Operation object | Capacity management actions | Theme | Quantity of |
| Consumer Z | Expansion/contraction of volume | X | W |
| Message queue Y | Expansion/contraction of volume | X | W |
The capacity management executor distributes corresponding resource information automatically by the resource management platform according to the list. After the resources are successfully allocated, the capacity management executor synchronizes the newly added resource information, such as configuration information of ips, ports and the like, into the message router. In specific implementation, the capacity evaluator determines whether expansion and contraction capacity is needed and the capacity value of the expansion and contraction capacity according to the capacity evaluation result, and generates an expansion and contraction capacity instruction; and the capacity management executor sends a capacity expansion and contraction request to the resource management platform according to the capacity expansion and contraction instruction so as to apply for capacity expansion and contraction and receive a capacity expansion and contraction processing result returned by the resource management platform. And the capacity management executor sends the expansion and contraction capacity information to the message router according to the expansion and contraction capacity processing result returned by the resource management platform so that the message router can update the route according to the expansion and contraction capacity information.
4. Message router
The message router is responsible for respectively recording the mapping relation of message forwarding between a producer (namely a production end) and a consumer (namely a consumption end) and the message queue, so that the producer and the consumer can update the forwarding path in time, and the message time sequence is ensured. Fig. 6 shows a schematic diagram of a message router process. Specifically, the message router needs to inform the producer and consumer of the newly added routing relationships, respectively.
The message router receives the configuration information sent by the capacity management executor, wherein the configuration information is the expansion and contraction capacity information, and the configuration information is stored specifically through a message routing file. The message router informs the consumer and producer of the newly added message routing information. The data storage mode of the message router is a memory type data structure, and a producer and a message producer can read the data in real time and dynamically load the data into effect. In other words, the message routing information recorded in the message routing file is stored in the memory, so that the producer and the messager can read the message routing information recorded in the message routing file from the memory in real time and dynamically load the message routing information recorded in the message routing file, thereby realizing real-time effectiveness. Thereby enabling real-time updating of the message path.
The consumer updates the number of consumers according to the number of the message queues, and the producer determines the forwarded message again according to the number of the message queues, and the relationship always ensures a one-to-one mapping relationship, namely, the business messages handled by the same user are always received by the same queue and are processed by the same consumer. Fig. 7 shows the mapping between each message queue and producer and consumer, with the dashed line portion being the newly added message queue and consumer. When a producer generates a message, the producer can dynamically determine a newly added message queue in real time by inquiring the message routing information recorded in the message routing file in the memory, and each consumer can determine which message queue to consume by inquiring the message routing information recorded in the message routing file in the memory, so that the fixed mapping relation between a consumer and a producer is ensured by the message routing information recorded in the message routing file in the memory.
Therefore, how to dynamically expand and contract the time sequence message queue with random arrival and random processing time of the service request is a technical difficulty to be overcome by the application. The application calculates the average stay time of the message in the queue by introducing the queuing model, so as to judge the expansion and contraction capacity requirement and the resource optimization scheme of the queue, and is further assisted by a capacity management executor and a message router, thereby achieving full-flow automatic expansion and contraction capacity. In the implementation, firstly, the service distribution of the historical information is counted, and the average arrival rate and the average service rate of the information are calculated. And then, calculating according to the queuing theory to obtain an operation efficiency evaluation index of the message queue, and evaluating the capacity water level of each subject message queue under the current resource configuration according to the operation efficiency evaluation index. And secondly, guiding resource optimization configuration by using a queuing theory, so that the resource utilization rate is the highest, and the traffic efficiency is maximized. Finally, the application decouples the message routing relation from the configuration file by introducing the message router module, thoroughly solves the problem that the current time sequence message queue can not be dynamically expanded and contracted, and greatly improves the efficiency of time sequence message queue capacity management.
In summary, the present application has at least the following advantages: according to the application, the queuing condition of the message queue is evaluated in advance by introducing the queuing model, and decision is not needed to be made when waiting for the queue to backlog, so that the hysteresis of capacity expansion is avoided. The capacity management is full-flow and automatic, so that the dynamic capacity expansion and contraction of the time sequence message queue is realized, the capacity expansion and contraction result is accurate and timely, the capacity expansion and contraction efficiency is improved, the pressure of operation and maintenance personnel is reduced, and the maintenance threshold is lowered.
Example III
Fig. 3 is a schematic structural diagram of a management device for message queues according to a third embodiment of the present application, which specifically includes:
the obtaining module 31 is adapted to dynamically obtain the number of messages contained in each message queue, and calculate the message enqueue index and the message dequeue index of each message queue in real time;
a calculation module 32 adapted to calculate a message processing duration of each message queue based on the message enqueue index and the message dequeue index of each message queue;
the adjustment module 33 is adapted to determine whether each message queue needs to perform capacity adjustment according to the message processing duration of each message queue and a preset duration processing threshold, and perform capacity adjustment processing for the message queue whose determination result is yes.
Optionally, the preset time length processing threshold includes a plurality of queue processing thresholds determined according to service attributes of the message queue and/or historical processing time length of the message queue; the adjustment module is specifically adapted to:
for each message queue, determining a queue processing threshold corresponding to the message queue according to the service attribute and/or the historical processing time length of the message queue, and judging whether the message queue needs to execute capacity adjustment according to the comparison result between the message processing time length of the message queue and the queue processing threshold.
Optionally, the adjustment module is specifically adapted to:
determining capacity adjustment configuration information aiming at a message queue with a judging result of yes;
and adjusting the corresponding message queue according to the capacity adjustment configuration information, and recording the message routing information in a message routing file so as to enable a consumer end and a production end corresponding to the message queue to route the message according to the message routing file.
Optionally, the capacity adjustment configuration information includes:
capacity adjustment type and capacity adjustment index; the capacity adjustment type comprises a capacity expansion type and a capacity reduction type, and the capacity adjustment index comprises a capacity expansion quantity and/or a capacity reduction quantity.
Optionally, the acquiring module is specifically adapted to:
determining the message enqueuing index of each message queue according to the total message enqueuing amount in unit time;
and determining the message dequeue index of each message queue according to the total message dequeue amount and the consumption end amount in unit time.
Optionally, the computing module is specifically adapted to:
and calculating the average queuing message number, the average queue length and/or the average waiting time of each message queue according to the message enqueuing index and the message dequeuing index of each message queue.
Optionally, the message queues are time sequence message queues, and a fixed mapping relationship is provided between a consuming end and a producing end of each time sequence message queue.
Example IV
A fourth embodiment of the present application provides a non-volatile computer storage medium, where at least one executable instruction is stored, where the computer executable instruction may perform the method for managing a message queue in any of the foregoing method embodiments. The executable instructions may be particularly useful for causing a processor to perform the operations corresponding to the method embodiments described above.
Example five
Fig. 4 shows a schematic structural diagram of an electronic device according to a fifth embodiment of the present application, and the specific embodiment of the present application is not limited to the specific implementation of the electronic device.
As shown in fig. 4, the electronic device may include: a processor 402, a communication interface (Communications Interface) 406, a memory 404, and a communication bus 408.
Wherein:
processor 402, communication interface 406, and memory 404 communicate with each other via communication bus 408.
A communication interface 406 for communicating with network elements of other devices, such as clients or other servers.
Processor 402 is configured to execute program 410, and may specifically perform relevant steps in the above-described method embodiment for managing a message queue.
In particular, program 410 may include program code including computer-operating instructions.
The processor 402 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the electronic device may be the same type of processor, such as one or more CPUs; but may also be different types of processors such as one or more CPUs and one or more ASICs.
Memory 404 for storing program 410. Memory 404 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 510 may be specifically configured to cause the processor 502 to perform the respective operations corresponding to the above-described method embodiments.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a construction of such a system is apparent from the description above. In addition, the present application is not directed to any particular programming language. It will be appreciated that the teachings of the present application described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present application.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the application, various features of the application are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed application requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this application.
Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.
Various component embodiments of the application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components in accordance with embodiments of the present application may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present application can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present application may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.
Claims (9)
1. A method of managing a message queue, comprising:
dynamically acquiring the number of messages contained in each message queue, and calculating the message enqueue index and the message dequeue index of each message queue in real time; the message enqueuing index is used for indicating the rate of message increase in the message queue, and the message dequeuing index is used for indicating the rate of message decrease in the message queue;
calculating the message processing time length of each message queue according to the message enqueuing index and the message dequeuing index of each message queue; the message processing duration of each message queue is used for reflecting the message processing efficiency of the message queue;
judging whether the current processing efficiency can meet the service requirement according to the message processing time length of each message queue and a preset time length processing threshold value, thereby judging whether each message queue needs to execute capacity adjustment, and executing capacity adjustment processing for the message queue with the judging result being yes;
wherein, calculating the message processing duration of each message queue according to the message enqueuing index and the message dequeuing index of each message queue includes:
message enqueuing index and message dequeuing according to each message queueCalculating an average queuing message number, an average queue length and/or an average waiting time of each message queue; wherein the average number of queued messages isAverage captain->Average waiting time is +.>Wherein P is 0 For the probability that the queue is idle,wherein k is a variable, and the values are sequentially 0 to c-1; where ρ is the service strength of the queue, ρ=λ/cμ, where λ is the average arrival rate of the message, +.>Mu is the average service rate of the message, +.>c is the number of consumer instances.
2. The method of claim 1, wherein the preset duration processing threshold comprises a plurality of queue processing thresholds determined according to traffic attributes of the message queue and/or historical processing durations of the message queue; judging whether each message queue needs to execute capacity adjustment according to the message processing time length of each message queue and a preset time length processing threshold value comprises the following steps:
for each message queue, determining a queue processing threshold corresponding to the message queue according to the service attribute and/or the historical processing time length of the message queue, and judging whether the message queue needs to execute capacity adjustment according to the comparison result between the message processing time length of the message queue and the queue processing threshold.
3. The method of claim 1, wherein the performing capacity adjustment processing for the message queue for which the determination result is yes comprises:
determining capacity adjustment configuration information aiming at a message queue with a judging result of yes;
and adjusting the corresponding message queue according to the capacity adjustment configuration information, and recording the message routing information in a message routing file so as to enable a consumer end and a production end corresponding to the message queue to route the message according to the message routing file.
4. The method of claim 3, wherein the capacity adjustment configuration information comprises:
capacity adjustment type and capacity adjustment index; the capacity adjustment type comprises a capacity expansion type and a capacity reduction type, and the capacity adjustment index comprises a capacity expansion quantity and/or a capacity reduction quantity.
5. The method of claim 1, wherein the calculating in real-time the message enqueue index and the message dequeue index for each message queue comprises:
determining the message enqueuing index of each message queue according to the total message enqueuing amount in unit time;
and determining the message dequeue index of each message queue according to the total message dequeue amount and the consumption end amount in unit time.
6. The method of any of claims 1-5, wherein the message queues are time-series message queues, and each time-series message queue has a fixed mapping relationship between a consuming end and a producing end.
7. A message queue management apparatus comprising:
the acquisition module is suitable for dynamically acquiring the number of the messages contained in each message queue, and calculating the message enqueue index and the message dequeue index of each message queue in real time; the message enqueuing index is used for indicating the rate of message increase in the message queue, and the message dequeuing index is used for indicating the rate of message decrease in the message queue;
the calculation module is suitable for calculating the message processing duration of each message queue according to the message enqueuing index and the message dequeuing index of each message queue; the message processing duration of each message queue is used for reflecting the message processing efficiency of the message queue;
the adjustment module is suitable for judging whether the current processing efficiency can meet the service requirement according to the message processing time length of each message queue and a preset time length processing threshold value, so as to judge whether each message queue needs to execute capacity adjustment, and executing capacity adjustment processing for the message queue with the judging result being yes;
the computing module is specifically configured to:
according to the message enqueuing index and the message dequeuing index of each message queue, calculating the average queuing message number, the average queue length and/or the average waiting time of each message queue; wherein the average number of queued messages isAverage captain->Average waiting time is +.>Wherein P is 0 For the probability that the queue is idle,wherein k is a variable, and the values are sequentially 0 to c-1; where ρ is the service strength of the queue, ρ=λ/cμ, where λ is the average arrival rate of the message, +.>Mu is the average service rate of the message, +.>c is the number of consumer instances.
8. An electronic device, comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;
the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform operations corresponding to the method for managing a message queue according to any one of claims 1 to 6.
9. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the method of managing message queues according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010053387.2A CN113138860B (en) | 2020-01-17 | 2020-01-17 | Message queue management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010053387.2A CN113138860B (en) | 2020-01-17 | 2020-01-17 | Message queue management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113138860A CN113138860A (en) | 2021-07-20 |
| CN113138860B true CN113138860B (en) | 2023-11-03 |
Family
ID=76809630
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010053387.2A Active CN113138860B (en) | 2020-01-17 | 2020-01-17 | Message queue management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113138860B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113687928B (en) * | 2021-08-03 | 2024-03-22 | 广州华多网络科技有限公司 | Message scheduling control method and corresponding device, equipment and medium thereof |
| CN113835905B (en) * | 2021-09-28 | 2023-09-05 | 北京奇艺世纪科技有限公司 | Message queue load balancing method and device, electronic equipment and medium |
| CN115225577B (en) * | 2022-09-20 | 2022-12-27 | 深圳市明源云科技有限公司 | Data processing control method and device, electronic equipment and readable storage medium |
| CN116112446A (en) * | 2022-11-30 | 2023-05-12 | 重庆紫光华山智安科技有限公司 | Delay feedback method based on message queue, computer equipment and storage medium |
| CN116166451B (en) * | 2022-12-09 | 2023-11-07 | 广州市玄武无线科技股份有限公司 | Dynamic adjustment method, system and device for topic number and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106453564A (en) * | 2016-10-18 | 2017-02-22 | 北京京东尚科信息技术有限公司 | Elastic cloud distributed massive request processing method, device and system |
| JP2017162059A (en) * | 2016-03-08 | 2017-09-14 | キヤノンマーケティングジャパン株式会社 | Information processing device, control method, and program |
| CN107547416A (en) * | 2016-06-28 | 2018-01-05 | 中兴通讯股份有限公司 | A kind of processing method and processing device of protocol massages |
| CN108574645A (en) * | 2017-03-14 | 2018-09-25 | 华为技术有限公司 | A kind of array dispatching method and device |
| CN108694107A (en) * | 2017-04-10 | 2018-10-23 | 北京京东尚科信息技术有限公司 | Backlog monitoring method, device, readable medium and the electronic equipment of message queue |
| CN109428901A (en) * | 2017-08-22 | 2019-03-05 | 中国电信股份有限公司 | Message treatment method and message processing apparatus |
| CN110175068A (en) * | 2019-04-16 | 2019-08-27 | 平安科技（深圳）有限公司 | Host number elastic telescopic method, apparatus and computer equipment in distributed system |
| CN110489225A (en) * | 2018-05-15 | 2019-11-22 | 中国移动通信集团浙江有限公司 | A kind of service expansion method, device and equipment based on message queue |
| CN110633151A (en) * | 2019-09-20 | 2019-12-31 | 北京小米移动软件有限公司 | Method, device and storage medium for balancing distributed message issuing cluster partitions |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9407556B2 (en) * | 2014-03-03 | 2016-08-02 | Viasat Inc. | Predictive delaycast feedback |
| CN107783721B (en) * | 2016-08-25 | 2020-09-08 | 华为技术有限公司 | Data processing method and physical machine |
| US10608961B2 (en) * | 2018-05-08 | 2020-03-31 | Salesforce.Com, Inc. | Techniques for handling message queues |
-
2020
- 2020-01-17 CN CN202010053387.2A patent/CN113138860B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2017162059A (en) * | 2016-03-08 | 2017-09-14 | キヤノンマーケティングジャパン株式会社 | Information processing device, control method, and program |
| CN107547416A (en) * | 2016-06-28 | 2018-01-05 | 中兴通讯股份有限公司 | A kind of processing method and processing device of protocol massages |
| CN106453564A (en) * | 2016-10-18 | 2017-02-22 | 北京京东尚科信息技术有限公司 | Elastic cloud distributed massive request processing method, device and system |
| CN108574645A (en) * | 2017-03-14 | 2018-09-25 | 华为技术有限公司 | A kind of array dispatching method and device |
| CN108694107A (en) * | 2017-04-10 | 2018-10-23 | 北京京东尚科信息技术有限公司 | Backlog monitoring method, device, readable medium and the electronic equipment of message queue |
| CN109428901A (en) * | 2017-08-22 | 2019-03-05 | 中国电信股份有限公司 | Message treatment method and message processing apparatus |
| CN110489225A (en) * | 2018-05-15 | 2019-11-22 | 中国移动通信集团浙江有限公司 | A kind of service expansion method, device and equipment based on message queue |
| CN110175068A (en) * | 2019-04-16 | 2019-08-27 | 平安科技（深圳）有限公司 | Host number elastic telescopic method, apparatus and computer equipment in distributed system |
| CN110633151A (en) * | 2019-09-20 | 2019-12-31 | 北京小米移动软件有限公司 | Method, device and storage medium for balancing distributed message issuing cluster partitions |
Non-Patent Citations (3)
| Title |
|---|
| A fast and resource-conscious MPI message queue mechanism for large-scale jobs;Judicael A. Zounmevo等;《Future Generation Computer Systems》;第30卷;第265-290页 * |
| 基于大数据架构的数据采集与存储系统的设计与实现;汤如;《中国优秀硕士学位论文全文数据库 信息科技辑》(第5期);I138-1085 * |
| 蚂蚁金服消息队列运维平台的设计与实现;韩旭;《中国优秀硕士学位论文全文数据库 信息科技辑》(第11期);I138-159 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113138860A (en) | 2021-07-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113138860B (en) | Message queue management method and device | |
| US10474504B2 (en) | Distributed node intra-group task scheduling method and system | |
| CN110825520B (en) | Cluster extremely-fast elastic telescoping method for realizing efficient resource utilization | |
| US20150295970A1 (en) | Method and device for augmenting and releasing capacity of computing resources in real-time stream computing system | |
| CN111400022A (en) | Resource scheduling method and device and electronic equipment | |
| CN105900064A (en) | Method and apparatus for scheduling data flow task | |
| CN110489217A (en) | A kind of method for scheduling task and system | |
| CN111381950A (en) | Task scheduling method and system based on multiple copies for edge computing environment | |
| CN109788315A (en) | Video transcoding method, apparatus and system | |
| CN107341041B (en) | Cloud task multidimensional constraint backfill scheduling method based on priority queue | |
| CN112148468B (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| CN113946431B (en) | Resource scheduling method, system, medium and computing device | |
| CN111506430A (en) | Method and device for data processing under multitasking and electronic equipment | |
| Glazebrook et al. | On the optimal allocation of service to impatient tasks | |
| CN108509280A (en) | A kind of Distributed Calculation cluster locality dispatching method based on push model | |
| US8028291B2 (en) | Method and computer program product for job selection and resource allocation of a massively parallel processor | |
| CN108491255B (en) | Self-service MapReduce data optimal distribution method and system | |
| CN108574645A (en) | A kind of array dispatching method and device | |
| CN103248622B (en) | A kind of Online Video QoS guarantee method of automatic telescopic and system | |
| CN109450803A (en) | Traffic scheduling method, device and system | |
| CN109992392A (en) | A kind of calculation resource disposition method, device and Resource Server | |
| CN105404554B (en) | Method and apparatus for Storm stream calculation frame | |
| CN116302453B (en) | Task scheduling method and device for quantum electronic hybrid platform | |
| Ogawa et al. | Cloud bursting approach based on predicting requests for business-critical web systems | |
| CN112156453B (en) | Example adaptive adjustment method, apparatus, computer readable storage medium and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |