CN118488012A - Queue distribution method, queue distribution device, electronic equipment and storage medium - Google Patents

Queue distribution method, queue distribution device, electronic equipment and storage medium Download PDF

Info

Publication number
CN118488012A
CN118488012A CN202310139774.1A CN202310139774A CN118488012A CN 118488012 A CN118488012 A CN 118488012A CN 202310139774 A CN202310139774 A CN 202310139774A CN 118488012 A CN118488012 A CN 118488012A
Authority
CN
China
Prior art keywords
queue
consumer device
consumer
consumed
queues
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310139774.1A
Other languages
Chinese (zh)
Inventor
王康
高育俊
刘炜强
杨汉京
郭斯佳
张浪
陈定一
李连胜
温浩天
黄幼玲
朱振媚
何莹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SF Technology Co Ltd
Original Assignee
SF Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SF Technology Co Ltd filed Critical SF Technology Co Ltd
Priority to CN202310139774.1A priority Critical patent/CN118488012A/en
Publication of CN118488012A publication Critical patent/CN118488012A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • H04L47/52—Queue scheduling by attributing bandwidth to queues
    • H04L47/522—Dynamic queue service slot or variable bandwidth allocation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/70—Admission control; Resource allocation
    • H04L47/82—Miscellaneous aspects
    • H04L47/827—Aggregation of resource allocation or reservation requests
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer And Data Communications (AREA)

Abstract

The application discloses a queue allocation method, a queue allocation device, electronic equipment and a computer readable storage medium. Wherein the method comprises the following steps: counting the first message quantity to be consumed of the first consumer device according to all queues allocated to the first consumer device; determining a first target queue in all queues under the condition that the number of the first messages to be consumed reaches a preset first number threshold value, wherein the first target queue is: waiting for a reassigned queue; searching a second consumer device with the consumption capability of the first target queue in a consumer cluster where the first consumer device is located; and under the condition that the second consumer-end equipment is found, the first target queue is redistributed to the second consumer-end equipment. The scheme of the application can dynamically allocate the queues according to the consumption pressure of consumers, promote the consumption speed of the queues and ensure the overall consumption performance.

Description

Queue distribution method, queue distribution device, electronic equipment and storage medium
Technical Field
The present application relates to the field of message processing technologies, and in particular, to a queue allocation method, a queue allocation device, an electronic device, and a computer readable storage medium.
Background
Message middleware is a supportive software system that provides reliable synchronous or asynchronous message transmission for application systems in a network environment based on queue and message passing technology, and various message middleware products such as kafka and RocketMq are currently available on the market, and all distribute queues to consumers through respective distribution algorithms.
However, based on existing allocation algorithms, the relationship of the queue and consumer is not changed after the determination. Because the number of messages in the queues is dynamically changed, the situation that the number of messages in different queues is unbalanced may occur, so that the consumption pressure of some consumers is too high, and the overall consumption performance is affected.
Disclosure of Invention
The application provides a queue allocation method, a queue allocation device, electronic equipment and a computer readable storage medium, which can dynamically allocate the queues according to the consumption pressure of consumers, improve the consumption speed of the queues and ensure the overall consumption performance.
In a first aspect, the present application provides a queue allocation method, including:
Counting the number of first messages to be consumed of the first consumer device according to all queues allocated to the first consumer device;
under the condition that the number of the first messages to be consumed reaches a preset first number threshold value, determining a first target queue in all queues, wherein the first target queue is as follows: waiting for a reassigned queue;
Searching a second consumer device with the consumption capability of the first target queue in a consumer cluster where the first consumer device is located;
And in the case of finding the second consumer device, reassigning the first target queue to the second consumer device.
In a second aspect, the present application provides a queue allocation apparatus comprising:
A first statistics module, configured to count a first number of messages to be consumed by the first consumer device according to all queues allocated to the first consumer device;
The determining module is configured to determine a first target queue in all queues when the number of first messages to be consumed reaches a preset first number threshold, where the first target queue is: waiting for a reassigned queue;
the searching module is used for searching a second consumer device with the consumption capability of the first target queue in the consumer cluster where the first consumer device is located;
And the reassignment module is used for reassigning the first target queue to the second consumer device under the condition that the second consumer device is found.
In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method of the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the steps of the method of the first aspect.
In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by one or more processors, implements the steps of the method of the first aspect described above.
Compared with the prior art, the application has the beneficial effects that: the first quantity of messages to be consumed expresses a consumption pressure of the first consumer device. When the number of the first messages to be consumed reaches a preset first number threshold, it is known that the consumption pressure of the first consumer device is high, and at this time, a queue waiting to be reassigned, that is, a first target queue, can be determined from the queues assigned to the first consumer device. In order to alleviate the consumption pressure of the first consumer device, a second consumer device with the consumption capability of the first target queue can be searched in the consumer cluster where the first consumer device is located, and the first target queue is redistributed to the second consumer device when the second consumer device is searched. Thus, the queue allocated to the first consumer device can be reduced, so that the originally larger consumption pressure of the first consumer device can be relieved. The process realizes dynamic allocation of the queues according to the consumption pressure of consumers, helps to promote the consumption speed of the queues, and can ensure the overall consumption performance.
It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of an implementation flow of a queue allocation method according to an embodiment of the present application;
FIG. 2 is a block diagram of a queue allocation apparatus according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
The embodiment of the application provides a queue allocation method. The queue allocation method can be applied to electronic devices other than the consumer device, including but not limited to the production device, and third party devices which do not belong to the production device and do not belong to the consumer device. Wherein, production end equipment refers to: in the case where the message system adopts a producer consumer mode, a device that takes the role of producer performs an operation of production by putting a message into a queue; consumer-side device refers to: in the case where the message system adopts a producer consumer mode, the device, which acts as a consumer, operates to consume by reading messages from the queue.
It should be noted that, the queue allocation method provided by the embodiment of the present application is mainly aimed at the following application scenarios: one queue can only be consumed by one consumer device, but one consumer device can consume multiple queues; that is, one queue can be allocated to only one consumer device at a time, but one consumer device can be allocated a plurality of queues at a time.
Specifically, the queue allocation method provided by the embodiment of the present application may be periodically executed after the allocation of all queues is initially completed by the original allocation algorithm, where the cycle time may be configured according to the actual application scenario, which is not limited herein. Referring to fig. 1, fig. 1 shows a flow chart of a queue allocation method, which is described in detail as follows:
Step 101, counting the number of first messages to be consumed of the first consumer device according to all queues allocated to the first consumer device.
When the queue allocation method provided by the embodiment of the application is executed, the electronic equipment can traverse in the consumer-end cluster, namely, traverse all the consumer-end equipment. It is understood that each currently traversed consumer device may be referred to as a first consumer device.
The electronic device may query all queues that are currently allocated to the first consumer device. Wherein, all queues allocated to the first consumer device include not only the queues allocated to the first consumer device and still consumed by the first consumer device currently through the original allocation algorithm, but also the queues reallocated to the first consumer device and still consumed by the first consumer device currently. That is, as long as the queue is still currently consumed by the first consumer device, the queue belongs to the queue currently allocated to the first consumer device, no matter how the queue is allocated to the first consumer device.
Because each queue allocated to the first consumer device is a consumption object of the first consumer device, the electronic device can count and sum the real-time message number allocated to each queue of the first consumer device, and the obtained result is the current message number to be consumed of the first consumer device. For ease of distinction, this number of messages to be consumed is referred to herein as the first number of messages to be consumed.
It can be understood that the first message quantity to be consumed expresses the current consumption pressure of the first consumer device to a certain extent, specifically: the larger the number of the first messages to be consumed, the larger the consumption pressure of the first consumer device; the smaller the number of first messages to be consumed, the less the consumption pressure of the first consumer device.
Step 102, determining a first target queue in all queues under the condition that the number of the first messages to be consumed reaches a preset first number threshold.
The electronic device may preset a first quantity threshold based on historical consumption experience of each consumer device in combination with downstream consumption speed and acceptance of the messaging system. It is understood that the first number threshold value expresses an upper consumption pressure limit acceptable to the first consumer device.
After obtaining the first number of messages to be consumed by the first consumer device, the electronic device may compare the first number of messages to be consumed with a first number threshold. If the first number of messages to be consumed has reached the first number threshold, the current consumption pressure of the first consumer device is considered to be too high, the first consumer device has a requirement of being supported by other consumer devices, and the first consumer device can be supported by the other consumer devices. Based on this, the electronic device may determine a first target queue among all queues that have been allocated to the first consumer device, the first target queue referring to: waiting for the queue to be reassigned. Of course, if the number of the first messages to be consumed does not reach the first number threshold, the current consumption pressure of the first consumption end device is considered to be still smaller, and at this time, the queue allocated by the first consumption end device may not be adjusted.
Step 103, searching a second consumer device with the consumption capability of the first target queue in the consumer cluster where the first consumer device is located.
After determining the first target queue, the electronic device may search the consumer-end cluster for consumer-end devices that have a consumption capability for the first target queue. For ease of distinction, the consumer queue that is found may be referred to as the second consumer device. The electronic device may examine the consumption capability of the consumer device from several dimensions: performance dimension, support status dimension, and consumption pressure dimension.
Step 104, in the case of finding the second consumer device, reassigning the first target queue to the second consumer device.
In the process of executing step 103, once the electronic device searches any second consumer device, the search operation is stopped immediately, and the first target queue is reassigned to the second consumer device. Through the reassignment operation, the first target queue is not consumed by the first consumer device any more, and the first target queue is consumed by the second consumer device instead, that is, the second consumer device supports the first consumer device, which can help relieve the consumption pressure of the first consumer device.
Otherwise, if the electronic device fails to find any second consumer device, it is known that all consumer devices in the consumer cluster do not have the consumption capability for the first target queue. At this time, only allocation to the first target queue can be maintained; that is, consumption of the first target queue can only continue by the first consumer device.
In some embodiments, the support of the first consumer device by the second consumer device is not permanent. In the event that the first consumer device is re-enabled with the consumption capability of the first target queue, the second consumer device may cancel the support for the first consumer device. Based on this, after step 104, the queue allocation method may further include:
A1, counting the number of second messages to be consumed of the first consumer device according to all queues allocated to the first consumer device.
In the case that the first consumer device is supported by the second consumer device, the electronic device may periodically trigger the execution of step A1, and the period of time may be configured according to the actual application scenario, which is not limited herein. Similar to step 101, the electronic device may query all queues currently allocated to the first consumer device, count the number of real-time messages in each of the queues and sum the number of real-time messages, where the result is the current number of messages to be consumed by the first consumer device when supported by the second consumer device. For ease of distinction, this number of messages to be consumed is referred to herein as the second number of messages to be consumed. It is understood that all queues allocated to the first consumer device herein refer to queues that the first consumer device still consumes when supported by the second consumer device, and will not be described herein.
A2, calculating the number of the third message to be consumed according to the number of the second message to be consumed and the number of the real-time messages of the first target queue;
the electronic device may estimate the number of messages to be consumed by the first consumer device after the second consumer device cancels the support for the first consumer device. For ease of distinction, this number of messages to be consumed is referred to herein as the third number of messages to be consumed.
Obviously, if the second consumer device cancels the support of the first consumer device, the first target queue is consumed by the first consumer device, and thus the third message to be consumed may be specifically the sum of the second message to be consumed and the real-time message of the first target queue.
It will be appreciated that the third number of messages to be consumed expresses the consumption pressure of the first consumer device after the second consumer device has removed the support for the first consumer device.
A3, returning the first target queue to the first consumer device under the condition that the number of the third messages to be consumed does not reach the first number threshold.
The first number threshold value has been described above as expressing an upper consumption pressure limit acceptable to the first consumer device. Based on this, the electronic device may compare the third number of messages to be consumed with the first number threshold after obtaining the third number of messages to be consumed by the first consumer device.
If the third number of messages to be consumed has reached the first number threshold, it is considered that the consumption pressure of the first consumer device after the support is canceled is likely to be excessive. In this case, the electronic device does not let the second consumer device trade cancel the support for the first consumer device; that is, the first target queue will still be consumed by the second consumer device.
Otherwise, if the number of the third messages to be consumed does not reach the first number threshold, the consumption pressure of the first consumer device after the support is canceled is considered to be within an acceptable range. In this case, the electronic device may consider letting the second consumer device cancel the support for the first consumer device; that is, the first target queue may be returned to the first consumer device, which consumes the first target queue.
In some embodiments, to save resources of the electronic device, the execution timing of step A2 may be limited, and step A2 may be specifically expressed as:
and under the condition that the number of the second messages to be consumed does not reach a preset second number threshold, calculating the number of the third messages to be consumed according to the number of the second messages to be consumed and the number of the real-time messages of the first target queue.
The electronic device may preset a second number threshold according to the first number threshold, where the second number threshold is smaller than the first number threshold. For example only, the first number threshold may be half of the second number threshold, which is not limited herein. After step A1, the electronic device may compare the counted number of the second messages to be consumed with a second number threshold.
If the second number of messages to be consumed has reached the second number threshold, the consumption pressure of the first consumer device is considered to be still at a higher level. In this case, the third message to be consumed obtained by subsequent calculation is highly likely to reach the first quantity threshold; that is, the probability of the subsequent cancellation of the support is small. Based on this, in this case, the electronic device may not calculate the third number of messages to be consumed in order to save system resources.
Conversely, if the second number of messages to be consumed does not reach the second number threshold, the consumption pressure of the first consumer device is considered to be already at a lower level. In this case, the third message to be consumed obtained by subsequent calculation is highly likely not to reach the first quantity threshold; that is, the probability of the subsequent cancellation of the support is large. Based on this, the electronic device does not have to calculate the third number of messages to be consumed.
In some embodiments, since each consumer device in the consumer cluster may be regarded as a first consumer device, and the queue allocation method is performed periodically, the first consumer device may support other consumer devices. Based on this, step 102 may specifically include:
b1, determining whether second target queues exist in all queues under the condition that the number of the first messages to be consumed reaches a preset first number threshold value.
It has been described above that when the first number of messages to be consumed reaches the first number threshold, it means that the consumption pressure of the first consumer device is too high. Since the counted objects are all queues allocated to the first consumer device, including not only the queues allocated to the first consumer device and still consumed by the first consumer device currently by the original allocation algorithm, but also the queues reallocated to the first consumer device and still consumed by the first consumer device currently, the electronic device may determine whether a second target queue exists in all the queues if the number of the first messages to be consumed reaches a preset first number threshold, where the second target queue is: is reassigned to the queue of the first consumer device.
B2, if the second target queue does not exist, determining the first target queue in all queues.
If the electronic device determines that the second target queue does not exist in the step B1, it is known that the first consumer device does not support other consumer devices currently; that is, all queues that have been currently allocated to a first consumer device are queues that have been allocated to the first consumer device by the original allocation algorithm and are still currently consumed by the first consumer device. In this case, the first consumer device has a need to be supported by other consumer devices, which may be considered to support the first consumer device. Based on this, the electronic device may determine a first target queue among all queues that have been allocated to the first consumer device.
In some embodiments, if the electronic device determines that the second target queue exists in step B1, it may be known that the first consumer device is currently supporting other consumer devices. In this case, the electronic device considers that the consumption pressure of the first consumer device is too great, and is unable to continue to support other consumer devices. Based on this, after step B1, the queue allocation method may further include:
b3, returning the second target queue to the third consumer device if the second target queue exists.
Wherein the third consumer device refers to: before the second target queue is reassigned to the first consumer device, the consumer device is assigned originally; that is, the second target queue is initially allocated to a certain consumer device, which is the third consumer device, by the original allocation algorithm. It will be appreciated that the presence of the second target queue means that the first consumer device is supporting the third consumer device.
Since the consumption pressure of the first consumption end device is too high, in this case, the electronic device may consider to let the first consumption end device cancel the support for the third consumption end device; that is, the second target queue may be returned to the third consumer device, which consumes the second target queue.
In some embodiments, the first consumer device may have been assigned multiple queues when the consumption pressure of the first consumer device is greater. In order to avoid excessive consumption pressure on other consumer devices while relieving the consumption pressure of the first consumer device, a suitable first target queue may be determined by the following procedure:
and C1, traversing all queues according to the sequence of at least more real-time information quantity.
The electronic device may traverse all queues that have been allocated to the first consumer device in order. In the embodiment of the application, from the aspect of relieving the consumption pressure of the first consumption terminal equipment, the all queues can be traversed in the order of at least more than the number of the real-time messages.
And C2, if the number of the real-time messages of the currently traversed queue does not reach a preset third number threshold, determining the currently traversed queue as a first target queue, and stopping traversing.
The electronic device may preset a third number threshold according to the first number threshold, where the second number threshold is less than or equal to the first number threshold, and specific values of the third number threshold are not limited herein. The electronic device can compare the number of real-time messages currently traversing the queue to the third number threshold. From the standpoint of avoiding excessive consumption pressure on other consumer-side devices, the currently traversed queue is determined to be the first target queue only when the number of real-time messages does not reach a preset third number threshold. In addition, in order to save resources of the electronic device, the traversing operation may be stopped at this time.
It can be appreciated that, through steps C1 and C2, the resulting first target queue is: among the queues having the number of real-time messages not more than the third number threshold, the queue having the largest number of real-time messages.
By way of example only, assuming that all queues that have been allocated to the first consumer device include queues a, b, and c, the number of real-time messages for queue a is 5000, the number of real-time messages for queue b is 9000, the number of real-time messages for queue c is 1000, and the third number threshold is 8000, then: the queue traversed by the electronic equipment is a queue b, and the real-time information quantity (9000) of the queue b is found to reach a preset third quantity threshold (8000) by comparison, so that the queue b is likely to bring excessive pressure to other consumer-end equipment, and the queue b does not meet the requirements; the queue that the electronic device traverses is a queue a, and the number (5000) of real-time messages of the queue a is found to be not up to a preset third number threshold (8000) by comparison, so that the queue a is the queue with the largest number of the real-time messages found on the premise of not bringing excessive pressure to other consumer-end devices, and the queue b meets the requirement. Thus, through steps C1 and C2, queue b is identified as the first target queue.
It will be appreciated that if steps B1, B2, C1 and C2 are combined, step 102 may be specifically: determining whether second target queues exist in all queues under the condition that the number of the first messages to be consumed reaches a preset first number threshold; if the second target queue does not exist, traversing all the queues based on the sequence of the real-time message quantity from more than at least; if the number of the real-time messages of the currently traversed queue does not reach a preset third number threshold, determining the currently traversed queue as a first target queue, and stopping traversing.
In some embodiments, if the electronic device fails to find a second consumer device with consumer capabilities for the first target queue, the electronic device may determine a new first target queue and continue to find a second consumer device with consumer capabilities for the new first target queue. The above process is repeated until the second consumer device is found.
Further, when determining the new first target queue through steps C1 and C2, the queue that has been traversed in the past will not be considered by the electronic device, which may be implemented by marking the queue, or may be implemented by reducing the third number threshold, which is not limited herein.
In some embodiments, to further ensure the overall performance of the messaging system, a suitable second consumer device may be determined by:
D1, determining the device usage rate of other consumer devices for each other consumer device except the first consumer device in the consumer cluster, and counting the fourth message to be consumed of the other consumer devices according to all queues allocated to the other consumer devices.
Take any other consumer device as an example:
The electronic device may query for device status parameters of the other consumer device and parse the device usage from the device status parameters. In addition, similar to step 101, the electronic device may query all queues currently allocated to the other consumer device, count the real-time message numbers of the queues and sum the real-time message numbers of the queues, and obtain the current message number to be consumed of the other consumer device. For ease of distinction, this number of messages to be consumed is referred to herein as the fourth number of messages to be consumed. It will be understood that all queues allocated to the other consumer device herein refer to queues still consumed by the other consumer device at present, and will not be described herein.
In some examples, device usage includes: device CPU usage and/or device memory usage.
And D2, calculating the fifth message quantity to be consumed according to the fourth message quantity to be consumed and the real-time message quantity of the first target queue.
Take any other consumer device as an example:
the electronic device may estimate the number of messages to be consumed by the first consumer device after the other consumer devices support the first consumer device. For ease of distinction, this number of messages to be consumed is referred to herein as the fifth number of messages to be consumed.
Obviously, if the other consumer device supports the first consumer device, the other consumer device consumes the first target queue, so the fifth message to be consumed may be specifically the sum of the fourth message to be consumed and the real-time message of the first target queue.
It will be appreciated that the fifth number of messages to be consumed expresses the consumption pressure of the other consumer device after the other consumer device supports the first consumer device.
And D3, if the equipment utilization rate is lower than the preset utilization rate, and no queue which is redistributed to other consumer-end equipment exists in all queues which are distributed to other consumer-end equipment, and the fifth quantity of the messages to be consumed does not reach the first quantity threshold, determining the other consumer-end equipment as second consumer-end equipment.
Take any other consumer device as an example:
From a performance dimension, if the other consumer device is also required to support the first consumer device in the case where the device usage of the other consumer device is already high, the device usage of the other consumer device must continue to be high, which is likely to cause the other consumer device to overload.
From the support status dimension, if the other consumer device is also required to support the first consumer device if there are queues reassigned to the other consumer device in all of the queues that have been assigned to the other consumer device, the other consumer device supports more than one consumer device, which is likely to cause overload of the other consumer device.
From the consumption pressure dimension, if the other consumer device is also required to support the first consumer device in the case that the consumption pressure of the other consumer device is too high (i.e. the fifth number of messages to be consumed has reached the first number threshold), the consumption pressure of the other consumer device will necessarily continue to increase, which will likely lead to overload of the other consumer device.
Based on this, only if the device usage rate of the other consumer device is lower than the preset usage rate, and no queue reassigned to the other consumer device exists in all queues assigned to the other consumer device, and the fifth number of messages to be consumed of the other consumer device does not reach the first number threshold, the other consumer device is considered to have the consuming capability of the first target queue, and the electronic device may request the other consumer device to support the first consumer device; that is, in this case, the electronic device may determine the other consumer device as the second consumer device.
Assuming that consumer devices A, B and C are in the consumer cluster, the first number threshold is 10000, the second number threshold is 5000, and the third number threshold is 10000. The consumer device a is herein taken as a first consumer device (in fact, the consumer devices A, B and C may be sequentially taken as the first consumer device, which is not described herein in detail), so as to facilitate understanding of the queue allocation method provided in the embodiment of the present application, and the following describes some possible scenarios by way of specific examples:
First scenario:
The electronic device knowing, by querying, that the queue assigned to the consumer device a comprises: queues a, b and c. Wherein, the real-time information quantity of the queue a is 6000, the real-time information quantity of the queue b is 3000, and the real-time information quantity of the queue c is 4000. Thus, the first number of messages to be consumed by the consumer device a is 13000 and the first number threshold 10000 has been reached. On the basis, the electronic equipment discovery queue B is originally distributed to the consumer equipment B through an original distribution algorithm; that is, because consumer device a supports consumer device B, the queue B is reassigned to consumer device a.
In this scenario, the electronic device may return queue B to consumer device B, thereby relieving consumer device a from the consumer pressure.
The second scenario:
the queue known to be allocated to consumer device a by the query includes: queues a, b and c. The real-time information quantity of the queue a is 6000, the real-time information quantity of the queue b is 3000, and the real-time information quantity of the queue c is 4000. Thus, the first number of messages to be consumed by the consumer device a is 13000 and the first number threshold 10000 has been reached. On the basis, the electronic equipment determines that the queues a, b and c are distributed to the consumer equipment A through the original distribution algorithm; that is, consumer device a does not support consumer devices B and C.
In this scenario, the electronic device determines queue a as the first target queue through a third number threshold. After comprehensively considering the performance dimension, the support state dimension and the consumption pressure dimension, the electronic device determines the consumption end device B as a second consumption end device. The electronic device may reassign queue a to consumer device B, thereby relieving consumer device a of the consumer pressure.
Third scenario:
Based on the second scenario, after reassigning the queue a to the consumer device B for a period of time, the electronic device obtains that the real-time message number of the queue a is 4000, the real-time message number of the queue B is 1000, and the real-time message number of the queue c is 3000 through inquiry. Thus, the electronic device knows that the second number of messages to be consumed by the consumer device a is 4000, and the second number threshold 5000 is not reached. At this time, the electronic device may continue to calculate that the third message number to be consumed of the consumer device a is 8000, and does not reach the first number threshold 10000.
In this scenario, the electronic device may return queue a to consumer device a; that is, the support of the consumer device B for the consumer device a is canceled.
From the above, in the embodiment of the present application, the first message quantity to be consumed expresses the consumption pressure of the first consumer device. When the number of the first messages to be consumed reaches a preset first number threshold, it is known that the consumption pressure of the first consumer device is high, and at this time, a queue waiting to be reassigned, that is, a first target queue, can be determined from the queues assigned to the first consumer device. In order to alleviate the consumption pressure of the first consumer device, a second consumer device with the consumption capability of the first target queue can be searched in the consumer cluster where the first consumer device is located, and the first target queue is redistributed to the second consumer device when the second consumer device is searched. Thus, the queue allocated to the first consumer device can be reduced, so that the originally larger consumption pressure of the first consumer device can be relieved. The process realizes dynamic allocation of the queues according to the consumption pressure of consumers, helps to promote the consumption speed of the queues, and can ensure the overall consumption performance.
Corresponding to the queue allocation method provided above, the embodiment of the application also provides a queue allocation device. As shown in fig. 2, the queue allocation device 2 includes:
A first statistics module 201, configured to count a first number of messages to be consumed of the first consumer device according to all queues allocated to the first consumer device;
A determining module 202, configured to determine a first target queue from all queues if the number of first messages to be consumed reaches a preset first number threshold, where the first target queue is: waiting for a reassigned queue;
a searching module 203, configured to search, in a consumer cluster where the first consumer device is located, a second consumer device that has a consuming capability on the first target queue;
And the reassignment module 204 is configured to reassign the first target queue to the second consumer device if the second consumer device is found.
In some embodiments, the queue allocation apparatus 2 further comprises:
The second statistics module is used for counting the number of second messages to be consumed of the first consumption terminal equipment according to all the queues distributed to the first consumption terminal equipment after the first target queue is redistributed to the second consumption terminal equipment;
The calculation module is used for calculating the third message quantity to be consumed according to the second message quantity to be consumed and the real-time message quantity of the first target queue;
And the first returning module is used for returning the first target queue to the first consumer device under the condition that the number of the third messages to be consumed does not reach the first number threshold value.
In some embodiments, the calculating module is specifically configured to calculate, when the second number of messages to be consumed does not reach a preset second number threshold, a third number of messages to be consumed according to the second number of messages to be consumed and the real-time number of messages in the first target queue, where the second number threshold is smaller than the first number threshold.
In some embodiments, the determination module 202 includes;
the first determining unit is configured to determine whether second target queues exist in all queues if the number of the first messages to be consumed reaches a preset first number threshold, where the second target queues are: a queue reassigned to the first consumer device;
And the second determining unit is used for determining the first target queue in all the queues if the second target queue does not exist.
In some embodiments, the queue allocation apparatus 2 further comprises:
and the second returning module is configured to return the second target queue to the third consumer device if the second target queue exists, where the third consumer device is: the second target queue is assigned to the consumer device before being reassigned to the first consumer device.
In some embodiments, determining the first target queue among all queues includes:
traversing all queues based on the sequence of the number of real-time messages from more than one;
If the number of the real-time messages of the currently traversed queue does not reach a preset third number threshold, determining the currently traversed queue as a first target queue, and stopping traversing, wherein the third number threshold is smaller than or equal to the first number threshold.
In some embodiments, the lookup module 203 includes:
a third determining unit, configured to determine, for each other consumer device except the first consumer device in the consumer cluster, a device usage rate of the other consumer device;
A statistics unit, configured to count a fourth number of messages to be consumed by other consumer devices according to all queues allocated to the other consumer devices;
The calculating unit is used for calculating the fifth message quantity to be consumed according to the fourth message quantity to be consumed and the real-time message quantity of the first target queue;
And the fourth determining unit is used for determining the other consumer-end devices as the second consumer-end device if the device usage rate is lower than the preset usage rate, no queue which is reassigned to the other consumer-end devices exists in all queues which are assigned to the other consumer-end devices, and the fifth number of messages to be consumed does not reach the first number threshold.
From the above, in the embodiment of the present application, the first message quantity to be consumed expresses the consumption pressure of the first consumer device. When the number of the first messages to be consumed reaches a preset first number threshold, it is known that the consumption pressure of the first consumer device is high, and at this time, a queue waiting to be reassigned, that is, a first target queue, can be determined from the queues assigned to the first consumer device. In order to alleviate the consumption pressure of the first consumer device, a second consumer device with the consumption capability of the first target queue can be searched in the consumer cluster where the first consumer device is located, and the first target queue is redistributed to the second consumer device when the second consumer device is searched. Thus, the queue allocated to the first consumer device can be reduced, so that the originally larger consumption pressure of the first consumer device can be relieved. The process realizes dynamic allocation of the queues according to the consumption pressure of consumers, helps to promote the consumption speed of the queues, and can ensure the overall consumption performance.
Corresponding to the queue allocation method provided above, the embodiment of the application also provides an electronic device. Referring to fig. 3, an electronic device 3 according to an embodiment of the present application includes: a memory 301, one or more processors 302 (only one shown in fig. 3) and computer programs stored on the memory 301 and executable on the processors. Wherein: the memory 301 is used for storing software programs and units, and the processor 302 executes various functional applications and data processing by running the software programs and units stored in the memory 301 to obtain resources corresponding to the preset events. Specifically, the processor 302 implements the following steps by running the above-described computer program stored in the memory 301:
Counting the number of first messages to be consumed of the first consumer device according to all queues allocated to the first consumer device;
under the condition that the number of the first messages to be consumed reaches a preset first number threshold value, determining a first target queue in all queues, wherein the first target queue is as follows: waiting for a reassigned queue;
Searching a second consumer device with the consumption capability of the first target queue in a consumer cluster where the first consumer device is located;
And in the case of finding the second consumer device, reassigning the first target queue to the second consumer device.
Assuming that the foregoing is a first possible implementation, in a second possible implementation provided by way of example of the first possible implementation, after reassigning the first target queue to the second consumer device, the queue assignment method further includes:
Counting the number of second messages to be consumed of the first consumer device according to all queues allocated to the first consumer device;
Calculating the third message quantity to be consumed according to the second message quantity to be consumed and the real-time message quantity of the first target queue;
And returning the first target queue to the first consumer device if the number of the third messages to be consumed does not reach the first number threshold.
In a third possible implementation manner provided by the second possible implementation manner, the calculating the third number of messages to be consumed according to the second number of messages to be consumed and the real-time number of messages of the first target queue includes:
And under the condition that the number of the second messages to be consumed does not reach a preset second number threshold, calculating the number of the third messages to be consumed according to the number of the second messages to be consumed and the number of the real-time messages of the first target queue, wherein the second number threshold is smaller than the first number threshold.
In a fourth possible implementation provided on the basis of the first possible implementation, or on the basis of the second possible implementation, or on the basis of the third possible implementation, determining a first target queue in all queues in case the number of first messages to be consumed reaches a preset first number threshold, including;
Determining whether a second target queue exists in all queues under the condition that the number of the first messages to be consumed reaches a preset first number threshold value, wherein the second target queue is: a queue reassigned to the first consumer device;
if the second target queue does not exist, the first target queue is determined in all queues.
In a fifth possible implementation manner provided by the fourth possible implementation manner as set forth above, after determining whether the second target queue exists in all queues, the queue allocation method further includes:
If the second target queue exists, returning the second target queue to the third consumer device, wherein the third consumer device is: the second target queue is assigned to the consumer device before being reassigned to the first consumer device.
In a sixth possible implementation provided by the first possible implementation as a basis, the second possible implementation as a basis, or the third possible implementation as a basis, determining the first target queue in all queues includes:
traversing all queues based on the sequence of the number of real-time messages from more than one;
If the number of the real-time messages of the currently traversed queue does not reach a preset third number threshold, determining the currently traversed queue as a first target queue, and stopping traversing, wherein the third number threshold is smaller than or equal to the first number threshold.
In a seventh possible implementation provided by the first possible implementation or the second possible implementation or the third possible implementation as a basis, searching for a second consumer device that has a consumption capability for the first target queue includes:
For each other consumer device except the first consumer device in the consumer cluster, determining the device usage rate of the other consumer device, and counting the fourth message to be consumed of the other consumer device according to all queues allocated to the other consumer device;
calculating the fifth message quantity to be consumed according to the fourth message quantity to be consumed and the real-time message quantity of the first target queue;
And if the equipment utilization rate is lower than the preset utilization rate, and no queue which is redistributed to other consumer-end equipment exists in all queues which are distributed to other consumer-end equipment, and the fifth quantity of the messages to be consumed does not reach the first quantity threshold, determining the other consumer-end equipment as second consumer-end equipment.
It should be appreciated that in embodiments of the present application, the Processor 302 may be a central processing unit (Central Processing Unit, CPU), which may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuits (ASICs), off-the-shelf Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Memory 301 may include read only memory and random access memory and provide instructions and data to processor 302. Some or all of memory 301 may also include non-volatile random access memory. For example, the memory 301 may also store information of a device class.
From the above, in the embodiment of the present application, the first message quantity to be consumed expresses the consumption pressure of the first consumer device. When the number of the first messages to be consumed reaches a preset first number threshold, it is known that the consumption pressure of the first consumer device is high, and at this time, a queue waiting to be reassigned, that is, a first target queue, can be determined from the queues assigned to the first consumer device. In order to alleviate the consumption pressure of the first consumer device, a second consumer device with the consumption capability of the first target queue can be searched in the consumer cluster where the first consumer device is located, and the first target queue is redistributed to the second consumer device when the second consumer device is searched. Thus, the queue allocated to the first consumer device can be reduced, so that the originally larger consumption pressure of the first consumer device can be relieved. The process realizes dynamic allocation of the queues according to the consumption pressure of consumers, helps to promote the consumption speed of the queues, and can ensure the overall consumption performance.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of external device software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the system embodiments described above are merely illustrative, e.g., the division of modules or units described above is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The integrated units described above, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may also be implemented by implementing all or part of the flow of the method of the above embodiment, or by instructing the associated hardware by a computer program, where the computer program may be stored on a computer readable storage medium, and where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. The computer program comprises computer program code, and the computer program code can be in a source code form, an object code form, an executable file or some intermediate form and the like. The above computer readable storage medium may include: any entity or device capable of carrying the computer program code described above, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer readable Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable storage medium described above may be appropriately increased or decreased according to the requirements of the jurisdiction's legislation and the patent practice, for example, in some jurisdictions, the computer readable storage medium does not include electrical carrier signals and telecommunication signals according to the legislation and the patent practice.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims (10)

1. A method of queue allocation, comprising:
Counting the first message quantity to be consumed of the first consumer device according to all queues allocated to the first consumer device;
Determining a first target queue in all queues under the condition that the number of the first messages to be consumed reaches a preset first number threshold value, wherein the first target queue is: waiting for a reassigned queue;
searching a second consumer device with the consumption capability of the first target queue in a consumer cluster where the first consumer device is located;
and under the condition that the second consumer-end equipment is found, the first target queue is redistributed to the second consumer-end equipment.
2. The queue allocation method according to claim 1, wherein after said reassigning the first target queue to the second consumer device, the queue allocation method further comprises:
counting the number of second messages to be consumed of the first consumer device according to all queues allocated to the first consumer device;
Calculating the third message quantity to be consumed according to the second message quantity to be consumed and the real-time message quantity of the first target queue;
Returning the first target queue to the first consumer device if the third number of messages to be consumed does not reach the first number threshold.
3. The queue allocation method according to claim 2, wherein the calculating a third message number to be consumed according to the second message number to be consumed and the real-time message number of the first target queue includes:
and under the condition that the number of the second messages to be consumed does not reach a preset second number threshold, calculating a third number of messages to be consumed according to the number of the second messages to be consumed and the number of the real-time messages of the first target queue, wherein the second number threshold is smaller than the first number threshold.
4. A queue allocation method according to any one of claims 1 to 3, in which in the event that the first number of messages to be consumed reaches a preset first number threshold, a first target queue is determined among the all queues, including;
Determining whether a second target queue exists in all queues or not under the condition that the number of the first messages to be consumed reaches a preset first number threshold value, wherein the second target queue is: a queue reassigned to the first consumer device;
if the second target queue does not exist, determining a first target queue in all the queues.
5. The queue allocation method according to claim 4, further comprising, after said determining whether a second target queue exists among the all queues:
if the second target queue exists, returning the second target queue to third consumer-end equipment, wherein the third consumer-end equipment is: the second target queue is assigned to the consumer device before being reassigned to the first consumer device.
6. A method of allocating queues according to any one of claims 1 to 3, wherein said determining a first target queue among said all queues comprises:
traversing all queues based on the sequence of the number of real-time messages from more than at least;
and if the number of the real-time messages of the currently traversed queue does not reach a preset third number threshold, determining the currently traversed queue as the first target queue, and stopping traversing, wherein the third number threshold is smaller than or equal to the first number threshold.
7. A method of queue allocation as claimed in any one of claims 1 to 3, wherein the locating a second consumer device having consumer capability for the first target queue comprises:
For each other consumer device in the consumer cluster except for the first consumer device, determining the device usage rate of the other consumer device, and counting the fourth message to be consumed of the other consumer device according to all queues allocated to the other consumer device;
calculating the fifth message quantity to be consumed according to the fourth message quantity to be consumed and the real-time message quantity of the first target queue;
And if the equipment utilization rate is lower than a preset utilization rate, and no queue which is reassigned to the other consumer-end equipment exists in all queues which are assigned to the other consumer-end equipment, and the fifth number of messages to be consumed does not reach the first number threshold, determining the other consumer-end equipment as the second consumer-end equipment.
8. A queue allocation apparatus, comprising:
a first statistics module, configured to count a first number of messages to be consumed of a first consumer device according to all queues allocated to the first consumer device;
A determining module, configured to determine a first target queue from all queues when the number of the first messages to be consumed reaches a preset first number threshold, where the first target queue is: waiting for a reassigned queue;
the searching module is used for searching a second consumer-end device with the consumption capability of the first target queue in the consumer-end cluster where the first consumer-end device is located;
And the reassignment module is used for reassigning the first target queue to the second consumer device under the condition that the second consumer device is found.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method according to any one of claims 1 to 7.
CN202310139774.1A 2023-02-13 2023-02-13 Queue distribution method, queue distribution device, electronic equipment and storage medium Pending CN118488012A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310139774.1A CN118488012A (en) 2023-02-13 2023-02-13 Queue distribution method, queue distribution device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310139774.1A CN118488012A (en) 2023-02-13 2023-02-13 Queue distribution method, queue distribution device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN118488012A true CN118488012A (en) 2024-08-13

Family

ID=92192576

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310139774.1A Pending CN118488012A (en) 2023-02-13 2023-02-13 Queue distribution method, queue distribution device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN118488012A (en)

Similar Documents

Publication Publication Date Title
US11734271B2 (en) Data query method, apparatus and device
TWI694339B (en) Blockchain consensus method, equipment and system
CN107241281B (en) Data processing method and device
RU2460222C2 (en) System and method of allocating memory in embedded systems or wireless communication systems
CN111290917A (en) YARN-based resource monitoring method and device and terminal equipment
CN111857992B (en) Method and device for allocating linear resources in Radosgw module
CN111490890A (en) Hierarchical registration method, device, storage medium and equipment based on micro-service architecture
CN106936867A (en) The response method and device of a kind of service request
CN105718315A (en) Task processing method and server
CN110928685A (en) Intelligent cooperative computing system and method
CN106603256B (en) Flow control method and device
CN102970349B (en) A kind of memory load equalization methods of DHT network
CN109819674B (en) Computer storage medium, embedded scheduling method and system
CN118488012A (en) Queue distribution method, queue distribution device, electronic equipment and storage medium
CN106411545A (en) Service attribute counting method and device
CN113111083A (en) Method, device, equipment, storage medium and program product for data query
CN112286930A (en) Method, device, storage medium and electronic equipment for resource sharing of redis business side
WO2017070869A1 (en) Memory configuration method, apparatus and system
CN111597041A (en) Calling method and device of distributed system, terminal equipment and server
CN114860449A (en) Data processing method, device, equipment and storage medium
CN113783912A (en) Request distribution method, device and storage medium
CN113254208A (en) Load balancing method and device for server, server and storage medium
CN114610575B (en) Method, apparatus, device and medium for calculating updated peak value of branch
CN113762954B (en) Hot account transaction processing method and device, electronic equipment and storage medium
CN111290850A (en) Data storage method, device and equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication