CN117435361A

CN117435361A - Method and device for realizing priority message queue and computer readable storage medium

Info

Publication number: CN117435361A
Application number: CN202311251670.6A
Authority: CN
Inventors: 刘砺志; 佟志臣; 蒋海俭; 闵青
Original assignee: China Unionpay Co Ltd
Current assignee: China Unionpay Co Ltd
Priority date: 2023-09-26
Filing date: 2023-09-26
Publication date: 2024-01-23

Abstract

The application provides a method, a device, a system and a computer readable storage medium for realizing a priority message queue, wherein the method comprises the following steps: dividing partitions under the Kafka message subject into a plurality of subgroups, wherein each subgroup corresponds to a priority, and the number of partitions allocated in each subgroup is proportional to the priority; processing resources are allocated according to the priority of each group. With the above method, the priority processing scheme can be implemented in a simple scheme.

Description

Method and device for realizing priority message queue and computer readable storage medium

Technical Field

The application belongs to the field of data, and in particular relates to a method and a device for realizing a priority message queue and a computer readable storage medium.

Background

This section is intended to provide a background or context for embodiments of the present application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

In recent years, with the rapid development of information technology, global data has the characteristics of explosive growth and mass aggregation. In big data scenarios, due to the huge amount of data, directly transmitting and processing data often leads to system performance problems. By introducing a Message Queue (MQ) for asynchronous communication, application decoupling, message asynchronization and flow peak clipping can be effectively realized, and further, the reliability, expandability and throughput of the system are improved, so that big data application is more efficient and flexible. Thus, message queue middleware has become one of the indispensable infrastructure in today's big data applications. The Apache Kafka, which is a distributed stream processing platform of an Apache foundation, becomes one of the most popular message queues at present, and is widely applied to the fields of Internet, finance, internet of things and the like due to the characteristics of high throughput, low delay, high reliability and the like.

In practice, it is often encountered that requests for users of different classes need to be processed at different response rates. For example, requests of general users are handled stepwise in response to requests of VIP users preferentially. From a system modeling perspective, this scenario can be implemented by prioritizing message queues. The priority message queue is used as a special message queue, and can orderly process and distribute the messages according to the priority of the messages, so that the messages with low priority are prevented from blocking the messages with high priority for a long time, and the messages with high priority are ensured to be processed in time.

Currently, kafka-based priority message queue implementations can be summarized in two broad categories: (1) use multiple topics. Creating a plurality of topics according to different priorities, each priority corresponding to one topic, and creating a consumer subscription process for each topic. Then, constructing a priority Consumer to aggregate consumers with multiple priorities, and setting different quotas for consumers with different priorities, wherein the higher the priority is, the larger the corresponding quota is. And the PrioritiConsumer sequentially pulls the data in the theme corresponding to each priority from high to low according to the quota. Therefore, the effect of 'queue' is realized by pulling the difference phase change of the number of the messages with each priority each time, and the aim that the messages with high priority are consumed preferentially is fulfilled. (2) reordering using buffering. A priority queue (such as a binary heap) buffer is established in the memory, and the messages are sent into the buffer and reordered according to the priority, so that the ordered output of the messages is realized at the consumption end.

However, both of the above solutions have respective limitations. For the first scheme, it needs to build one topic for each priority, so that one message topic logically is split into multiple in physical storage, and the cohesiveness of the system is destroyed. And, multiple topics need to be created when priorities are high. If there are many priority message queues in a single Broker node, a large number of topic partitions will be caused on the whole, and the processing performance of the cluster is seriously reduced. For the second scheme, if the buffering is implemented by using an unbounded queue, if the consumption speed of the consumer is not fast enough, the memory overflow is most likely to be caused after the 'wave crest' is surging in. If a bounded blocking queue is used, although the problem of insufficient memory is avoided, when the data volume is large, the queue is in a blocking state for most of the time, and the throughput of the system is greatly impaired.

Therefore, how to handle users of different classes to handle their requests at different response rates is a challenge.

Disclosure of Invention

In order to solve the problems in the prior art, a method, a device and a computer readable storage medium for implementing a priority message queue are provided, and the problems can be solved by using the method, the device and the computer readable storage medium.

The present application provides the following solutions.

In a first aspect, a method for implementing a priority message queue is provided, including: dividing partitions under the Kafka message subject into a plurality of subgroups, wherein each subgroup corresponds to a priority, and the number of partitions allocated in each subgroup is proportional to the priority; processing resources are allocated according to the priority of each group.

In one embodiment, partitioning the partitions under the Kafka message topic into multiple subgroups includes: and distributing the corresponding partition numbers to each subgroup according to the partition proportion corresponding to each priority and the total partition number under the Kafka theme.

In one embodiment, partitioning the partitions under the Kafka message subject matter into a plurality of subgroups further comprises: and according to the priority weights of the priorities and the total number of the partitions, corresponding partition numbers are distributed to the subgroups according to the linear proportion.

In one embodiment, the corresponding number of partitions is assigned to each subgroup in linear proportion using the following formula:

n is the number of priority, and the priority weights are w in turn ₁ >w ₂ >…>w _N P is the total number of partitions.

In one embodiment, partitioning the partitions under the Kafka message subject matter into a plurality of subgroups further comprises: and distributing the corresponding partition numbers to each subgroup according to the priority weights of the priorities and the total number of the partitions in an exponential proportion.

In one embodiment, the corresponding number of partitions is exponentially assigned to each subgroup using the following formula:

In one embodiment, the method further comprises: in a code implementation, each subgroup is represented by a priority subgroup object; the member variables of the object include a list storing all subject partition numbers assigned by the group and variables representing the corresponding priority weight values of the group.

In one embodiment, the method further comprises: sequentially distributing the partitions to corresponding priority subgroups from high priority to low priority; if there are remaining partitions that are not allocated, the partitions are assigned to the groups one by one from high to low in priority by polling.

In one embodiment, the method further comprises: setting a priority partitioner for determining a partition number to be sent according to priority information represented by a key value carried in a message record; the message record specifies a key value for indicating the priority corresponding to the message.

In one embodiment, the method further comprises: the priority allocation policy is set to ensure that consumers consume only the partitions in the group to which the priority to be processed corresponds.

In one embodiment, the method further comprises: the priority partitioner adopts a polling mode to route the message records to different partitions in turn; the priority allocation priority assignment is used for allocating the partitions under the corresponding priority subgroups to each consumer in the consumer group by means of polling.

In one embodiment, the method further comprises: designating use of the priority partitioner by a first configuration parameter; a priority producer class is created based on the first configuration information to provide to an producer client having a priority message queue characteristic to an applicator.

In one embodiment, the method further comprises: designating use of the priority allocation priority by a second configuration parameter; creating a priority consumer class based on the second configuration parameters, and providing the consumer client with priority message queue characteristics to an application.

In a second aspect, there is provided a priority message queue implementing apparatus, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform: the method of the first aspect.

In a third aspect, there is provided a computer readable storage medium storing a program which, when executed by a multi-core processor, causes the multi-core processor to perform the method of the fourth aspect.

One of the advantages of the above embodiments, the core idea of the embodiments of the present application is to divide the partitions under the Kafka message topic into several subgroups, each subgroup corresponding to a priority. The greater the number of partitions in a group, the higher the priority. Thus, the variable number of partitions is regarded as a "weight" given priority, and the effect of "priority processing" is achieved by allocating more processing resources to the high-priority messages. The scheme not only does not need to split the message theme to destroy the system cohesion, but also does not have the problems of memory overflow or throughput reduction like the use of a priority queue buffer.

Other advantages of the present application will be explained in more detail in connection with the following description and accompanying drawings.

It should be understood that the foregoing description is only an overview of the technical solutions of the present application so that the technical means of the present application may be more clearly understood and may be implemented in accordance with the content of the specification. The following specific embodiments of the present application are illustrated in order to make the above and other objects, features and advantages of the present application more comprehensible.

Drawings

The advantages and benefits described herein, as well as other advantages and benefits, will become apparent to those of ordinary skill in the art upon reading the following detailed description of the exemplary embodiments. The drawings are only for purposes of illustrating exemplary embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:

FIG. 1 is a basic architectural diagram of Apache Kafka;

fig. 2 is a schematic diagram of a priority message queue implementation device according to an embodiment of the present application;

fig. 3 is a flow chart of a method for implementing a priority message queue according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a method for implementing a priority message queue according to an embodiment of the present application;

fig. 5 is a UML class diagram of a method for implementing a priority message queue according to an embodiment of the present application;

fig. 6 is an effect schematic diagram of a method for implementing a priority message queue according to an embodiment of the present application;

in the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

Detailed Description

Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

In the description of embodiments of the present application, it should be understood that terms such as "comprises" or "comprising" are intended to indicate that the disclosed features, numbers, steps, acts, components, portions, or combinations thereof, are present in the specification, and do not preclude the presence or addition of one or more other features, numbers, steps, acts, components, portions, or combinations thereof.

Unless otherwise indicated, "/" means or, e.g., A/B may represent A or B; "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone.

The terms "first," "second," and the like are used merely for convenience of description to distinguish between the same or similar technical features, and are not to be construed as indicating or implying a relative importance or quantity of such technical features. Thus, a feature defined by "first," "second," etc. may explicitly or implicitly include one or more such feature. In the description of embodiments of the present application, the term "plurality" means two or more unless otherwise indicated.

First, background knowledge of Apache Kafka will be described.

Apache Kafka is a high-performance, distributed stream processing platform for building scalable, fault tolerant and real-time data stream applications. It is an open-source project developed and maintained by the Apache software foundation and has become the solution of choice for many large enterprises and organizations in processing real-time data. The basic architecture of Apache Kafka is shown in fig. 1.

Apache Kafka has several core concepts:

1. producer (Producer): the producer is the sender of the data and is responsible for publishing the data to the Kafka cluster. It sends a message to a specific Topic (Topic) and writes it into one or more partitions (Partition).

2. Proxy (Broker): the proxy is a server in the Kafka cluster that is responsible for receiving, storing and distributing messages. The system is a core component of the whole system, can be deployed on a plurality of machines in a cluster, and realizes high availability and scalability of data.

3. Consumer (Consumer): a consumer is a recipient of data that subscribes to one or more topics and pulls messages from agents for processing. Consumers may organize in different ways to form Consumer groups (Consumer groups) to enable parallel processing and load balancing of messages.

4. Theme (Topic): the subject is a logical container of message streams with the same characteristics. It is the basic unit of message publishing and subscribing, and messages can be classified and organized by topic.

5. Partition (Partition): the topic may be divided into one or more partitions, each of which is an ordered and persistent sequence of messages. Partitioning allows Kafka clusters to process and store messages in parallel on multiple servers.

Apache Kafka achieves high performance, scalable and real-time distributed data stream processing through the core concepts described above. Its architectural design enables it to handle large amounts of data and highly concurrent accesses, and provides reliability and scalability guarantees. Whether building a real-time data processing system, log aggregation, event driven applications, etc., kafka is an ideal choice.

The workflow of the producer in Apache Kafka to send a message is as follows:

1. constructing a message record: the producer creates a producer rRecap object to encapsulate the message. The message record object needs to contain a message Topic (Topic) and a Value (Value), and a Key (Key) or a Partition (Partition) can be selectively specified.

2. Serializing: to facilitate transmission over a network, when sending a message, the producer will serialize the keys and values into byte arrays via a Serializer (Serializer).

3. Message partition selection: if a partition is specified in the message record, the Partitioner (Partitioner) returns the partition directly; otherwise, the partitioner will make a partition selection based on the partition policy. This decision may be based on the key of the message or other rules that are customized to decide to which partition the message is sent.

4. Sending a message: after knowing the topic and partition to which the message belongs, the producer writes the message into the corresponding partition for the specified topic. To improve performance, producers typically store multiple messages in a cache (RecordAccumulator) for bulk delivery.

5. Message confirmation: upon successful or unsuccessful message delivery, the producer will receive corresponding acknowledgment information. If the sending is successful, the producer can continue to send more messages; if the transmission fails, the producer may choose to retry or take other processing steps.

The workflow in Apache Kafka for a consumer to pull a cancel message is as follows:

1. configuring the consumer: first, the consumer needs to configure, including specifying the address and port of the Kafka cluster, as well as other relevant parameters. These configuration information will be used to establish a connection with the Kafka cluster.

2. Creating a consumer instance: a consumer instance is created through the client API provided by Kafka. This instance will be used to communicate with the Kafka cluster.

3. Subscription topic: consumers need to subscribe to one or more topics to specify the source of the message of interest. By invoking the subscription method, the consumer can subscribe to the topic and specify an optional partition allocator (PartitionAssignor).

4. Partition allocation: if a consumer specifies a partition allocator, it will be responsible for deciding which partitions of the topic to allocate to the consumer.

5. Pulling a message request: the consumer uses the consumer instance to send a request to the Kafka cluster to pull the message. The request contains the topic and partition information subscribed to by the consumer, and the consumer's current Offset (Offset).

6. Agent response: after the Kafka agent receives the pull message request, it retrieves the messages in the corresponding partition and returns the available messages to the consumer. The response contains information such as offset, key, value, etc. of the message.

7. Processing a message: after receiving the response from the agent, the consumer processes the pulled message.

8. Offset commit: after the consumer has processed a batch of messages, the consumer needs to submit an offset to identify the messages that have been consumed. In this way, the consumer can continue to consume the message from the last submitted offset location the next time the message is pulled.

The embodiments of the present application are specifically described based on the background knowledge description of Apache Kafka above.

In addition, it should be noted that, without conflict, the embodiments and features of the embodiments in the present application may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Fig. 2 is a schematic structural diagram of a hardware running environment according to an embodiment of the present application.

It should be noted that fig. 1 may be a schematic structural diagram of a hardware running environment implemented by using a priority message queue. The implementation device based on the priority message queue in the embodiment of the application can be terminal devices such as a PC (personal computer), a portable computer and the like.

As shown in fig. 2, the partition grouping apparatus based on the Kafka theme may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.

Those skilled in the art will appreciate that the priority message queue implementing apparatus structure shown in fig. 1 does not constitute a limitation of the priority message queue implementing apparatus, and may include more or fewer components than shown, or may combine certain components, or may be a different arrangement of components.

As shown in fig. 2, an operating system, a network communication module, a user interface module, and a blockchain-based information transfer program may be included in the memory 1005 as one type of computer storage medium. The operating system is a program for managing and controlling hardware and software resources of the priority message queue implementation device, and supports the execution of the priority message queue implementation program and other software or programs.

In the priority message queue implementing apparatus shown in fig. 2, the user interface 1003 is mainly used for receiving requests, data, etc. sent by the first terminal, the second terminal, and the supervision terminal; the network interface 1004 is mainly used for connecting the background server and the background server for data communication; and the processor 1001 may be configured to call a priority message queue implementation program stored in the memory 1005, and perform the following operations:

dividing partitions under the Kafka message subject into a plurality of subgroups, wherein each subgroup corresponds to a priority, and the number of partitions allocated in each subgroup is proportional to the priority; processing resources are allocated according to the priority of each group.

The above method divides the partitions under the Kafka message topic into several subgroups, each subgroup corresponding to a priority, and the messages are routed to a particular subgroup according to the priority. The higher the priority, the greater the number of partitions in the assigned group. Thus, the variable number of partitions looks is considered a "weight" given priority. The higher the priority, the more processing resources can be allocated, and the faster the message is processed, thereby realizing differentiated response of data with different priorities.

Fig. 3 is a flow chart illustrating a method for implementing a priority message queue according to an embodiment of the present application. In this flow, from a device perspective, the execution subject may be one or more electronic devices; from the program perspective, the execution subject may be a program mounted on these electronic devices, accordingly. In this embodiment, the execution body of the method may be a priority message queue implementation in the embodiment shown in fig. 2.

Referring to fig. 3, the method includes:

step 310, dividing partitions under the Kafka message subject into a plurality of subgroups;

wherein each subgroup corresponds to a priority, and the number of partitions allocated in each subgroup is proportional to the priority;

step 320, allocating processing resources according to the priority of each subgroup.

Fig. 4 shows a schematic diagram of a priority message queue designed according to this embodiment, fig. 5 shows a UML class diagram of a code implementation of the scheme of fig. 4, and referring to fig. 4 and fig. 5, in Kafka, a topic may be subdivided into a plurality of partitions, such as partitions 0,1,2, …, where a partition belongs to only a single topic, and messages contained in different partitions under the same topic are different. As a basic storage unit of Kafka, a partition can be regarded as an appendable Log (Log) file on a storage level. The message is assigned a specific offset when appended to the partition log file as a unique identification of the message in the partition and thereby ensuring that messages from the same partition are in order. This means that Kafka can only guarantee that the partitions are ordered and not the whole topic. Once the message is appended to the partition log, the order cannot be adjusted. Therefore, by adjusting the ordering of messages directly according to priority in Kafka is not possible.

To achieve the macroscopic property of the Kafka topic with priority message queues, the core idea of the implementation scheme based on the Kafka topic partition grouping proposed in this embodiment is to divide the partitions under the Kafka message topic into several groups (groups), each group corresponding to a priority. The greater the number of partitions in a group, the higher the priority. Thus, the variable number of partitions is treated as a "weight" to achieve the "priority handling" effect by allocating more processing resources to the high priority messages.

Further, to embody the core concept, the following implementation schemes may be adopted:

in one embodiment, in order for the producer to ensure that the message is correctly written into the partition under the group corresponding to the priority thereof when the message is sent to the topic, a priority partitioner (at the producer side) may be set, for determining the partition number to be sent according to the priority information represented by the key value carried in the message record; the message record specifies a key value for indicating the priority corresponding to the message.

In one embodiment, in order for the consumer to know in which group or partitions the message corresponding to the priority to be processed is stored when subscribing to the topic, a priority allocation priority (on the consumer side) may also be set to ensure that the consumer only consumes the partition in the group corresponding to the priority to be processed.

Fig. 4 shows a schematic diagram of a priority message queue of the proposed design, and fig. 5 is a UML class diagram of a corresponding code implementation.

The principle of partitioning is to assign more partitions to a high priority group. Various alternative partition allocation policies are provided below in embodiments of the present application:

For example, consider three priorities HIGH > MEDIUM > LOW, designating the fraction of partitions to which they can be assigned as [60%,30%,10% ]. If there are 32 subject partitions, the number of partitions in the subgroup corresponding to the three priorities is [19,10,3] in turn.

Specifically, for N priorities, their priority weights are in turn w ₁ >w ₂ >…>w _N If the total number of partitions is P, the number of partitions that can be allocated to each priority is:

for example, three priorities HIGH > MEDIUM > LOW are still considered, with their priority weights of [5,2,1]. If there are 32 subject partitions, the number of partitions in the subgroup corresponding to the three priorities is [20,8,4] in turn.

For N priorities, the priority weights are w in turn ₁ >w ₂ >…>w _N If the total number of partitions is P, the number of partitions allocated to each priority

For example, consider three priorities HIGH > MEDIUM > LOW, with priority weights of [3,2,1]. If there are 32 subject partitions, the number of partitions in the subgroup corresponding to the three priorities is [21,8,3] in turn.

The following describes in detail the code implementation of the embodiments of the present application.

In one embodiment, in code implementation, each group is represented by a priority group (priority group) object; the member variables of the object include a list storing all subject partition numbers assigned by the group and variables representing the corresponding priority weight values of the group.

Specifically, the program first obtains the total number of available partitions currently in the subject. Then, the proportion is allocated according to the number of the partitions, and the partitions are allocated to the corresponding priority subgroups in sequence from high priority to low priority. If there are more partitions that are not allocated (e.g., zero-bits resulting from rounding when the number of partitions is scaled), the program will allocate partitions to each group one by one from high to low priority by polling to ensure uniform overall partition allocation.

In addition, since Kafka allows partition expansion, the number of partitions can be compared with the number of partitions in the last round of sending or pulling before each message sending or pulling, and if the number is changed, the repartition of the partitions can be triggered, so that the self-adaptive elastic expansion and contraction of the system are realized.

In one aspect, the producer-side implementation of the present solution is described in detail below.

To specify the priority, when constructing a message record (producecrscord), in addition to the topic attribute and the value attribute, a key attribute needs to be specified to indicate the priority to which the message corresponds. In an implementation, the name of the priority is used to indicate which priority, such as "HIGH", "LOW", etc., and thus the type of key is set to java.

To achieve correct writing of the message into the partition under the group to which the priority corresponds, it is critical to reasonably implement the priority partitioner. As shown in fig. 4 and 5, the present application sets a priority partitioner class prioritypartiner for implementing partition routing of message records. The class implements the org.apache.kafka.clients.producer.partner interface, according to the group and partition number to which the priority decision is to be sent, as represented by the value of the key in the message record. Ensuring uniform distribution of messages among partitions in a group, and enabling a partitioner to alternately route messages to different partitions each time by adopting a Round-Robin mode, namely, writing partition number i of an nth message _n ＝(i _n-1 +1) mod P, where P is the total number of partitions.

After the custom priority partitioner (prioritypartiner) class is implemented, this priority partitioner is designated for use by configuring a parameter partitioner class, and a priority producer (priorityproducter) class is created based on this configuration, thereby providing the producer client with priority message queue characteristics to the applicator.

On the other hand, the consumer-side implementation is described in detail below.

To achieve a partition in the group to which the consumer only spends priority to process, it is critical to achieve a partition allocator reasonably. The embodiment of the application sets a priority allocation priority assignment category to realize a partition allocation policy between consumers in the consumer group and subscription topics. This class implements the org.apache.kafka.clients.consumer.abstract partitionassignor interface, which assigns partitions under a corresponding priority subgroup to each consumer in the consumer group by means of polling for the set priorities to be handled by that consumer. Generally, a consumer group is built for each priority, the number of consumer instances in each consumer group should be consistent with the number of partitions in the priority group, and each partition corresponds to one consumer, so as to achieve the maximization of consumption efficiency.

After the custom priority allocation priorityissignor class is implemented, the priority allocation is designated to be used by the configuration parameter partition.

The technical effects of the embodiments of the present application are described below in terms of simulation experiments.

In order to actually verify the priority message queue scheme provided by the embodiment of the application, the embodiment of the application designs a simulation experiment for testing. The test environment used was a cloud host configured with a 2-core CPU, 4G memory, 60G SSD hard disk, kafka version 3.2.0. The created theme has 10 partitions in total and the copy factor is set to 1. We devised three levels of priority HIGH > MEDIUM > LOW, with their rank weights of [60%,30%,10% ], and then the number of partitions that can be allocated is in turn [6,3,1]. At the consumer side, we created one consumer group for each of the three priorities and each consumer consumed one partition on its own, so the number of consumer instances within the three consumer groups was [6,3,1]. At the producer side, the producer sends 10000 pieces of data each of priority HIGH, MEDIUM, LOW.

The comparison of the processing speeds of the three priority messages is shown in fig. 6. As expected, higher priority messages are processed faster. And the processing speeds of the three priority messages are approximately 6:3:1, which indicates that the design scheme of the proposal can well adjust the processing capacity of the messages with different priorities according to the designated priority weight.

In the description of the present specification, descriptions with reference to the terms "some possible embodiments," "some embodiments," "examples," "specific examples," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiments or examples is included in at least one embodiment or example of the present application, and that the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the various embodiments or examples described in this specification and the features of the various embodiments or examples may be combined and combined by those skilled in the art without contradiction.

With respect to the method flowcharts of the embodiments of the present application, certain operations are described as distinct steps performed in a certain order. Such a flowchart is illustrative and not limiting. Some steps described herein may be grouped together and performed in a single operation, or may be divided into multiple sub-steps and may be performed in an order different than that shown herein. The various steps illustrated in the flowcharts may be implemented in any manner by any circuit structure and/or tangible mechanism (e.g., by software running on a computer device, hardware (e.g., processor or chip implemented logic functions), etc., and/or any combination thereof).

It will be appreciated by those skilled in the art that in the methods described in the above embodiments, the written order of steps does not imply a strict order of execution, and that the specific order of execution of the steps should be determined by its function and possible inherent logic.

It should be noted that, the apparatus in the embodiments of the present application may implement each process of the foregoing method embodiments and achieve the same effects and functions, which are not described herein again.

According to some embodiments of the present application, there is provided a priority message queue implementing apparatus according to an embodiment of the present application, for performing the priority message queue implementing method shown in fig. 3, the apparatus including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described in the above embodiments.

According to some embodiments of the present application, there is provided a non-transitory computer storage medium having stored thereon computer executable instructions configured to, when executed by a processor, perform: the method of the above embodiment.

Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of random access memory, read only memory, electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required to either imply that the operations must be performed in that particular order or that all of the illustrated operations be performed to achieve desirable results. In addition, some steps may be omitted, multiple steps may be combined into one step to be performed, and/or one step may be decomposed into multiple sub-steps to be performed.

While the spirit and principles of the present application have been described above with reference to several embodiments, it should be understood that the application is not limited to the particular embodiments disclosed nor does the division of aspects mean that features in these aspects cannot be combined. The application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for implementing a priority message queue, comprising:

dividing partitions under the Kafka message subject into a plurality of subgroups, wherein each subgroup corresponds to a priority, and the number of partitions allocated in each subgroup is proportional to the priority;

processing resources are allocated according to the priority of each group.

2. The method of claim 1, wherein partitioning the partitions under the Kafka message subject into a plurality of subgroups comprises:

and distributing the corresponding partition numbers to each subgroup according to the partition proportion corresponding to each priority and the total partition number under the Kafka theme.

3. The method of claim 1, wherein partitioning the partitions under the Kafka message subject into a plurality of subgroups, further comprising:

and according to the priority weights of the priorities and the total number of the partitions, corresponding partition numbers are distributed to the subgroups according to the linear proportion.

4. The method of claim 1, wherein the corresponding number of partitions is assigned to each subgroup in linear proportion using the formula:

5. The method of claim 1, wherein partitioning the partitions under the Kafka message subject into a plurality of subgroups, further comprising:

and distributing the corresponding partition numbers to each subgroup according to the priority weights of the priorities and the total number of the partitions in an exponential proportion.

6. The method of claim 1, wherein the respective sub-groups are exponentially proportional to the number of partitions using the formula:

7. The method as recited in claim 1, further comprising:

in a code implementation, each subgroup is represented by a priority subgroup object;

the member variables of the object include a list storing all subject partition numbers assigned by the group and variables representing the corresponding priority weight values of the group.

8. The method of claim 1, wherein assigning a corresponding number of partitions to each group further comprises:

sequentially distributing the partitions to corresponding priority subgroups from high priority to low priority;

if there are remaining partitions that are not allocated, the partitions are assigned to the groups one by one from high to low in priority by polling.

9. The method as recited in claim 1, further comprising:

setting a priority partitioner for determining a partition number to be sent according to priority information represented by a key value carried in a message record;

the message record specifies a key value for indicating the priority corresponding to the message.

10. The method as recited in claim 1, further comprising:

the priority allocation policy is set to ensure that consumers consume only the partitions in the group to which the priority to be processed corresponds.

11. The method as recited in claim 1, further comprising:

the priority partitioner adopts a polling mode to route the message records to different partitions in turn;

the priority allocation priority assignment is used for allocating the partitions under the corresponding priority subgroups to each consumer in the consumer group by means of polling.

12. The method as recited in claim 1, further comprising:

designating use of the priority partitioner by a first configuration parameter;

a priority producer class is created based on the first configuration information to provide to an producer client having a priority message queue characteristic to an applicator.

13. The method as recited in claim 1, further comprising:

designating use of the priority allocation priority by a second configuration parameter;

creating a priority consumer class based on the second configuration parameters, and providing the consumer client with priority message queue characteristics to an application.

14. A priority message queue implementing apparatus, comprising:

at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform: the method of any one of claims 1-13.

15. A computer readable storage medium storing a program which, when executed by a multi-core processor, causes the multi-core processor to perform the method of any of claims 1-13.