WO2021174382A1

WO2021174382A1 - Message pulling method and device, and computer storage medium

Info

Publication number: WO2021174382A1
Application number: PCT/CN2020/077411
Authority: WO
Inventors: 郭子亮
Original assignee: 深圳市欢太科技有限公司; Oppo广东移动通信有限公司
Priority date: 2020-03-02
Filing date: 2020-03-02
Publication date: 2021-09-10
Also published as: CN115211092B; CN115211092A

Abstract

Disclosed in embodiments of the present application are a message pulling method and device, and a computer storage medium. The method comprises: determining a plurality of subtopics comprised in a topic to be consumed and first pulling proportions respectively assigned to the subtopics, wherein different subtopics have different priority parameters; determining, according to the first pulling proportions, an actual pulling quantity of messages pulled by said topic; if the actual pulling quantity of the messages pulled by said topic is less than a preset rated pulling quantity, adjusting the first pulling proportions to obtain second pulling proportions respectively assigned to the plurality of subtopics; and pulling messages for said topic according to the second pulling proportions.

Description

Message pulling method, device and computer storage medium

Technical field

The embodiments of the present application relate to the field of data application technology, and in particular, to a method, device, and computer storage medium for pulling messages.

Background technique

Kafka is a high-throughput distributed publish-subscribe messaging system. Each message published to the Kafka cluster has a topic, which is called Topic. Specifically, multiple subtopics with different priorities can be created for the same topic, each subtopic is configured with a consumer group, and the proportion of the number of messages corresponding to different priority levels of different consumer groups can be calculated according to the default or user-defined configuration. The maximum pull amount of different priorities.

However, considering that there may be no message accumulation in one or several subtopics among multiple subtopics, the amount of messages pulled by consumers at this time does not meet the maximum pull amount allocated during initialization. If there are news accumulations in some subtopics, but the news is still pulled at a fixed ratio, the amount of news pulled by consumers will be less than the maximum amount, which will give users the illusion of abnormality; in addition, when it is lower When messages are piled up in priority and there is no message with higher priority, if the lower priority still pulls messages at a low quota ratio, it will cause consumers to frequently pull small batches of messages, which will increase the disk IO load, and The performance advantage brought by the pre-reading of the operating system is reduced, resulting in a performance bottleneck in the Kafka cluster.

Summary of the invention

This application provides a method, device, and computer storage medium for pulling messages. Through an adaptive dynamic quota method, the flow control rate of each subtopic can be allocated and updated in real time, and it can avoid the frequent small batches of messages being pulled by consumer groups. The computing rate drops and the performance bottleneck of Kafka cluster.

The technical solutions of the embodiments of the present application can be implemented as follows:

In the first aspect, an embodiment of the present application provides a message pulling method, which includes:

Determine the multiple sub-themes included in the theme to be consumed and the first pull ratio assigned to each; among them, different sub-themes have different priority parameters;

According to the first pull ratio, determine the actual pull amount of messages pulled by the topic to be consumed;

If the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, the first pull ratio is adjusted to obtain the second pull ratio assigned to each of the multiple subtopics ；

According to the second pulling ratio, message pulling is performed on the topic to be consumed.

In a second aspect, an embodiment of the present application provides a message pulling device, the message pulling device includes a determining unit, an adjusting unit, and a pulling unit, wherein:

The determining unit is configured to determine a plurality of sub-themes included in the theme to be consumed and a first pull ratio allocated to each; wherein, different sub-themes have different priority parameters;

The determining unit is further configured to determine the actual pull amount of messages pulled by the topic to be consumed according to the first pull ratio;

The adjustment unit is configured to, if the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, adjust the first pull ratio to obtain each of the multiple subtopics The allocated second pull ratio;

The pulling unit is configured to pull messages of the topic to be consumed according to the second pulling ratio.

In a third aspect, an embodiment of the present application provides a message pulling device, the message pulling device includes a memory and a processor, wherein:

The memory is used to store a computer program that can run on the processor;

The processor is configured to execute the method described in the first aspect when the computer program is running.

In a fourth aspect, an embodiment of the present application provides a computer storage medium that stores a computer program that implements the method described in the first aspect when the computer program is executed by at least one processor.

The embodiments of the present application provide a message pull method, device, and computer storage medium, which determine the multiple subtopics included in the topic to be consumed and the first pull ratio assigned to each; wherein, different subtopics have different priorities. Level parameter; according to the first pull ratio, determine the actual pull amount of messages pulled by the topic to be consumed; if the actual pull amount of messages pulled by the topic to be consumed is less than the preset rated pull Then, the first pull ratio is adjusted to obtain the second pull ratio assigned to each of the multiple subtopics; according to the second pull ratio, message pull is performed on the topic to be consumed. In this way, dynamic flow control is carried out through the adaptive dynamic quota method. When there is no message accumulation in some subtopics, it can avoid the problem of the decrease in the calculation rate and the performance bottleneck of the Kafka cluster caused by the frequent small batches of messages pulled by the consumer group. Optimization of computing performance; In addition, in the case of a subtopic burst of large-flow message writing, it can also allocate and update its flow control rate in real time, thereby achieving optimal consumption performance.

Description of the drawings

FIG. 1 is a schematic flowchart of a message pulling system provided by related technical solutions;

Fig. 2 is a schematic diagram of a scenario for pulling messages provided by related technical solutions;

FIG. 3 is a schematic flowchart of a method for pulling messages according to an embodiment of the application;

FIG. 4 is a schematic diagram of a scenario in which a preset reduction mode is applied according to an embodiment of the application;

FIG. 5 is a schematic diagram of an application scenario of a preset solicitation mode provided by an embodiment of the application;

FIG. 6 is a schematic diagram of the composition structure of a message pulling device provided by an embodiment of the application;

FIG. 7 is a schematic diagram of the composition structure of another message pulling device provided by an embodiment of the application;

FIG. 8 is a schematic diagram of the hardware structure of a message pulling device provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. It is understandable that the specific embodiments described here are only used to explain the related application, but not to limit the application. In addition, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the drawings.

Message middleware is a kind of software widely used in distributed systems for communication between nodes. In the large-scale high-concurrency back-end service architecture system, Kafka, as a commonly used message middleware, is widely used.

Kafka is a message middleware developed and open sourced by Linkedin. It is a high-throughput distributed publish-subscribe messaging system that can process all the action flow data of consumers on the website. Among them, the main goal of kafka development is to build a data processing framework for processing massive logs, user behavior and website operation statistics. In the case of combining data mining, behavior analysis, operation monitoring and other requirements, it needs to be able to meet various Real-time online and batch offline processing applications require low latency and batch throughput performance, and real-time messages can be provided through clusters.

A Kafka cluster contains one or more servers. This server is called a broker. Each message published to the Kafka cluster has a topic. The topic is called a topic. Each topic contains one or more subtopics. Refer to FIG. 1, which shows an example of a flow diagram of a message pulling system provided by related technical solutions; as shown in FIG. 1, the message pulling system 10 includes a message producer 101, a Kafka cluster 102, and a client 103 And the consumer 104; where the client 103 is usually a software development kit (SDK) provided by a third-party developer, that is, the SDK provides developers with quick access to the adapted client provided by the middleware. In this way, after the message producer 101 generates a message, the priority parameter of the message can be configured at the same time; for example: message1{priority:2; data:A3615C}, message2{priority:5; data:B36D4}, etc.; That is, the priority message logic is encapsulated in the client 103, and then subtopics (topic-1, topic-2, topic-3, topic-4 and topic -5, etc.) to simulate the writing of messages with different priority parameters; after the message is written to the subtopic of the corresponding priority, the client 103 is then used to pull messages from different subtopics in the Kafka cluster 102, such as the pull The messages are message1 (priority: 2; data: A3615C) and message 2 (priority: 5; data: B36D4). Finally, the pulled message is sent to the consumer 104, and the consumer 104 performs related processing on the pulled message .

Specifically, each subtopic corresponding to a priority parameter is configured with a consumer group, and different consumer groups are pulled according to the default or user-defined configuration of different priority parameters corresponding to the pull ratio, multiplied by the rating of the messages pulled by all priorities By the amount of pull, you can calculate the benchmark quotas allocated by different priorities, and then summarize the messages pulled by different subtopics through the supporting client SDK, and return to the user a batch of pulled messages, the batch of messages It contains the amount of pulled messages corresponding to different priority parameters, so as to realize Kafka's support for messages pulled by different priorities.

Considering that for the subtopics corresponding to multiple priority parameters, there may be no message accumulation for one or several subtopics at a specific moment. At this time, the actual pull amount of messages pulled by consumers does not meet the rated pull allocated during initialization quantity. As shown in FIG. 2, it shows a schematic diagram of a scenario of a message pulling application provided by related technical solutions. In Figure 2, in the five subtopics of topic-1, topic-2, topic-3, topic-4 and topic-5 included in the same topic, the priority parameter includes 5 levels, and the priority parameter is equal to 5. It means the highest priority; the priority parameter is equal to 1, which means the lowest priority; and there is message accumulation in topic-1 and topic-5, but topic-2, topic-3 and topic-4 have no message accumulation; assuming the default rated pull The amount is 500, and the corresponding pull ratios for different priority parameters (from high to low) are: 40%, 30%, 15%, 10%, 5%; specifically, topic-5 pull quota It is 200, that is, the subtopic with priority parameter equal to 5 can pull 200 messages each time; and topic-4 has a pull quota of 150, topic-3 has a pull quota of 75, and topic-2 has a pull quota. It is 50, and the pull quota for topic-1 is 25; that is, only messages can be pulled from topic-5 and topic-1, and the actual pull total is 225 messages. In this way, based on the scenario example shown in Figure 2, when there are subtopics with message accumulation, the user is expected to be able to pull 500 messages at a time, but due to the pull ratio restrictions corresponding to different priorities, the user actually only 225 messages were pulled. In this way, there will be the following problems: On the one hand, when there is a message accumulation in some subtopics, but the actual pull amount of the messages pulled by the SDK is less than the rated pull amount set by the user, this will cause the user to exist. Abnormal illusion; on the other hand, if there is a accumulation of messages with a lower priority and no messages with a higher priority (in idle state), then the lower priority still pulls messages at a low quota ratio, which may cause consumers to pull frequently Fetch small batches of messages; and frequent small batches of messages will cause a large number of random reads, which will lead to an increase in disk IO load, and reduce the performance advantage brought by the operating system read-ahead, which will cause a performance bottleneck in the Kafka cluster.

Based on this, an embodiment of the present application provides a message pull method by determining multiple subtopics included in the topic to be consumed and a first pull ratio allocated to each; wherein, different subtopics have different priority parameters; According to the first pull ratio, determine the actual pull amount of messages pulled by the topic to be consumed; if the actual pull amount of messages pulled by the topic to be consumed is less than the preset rated pull amount, then The first pull ratio is adjusted to obtain the second pull ratio assigned to each of the multiple subtopics; and the message pull is performed on the topic to be consumed according to the second pull ratio. In this way, dynamic flow control is carried out through the adaptive dynamic quota method. When there is no message accumulation in some subtopics, it can avoid the problem of the decrease in the calculation rate and the performance bottleneck of the Kafka cluster caused by the frequent small batches of messages pulled by the consumer group. Optimization of computing performance; In addition, in the case of a subtopic burst of large-flow message writing, it can also allocate and update its flow control rate in real time, thereby achieving optimal consumption performance.

Hereinafter, each embodiment of the present application will be described in detail with reference to the accompanying drawings.

In an embodiment of the present application, refer to FIG. 3, which shows a schematic flowchart of a message pulling method provided in an embodiment of the present application. As shown in Figure 3, the method may include:

S301: Determine a plurality of sub-themes included in the theme to be consumed and their respective first pull ratios; wherein, different sub-themes have different priority parameters;

It should be noted that this method is applied to a message pull system. In the message pull system, the SDK pulls messages from topics to be consumed in the kafka cluster. Among them, the same topic to be consumed can create multiple subtopics, and these multiple subtopics have different priority parameters.

It should also be noted that the first pull ratio is the default pull ratio set in advance according to different priority parameters; here, the first pull ratio indicates that the preset number of messages to be pulled for each subtopic is relative to the total amount to be consumed. The proportion of the rated pull volume preset by the theme. For example, suppose that multiple subtopics can include five subtopics: topic-5, topic-4, topic-3, topic-2, and topic-1; and the priority parameters are 5, 4, 3, 2, 1, and 5. The priority is the highest, and 1 corresponds to the lowest priority. At this time, the first pull ratio can be set to 40%, 30%, 15%, 10%, and 5%.

In addition, among these multiple subtopics, it is possible that some subtopics store messages to be consumed, while some subtopics do not store messages to be consumed. For the messages to be consumed stored in each subtopic, they are produced by the message producer, and then put into the corresponding subtopic according to the priority parameter. Specifically, in some embodiments, before S301, the method may further include:

Receiving a message to be consumed sent by a message producer, where the message to be consumed includes message content and priority parameters;

The received messages to be consumed are saved in the subtopics corresponding to the priority parameters, and the accumulation of messages corresponding to each subtopic is obtained.

That is to say, after the message producer produces the message to be consumed, the priority parameter corresponding to the message to be consumed will also be configured; in this way, after receiving the message to be consumed sent by the message producer through the Kafka cluster, it can be based on the priority Level parameters, save the received messages to be consumed in the corresponding subtopics, so as to obtain the accumulation of messages for each subtopic.

Among them, if there are more messages with a certain priority parameter produced by the message producer, then in the subtopic corresponding to the priority parameter, the amount of message accumulation will be greater; if the message producer with a certain priority parameter produces more messages If there is less or no, then the message accumulation amount in the subtopic corresponding to the priority parameter is smaller, and even the message accumulation amount is equal to 0.

S302: According to the first pull ratio, determine the actual pull amount of messages pulled by the topic to be consumed;

It should be noted that after the first pull ratio is obtained, the message can be pulled according to the first pull ratio, so as to determine the actual pull amount of each subtopic in the topic to be consumed, so as to determine the topic to be consumed The actual amount of messages pulled. Specifically, in some embodiments, for S302, the determining the actual pull amount of messages pulled by the topic to be consumed according to the first pull ratio may include:

According to the first pull ratio, perform message pull on the message accumulation amount corresponding to each subtopic, and obtain the actual pull amount of messages pulled by each subtopic;

The actual pull amount of messages pulled by each subtopic is accumulated to obtain the actual pull amount of messages pulled by the topic to be consumed.

That is to say, after the first pull ratio is obtained, the message accumulation amount corresponding to each subtopic can be pulled according to the first pull ratio, and the actual pull amount of the messages pulled by each subtopic can be obtained. ; Then the actual pull amount of messages pulled by each subtopic is accumulated to obtain the actual pull amount of messages pulled by the topic to be consumed.

S303: If the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, adjust the first pull ratio to obtain the second pull allocated by each of the multiple subtopics. Take the ratio

It should be noted that when the accumulation of messages corresponding to one of the multiple subtopics is equal to 0, then the actual pull volume of the messages pulled by the subtopic can be obtained as 0; at this time, the messages pulled by the topic to be consumed The actual pull amount of will be less than the preset rated pull amount, that is, the method may also include: if the message accumulation amount corresponding to one of the subtopics is equal to 0, then the first pull ratio is adjusted to obtain The second pull ratio assigned to each of the multiple subtopics.

Further, in some embodiments, the method may further include:

Determine the preset quota corresponding to each of the multiple subtopics based on the preset rated pull amount and the first pull ratio;

Comparing the actual pull amount of messages pulled by each subtopic of the plurality of subtopics with the preset quota corresponding to each subtopic;

If there is at least one subtopic whose actual pull amount is less than the preset quota among the multiple subtopics, it is determined that the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount.

That is to say, according to the preset rated pull amount and the first pull ratio, the preset quota (also referred to as base station quota) corresponding to each of the multiple subtopics can be determined. Specifically, it may include: for the multiple subtopics, respectively multiplying the first pull ratio corresponding to each subtopic by the preset rated pull amount to obtain the preset quota corresponding to each subtopic , Thereby determining the preset quota corresponding to each of the multiple subtopics.

Here, the preset quota corresponding to each subtopic represents the preset number of messages to be pulled when each subtopic is pulled. After calculating the preset quota corresponding to each subtopic, the actual pull amount of messages pulled by each subtopic can be compared with the preset quota corresponding to each subtopic; when there are actual pulls in multiple subtopics If the amount of at least one subtopic is less than the preset quota, it is determined that the actual amount of messages pulled by the topic to be consumed is less than the preset rated amount.

In this way, when the actual pull amount of messages pulled by the topic to be consumed is less than the preset rated pull amount, it indicates that the first pull ratio needs to be dynamically adjusted at this time, so that each priority parameter corresponds to the subtopic The allocated pull quota is changed; the allocation ratio corresponding to the transformed pull quota is the second pull ratio, so in the next message pull, the message can be pulled according to the second pull ratio.

In other words, this adaptive adjustment of the pull ratio allocated by different priority parameters can be called a dynamic quota. Specifically, by dynamically adjusting the pull quotas (or pull ratios) of different priorities, dynamic flow control can be achieved; that is, under a given total flow control rate, the flow control rate of each priority is dynamically allocated, such as when a certain When there is no message accumulation in a priority queue, or when the message accumulation amount is equal to 0, its flow control rate (that is, the pull quota) can be dynamically allocated to other priorities, that is, the pull ratio can be adjusted to achieve the total Optimal consumption rate.

It should also be noted that the preset reduction mode and/or the preset collection mode can be used to implement dynamic quota/dynamic flow control. Here, the preset reduction mode can be regarded as active reduction, and the preset collection mode can be regarded as passive collection.

Optionally, in some embodiments, for S303, the adjusting the first pull ratio may include:

The preset reduction mode is used to reduce the preset quota corresponding to the sub-topic to be reduced among the multiple sub-topics, and the obtained reduction amount is allocated to the sub-topic corresponding to the next priority parameter to achieve the first reduction. Proportion adjustment; wherein, the sub-topics to be reduced represent sub-topics of the multiple sub-topics whose actual pull amount is less than a preset quota.

Use the preset collection mode to collect the preset quotas corresponding to the multiple subtopics, and allocate the collected amount to the subtopics to be expanded, so as to realize the adjustment of the first pull ratio; wherein, the waiting Expanding a sub-theme represents a sub-theme whose actual pull amount is equal to a preset quota among the multiple sub-themes.

That is to say, in the preset reduction mode, when the actual pull amount of the message pulled for the subtopic corresponding to a certain priority parameter is insufficient, it will actively release its own quota to other priorities to achieve the first pull. Adjust the proportion; and in the default solicitation mode, when the actual amount of messages pulled for a subtopic corresponding to a priority parameter meets the preset quota (benchMarkAllocation), if there is an idle quota at this time, such as in The priority parameter of the BackPress state corresponds to the amount of quota actively reduced by the sub-theme. At this time, the sub-theme corresponding to the priority parameter can collect this part of the quota to achieve the adjustment of the first pull ratio; among them, benchMarkAllocation represents the calculated benchmark quota based on the pull ratio allocated by default or user-defined pre-configured different priority parameters; BackPress state represents a flow control method, when the actual pull of the message pulled by a certain subtopic The amount is lower than the benchmark quota, which indicates that the sub-topic is under pressure.

S304: According to the second pull ratio, pull messages for the topic to be consumed.

In this way, after adjusting the pull ratios corresponding to different priority parameters, the second pull ratio can be obtained. At this time, if the user triggers the SDK to pull messages again, they can pull messages for the topic to be consumed according to the second pull ratio, so as to obtain the actual pull amount of messages pulled by the topic to be consumed again. In this way, after S304, the method may further include:

If the obtained actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, continue to adjust the second pull ratio to obtain the third pull allocated by each of the multiple subtopics Take the ratio

According to the third pull ratio, message pull is performed on the topic to be consumed.

In other words, as long as the actual pull amount of messages pulled by the topic to be consumed is less than the preset rated pull amount, you need to continue to adjust the current pull ratio, and then the next time the user triggers the SDK to pull messages When fetching, the message will be fetched according to the adjusted new fetching ratio; in order to realize the dynamic adjustment of the fetching ratio, the dynamic quota for multiple subtopics corresponding to different priority parameters is also realized.

It should be noted that in the message pulling method in the embodiment of the present application, in addition to kafka, dynamic quota or dynamic flow control methods can also be applied to other middleware with dynamic flow control requirements, so as to obtain optimal consumption performance. .

This embodiment provides a method for pulling messages by determining multiple subtopics included in the topic to be consumed and a first pull ratio allocated to each; wherein, different subtopics have different priority parameters; A pull ratio determines the actual pull amount of the message pulled by the topic to be consumed; if the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, the second A pull ratio is adjusted to obtain a second pull ratio assigned to each of the multiple subtopics; according to the second pull ratio, message pull is performed on the topic to be consumed. In this way, dynamic flow control is carried out through the adaptive dynamic quota method. When there is no message accumulation in some subtopics, it can avoid the problem of the decrease in the calculation rate and the performance bottleneck of the Kafka cluster caused by the frequent small batches of messages pulled by the consumer group. Optimization of computing performance; In addition, in the case of a subtopic burst of large-flow message writing, it can also allocate and update its flow control rate in real time, thereby achieving optimal consumption performance.

In another embodiment of the present application, taking the preset reduction mode as an example, for S303, the adjusting the first pull ratio may include:

If the actual pull amount corresponding to one of the multiple subtopics is less than the preset quota, then the one of the subtopics is regarded as the subtopic to be reduced, and the preset quota corresponding to the subtopic to be reduced is performed Reduce to get the remaining preset quota;

When the remaining preset quota is not lower than the preset minimum quota, calculating the difference between the preset quota corresponding to the subtopic to be reduced and the remaining preset quota to obtain the available quota;

Decrease the priority parameters in order, and use the subtopic corresponding to the reduced priority parameter as the first subtopic;

Determining whether the preset quota corresponding to the first sub-topic is greater than a maximum quota threshold and whether the first sub-topic is in a full-speed processing state;

When the preset quota corresponding to the first sub-topic is not greater than the maximum quota threshold and the first sub-topic is in a full-speed processing state, the available quota is allocated to the first sub-topic, and the first sub-topic is updated. The preset quota corresponding to the subtopic;

When the preset quota corresponding to the first sub-topic is greater than the maximum quota threshold and the first sub-topic is in a non-full-speed processing state, continue to execute the lower priority parameter in order, and change the child corresponding to the lower priority parameter The subject is used as the first sub-topic until the available quota is allocated to the sub-topic corresponding to the lowest priority parameter, so as to realize the adjustment of the first pull ratio.

It should be noted that the "order lower priority parameter" indicates that the current subtopic corresponding to priority 4, after the execution order reduces the priority parameter, the subtopic corresponding to the reduced priority parameter is the subtopic corresponding to priority 3; After the priority parameter is reduced in the execution order again, the subtopic corresponding to the reduced priority parameter is the subtopic corresponding to priority 2, and so on, when all priority parameters are in the non-full-speed processing state, until the available quota is allocated To the subtopic corresponding to the lowest priority parameter.

It should also be noted that the maximum quota threshold can be expressed as a highWaterMark limit, which is mainly to avoid dynamic quotas causing all preset rated pulls to be centrally allocated to the subtopics corresponding to specific priority parameters. At this time, it is necessary to set each priority Parameter configuration maximum pull limit. The full-speed processing state can be represented by the FullProgress state, which means that the actual pull amount after the dynamic quota of the current subtopic is greater than or equal to the benchmark quota, which means that the current subtopic is consuming messages at full speed.

In this way, in terms of the preset reduction mode, each time a message is pulled, its own quota can be reduced for the subtopics corresponding to the priority parameters whose actual pull amount is less than the preset allocation amount. Here, the reduced quota amount can be set according to the actual situation, such as cutting the quota amount by half; then the difference between the quota of the sub-topic itself and the remaining quota of the sub-topic is calculated, and the obtained difference is used as the available The quota is allocated, and quotas are allocated only to the subtopics corresponding to the second-level priority parameters. If the subtopics corresponding to the second-level priority parameters reach the maximum quota threshold (highWaterMark limit), the excess quotas can continue Assign the subtopics corresponding to the lower priority parameters. In addition, the priority parameters subject to quotas need to be in the full progress state. If all priority parameters are not in the full progress state, the available quotas will eventually be aggregated into the subtopics corresponding to the lowest priority parameters, and the lowest priority Do not do highWaterMark restrictions.

Furthermore, the quota reduction for the sub-topic to be reduced will not continue to be reduced indefinitely. When the dynamic quota corresponding to the sub-topic to be reduced is lower than the preset minimum quota, the quota reduction for the sub-topic to be reduced will be stopped; When the dynamic quota corresponding to all the subtopics to be reduced is lower than the preset minimum quota, the dynamic adjustment of the pull ratio will be stopped. Therefore, in some embodiments, the method may further include:

After each message pull on the topic to be consumed, update the first pull ratio by using the second pull ratio;

Pull messages for the topic to be consumed according to the updated first pull ratio, and obtain the updated actual pull amount of the messages pulled by the topic to be consumed;

If the actual pull amount after the update of the message pulled by the topic to be consumed is less than the preset rated pull amount, continue to adjust the first pull ratio to obtain the second pull ratio, and return The step of using the second pulling ratio to update the first pulling ratio;

After multiple message pulls are performed on the topic to be consumed, when the dynamic quotas corresponding to the subtopics to be reduced all reach the preset minimum quota, the step of adjusting the first pull ratio is stopped.

In other words, after each message is pulled for the topic to be consumed, the second pull ratio can be used to update the first pull ratio; in this way, the next time the client SDK is triggered to pull messages, the updated first pull ratio can be used. Pull the message with a pull ratio, and then obtain the updated actual pull amount of the message pulled by the topic to be consumed; if the obtained actual pull amount of the message pulled from the topic to be consumed is less than the preset rated pull If you take the amount, then you need to continue to adjust the first pull ratio.

In this way, after multiple message pulls, when the dynamic quotas corresponding to the subtopics to be reduced all reach the preset minimum quota, it indicates that the dynamic quotas corresponding to the subtopics to be reduced have all reached the low water limit. Stop adjusting the first pull ratio, and when you pull the message again by triggering the client SDK later, you can pull the message according to the pull ratio after the latest update. It should also be noted that the preset minimum quota is mainly to prevent the quota corresponding to the subtopics without message accumulation from being automatically reduced to 0, and the lower limit of the quota is set. Here, the preset minimum quota can be specifically set according to actual conditions. Generally, one-sixteenth of the preset quota configured by default (or referred to as a reference quota) can be used as the preset minimum quota, but it is not specifically limited.

Refer to FIG. 4, which shows a schematic diagram of an application scenario of a preset reduction mode provided by an embodiment of the present application. In FIG. 4, in addition to the Kafka cluster 102, the client 103, and the consumer 104, the message pulling system 10 may also include an internal state maintainer 105; here, the internal state maintainer 105 may be a state window (window) . Specifically, in the five subtopics of topic-1, topic-2, topic-3, topic-4, and topic-5 included in the same topic, the priority parameter includes 5 levels, and the priority parameter is equal to 5, indicating priority The priority is the highest; the priority parameter is equal to 1, which means the priority is the lowest; and there is a message accumulation in topic-1 and topic-5, but there is no message accumulation in topic-2, topic-3 and topic-4; assuming the default rated pull It is 500. The default pull ratios corresponding to different priority parameters (from high to low) are: 40%, 30%, 15%, 10%, 5%; specifically, the preset quota of topic-5 Is 200, the default quota for topic-4 is 150, the default quota for topic-3 is 75, the default quota for topic-2 is 50, and the default quota for topic-1 is 25; Pull messages from topic-5 and topic-1; at this time, use the preset reduction mode, topic-2, topic-3, and topic-4 as the subtopics to be reduced; each time a message is pulled, due to the actual pull amount If the quota is lower than the preset quota, half of the quota will be actively cut for one of the sub-topics to be reduced, and the reduced quota will be allocated to the sub-topics corresponding to the sub-priority parameter in turn; if the sub-priority parameter has no message accumulation, it will continue downward The first-level priority parameter allocation will eventually fall to the lowest priority to achieve the adjustment of the pull ratio; after multiple pull messages, topic-2, topic-3, and topic-4 are three subtopics to be reduced. After the dynamic quota is reduced to 1/16 of its default quota, the updated quotas for topic-2, topic-3, and topic-4 are: 3, 4, and 9; and topic-2, topic-3, and topic- 4 The released quota can be used for topic-1 and topic-5, that is, it can be used as the solicitable quota for topic-1 and topic-5; here, the solicitable quota for topic-1 and topic-5 is 259.

If the actual pull amount corresponding to one of the multiple subtopics is equal to the preset quota, then the one of the subtopics is regarded as the subtopic to be expanded, and the corresponding amount of the subtopic to be expanded is increased according to the preset increase amount. Preset quota;

Regarding the preset increase amount, the sub-topic corresponding to the lowest priority is taken as the second sub-topic;

Determine whether the second sub-topic is in a full-speed processing state;

When the second sub-topic is in a full-speed processing state, perform quota collection from the second sub-topic;

If the collected quota amount is less than the preset increase amount, calculate the difference between the preset increase amount and the collected quota amount;

For the calculated difference, the priority parameters are increased in order, the subtopic corresponding to the increased priority parameter is regarded as the second subtopic, and the step of determining whether the second subtopic is in a full-speed processing state is continued. , Until the solicited quota amount is equal to the preset increase amount, so as to realize the adjustment of the first pull ratio.

Further, after determining whether the second sub-topic is in a full-speed processing state, the method may further include:

When the second sub-topic is in the non-full-speed processing state, the priority parameters are sequentially increased, the sub-topic corresponding to the increased priority parameter is taken as the second sub-topic, and the judgment of the second sub-topic is continued. Whether it is in the full-speed processing state.

It should be noted that the "increase priority parameter in order" means that it is currently the subtopic corresponding to priority 2. After the priority parameter is increased in the execution order, the subtopic corresponding to the increased priority parameter is the one corresponding to priority 3. Sub-topic; after the priority parameter is increased in the execution order again, the sub-topic corresponding to the increased priority parameter is the sub-topic corresponding to priority 4, and so on, the solicited quota amount is equal to the preset increase amount .

It should also be noted that, to collect quotas from the second sub-topic, first calculate the amount of quotas that can be collected by the second sub-topic, that is, the quota added to the priority parameter due to the active reduction of other priority parameters; The current dynamic quota of the second subtopic minus the preset quota, the difference is the amount of quota that can be collected for the second subtopic; then the amount of quota that can be collected is compared with the preset increase; when the amount of quota collected is less than When the preset increase amount is calculated, the difference between the preset increase amount and the collected quota amount is calculated, and the difference is recursively collected from the subtopic corresponding to the higher priority parameter; when the collected quota amount is not less than the preset increase In the case of quota, the preset increase amount can be solicited from this sub-theme; then the quota solicitation is continued for other sub-themes to be expanded.

In this way, when there is no message accumulation for a specific priority parameter, and then the message accumulation situation caused by message production, then the subtopic corresponding to the priority parameter of the previously reduced quota needs to be supplemented with quota. Specifically, it can be that the actual amount of messages pulled from the subtopic corresponding to a certain priority parameter by the consumer group this time is equal to the preset quota currently dynamically allocated for it, then the quota for the subtopic can be updated to It doubles the pull quota, that is, the preset increase amount can be the current preset quota of the subtopic; and the preset increase amount (which can be represented by A) needs to be collected from the subtopics corresponding to other priority parameters, and the solicitation strategy In order to collect priority from the subtopic corresponding to the lowest priority parameter; for the subtopic corresponding to the lowest priority parameter, it is necessary to calculate the quota expansion of the subtopic (that is, the amount of quota expansion added to this priority parameter due to the active reduction of other priority parameters) Quota), if the current quota expansion amount B is lower than the preset increase amount A, the remaining part (AB) is recursively collected from a higher priority until the required preset increase amount is reached. Here, the subtopics corresponding to the high priority parameters collected must be in the Full Progress state, and the amount of quotas that can be collected is not only limited by the highWaterMark corresponding to the current priority parameter, but also limited by the quota expansion corresponding to other priority parameters. , It is also necessary to keep the total amount of dynamic quotas consistent.

Furthermore, the quota expansion for the sub-topic to be expanded will not continue to expand indefinitely. When the dynamic quota corresponding to the sub-topic to be expanded is higher than the preset maximum quota, the quota expansion of the sub-topic to be expanded will be stopped; When the dynamic quota corresponding to all the subtopics to be reduced is higher than the preset minimum quota, the dynamic adjustment of the pull ratio will be stopped. Therefore, in some embodiments, the method may further include:

After multiple message pulls for the topic to be consumed, when the dynamic quotas corresponding to the subtopics to be expanded all reach the preset maximum quota, the step of adjusting the first pull ratio is stopped.

In this way, after multiple message pulls, when the dynamic quotas corresponding to the subtopics to be expanded all reach the preset maximum quota, it indicates that the dynamic quotas corresponding to the subtopics to be expanded have reached the high water limit. Stop adjusting the first pull ratio, and when you pull the message again by triggering the client SDK later, you can pull the message according to the pull ratio after the latest update. It should also be noted that the preset maximum quota is mainly used to prevent the passive collection of quotas corresponding to subtopics with accumulated messages and then concentrate all the quotas on the subtopics corresponding to specific priority parameters. Here, the preset maximum quota can be specifically set according to actual conditions. Generally, for the lowest priority parameter, the preset maximum quota can be twice the preset quota (or referred to as the benchmark quota) configured by default; and For other priority parameters, the preset maximum quota can be eight times of its own benchmark quota, but it is not specifically limited.

Refer to FIG. 5, which shows a schematic diagram of an application scenario of a preset solicitation mode provided by an embodiment of the present application. In FIG. 5, in addition to the Kafka cluster 102, the client 103, and the consumer 104, the message pulling system 10 still includes an internal state maintainer 105. Specifically, in the five subtopics of topic-1, topic-2, topic-3, topic-4, and topic-5 included in the same topic, the priority parameter includes 5 levels, and the priority parameter is equal to 5, indicating priority The priority is the highest; the priority parameter is equal to 1, which means the priority is the lowest; and there is a message accumulation in topic-1 and topic-5, but there is no message accumulation in topic-2, topic-3 and topic-4; assuming the default rated pull It is 500. The default pull ratios corresponding to different priority parameters (from high to low) are: 40%, 30%, 15%, 10%, 5%; specifically, the preset quota of topic-5 Is 200, the default quota for topic-4 is 150, the default quota for topic-3 is 75, the default quota for topic-2 is 50, and the default quota for topic-1 is 25; that is, it is still only Messages can be pulled from topic-5 and topic-1; at this time, the preset collection mode is used. For topic-1, it is used as the lowest priority parameter, and the preset maximum quota can be the preset quota configured by default Doubled, the quota that can be increased is 25, that is, the quota after topic-1 is updated is 50; here, the increased quota can be collected from topic-2; due to topic-2, topic-3, topic- 4 etc. The available quota released by the three priority parameters is 259; then for topic-5, it is the highest priority parameter. From the available quota, the amount of quota that can be increased is 234. In other words, the quota after topic-5 is updated is 434; in this case, when the client SDK is triggered again to pull messages, the actual pull amount of messages pulled by the topic to be consumed is 484.

That is to say, based on the scenario example shown in Figure 5, topic-1 and topic-5 are subtopics to be expanded, which can be released from the three priority parameters of topic-2, topic-3, and topic-4. Quota, and then according to the high-priority parameter corresponding to the proportion of the high-quota pull message and the high water mark limit, so as to ensure that the high-priority parameter can have more computing resources to process the message.

In the embodiment of this application, dynamic quotas and dynamic flow control are implemented through two strategies, such as a preset reduction mode and a preset collection mode, which can avoid the problem of pulling messages in a fixed proportion when there is no accumulation of messages in some subtopics. The consumption group frequently pulls messages in small batches, resulting in the decrease of computing rate and the performance bottleneck of the Kafka cluster, which achieves the optimal computing performance; and relying on dynamic flow control implemented by dynamic quotas, bursts of large traffic in a certain subtopic In the case of message writing, the flow control rate can also be allocated and updated in real time to achieve optimal consumption performance. That is to say, there is a status window (Window) in the implementation of the message pull method in the embodiment of the present application, which is used to collect metadata, save the default quota, the current quota, the quota high water level, the quota low water level, and each priority. Processing status, etc.; Window will be called by specific implementation strategies (such as LoadBalance) to realize the allocation and collection of quota ratios, and the actual pull amount of each message pulled by the subtopics corresponding to each priority parameter It can be written to Window as metadata, and Window will update the processing status of each priority parameter in real time, and then called by Load Balance within a given time interval to trigger dynamic quota and dynamic flow control, and dynamic quota will calculate the quota change The priority parameter is updated by reflection to update the actual pull amount of each message pulled by consumers in the Kafka cluster. Dynamic flow control is calculated by calculating the ratio of each priority parameter to the default preset quota after the dynamic quota. The current flow control rate that each priority parameter should be allocated, and keep the total amount of dynamic quotas consistent.

This embodiment provides a message pull method. The specific implementation of the foregoing embodiment is described in detail through the foregoing embodiment. It can be seen from the above that the dynamic quota is realized through two strategies, namely, a preset reduction mode and a preset collection mode. And dynamic flow control, when there is no accumulation of messages in some subtopics, it can avoid the decrease of the calculation rate and the performance bottleneck of the Kafka cluster caused by the frequent small batches of messages pulled by the consumer group, and the optimization of the calculation performance is realized; in addition, in When a certain subtopic is written in bursts of high-traffic messages, it can also allocate and update its flow control rate in real time, thereby achieving optimal consumption performance.

Based on the same inventive concept as the foregoing embodiment, refer to FIG. 6, which shows a schematic diagram of the composition structure of a message pulling device provided in an embodiment of the present application. As shown in FIG. 6, the message pulling device 60 may include a determining unit 601, an adjusting unit 602, and a pulling unit 603; among them,

The determining unit 601 is configured to determine a plurality of subtopics included in the topic to be consumed and the first pull ratio allocated to each; wherein, different subtopics have different priority parameters;

The determining unit 601 is further configured to determine the actual pull amount of the message pulled by the topic to be consumed according to the first pull ratio;

The adjusting unit 602 is configured to, if the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, adjust the first pull ratio to obtain the respective distribution of the multiple subtopics Second pull ratio;

The pulling unit 603 is configured to pull messages of the topic to be consumed according to the second pulling ratio.

In the above solution, referring to FIG. 7, the message pulling device 60 may further include a receiving unit 604 and a storing unit 605; wherein,

The receiving unit 604 is configured to receive a message to be consumed sent by a message producer, where the message to be consumed includes message content and priority parameters;

The saving unit 605 is configured to save the received messages to be consumed in the subtopics corresponding to the priority parameters to obtain the message accumulation amount corresponding to each subtopic.

In the above solution, referring to FIG. 7, the message pulling device 60 may further include a calculation unit 606; wherein,

The pulling unit 603 is further configured to pull the message accumulation amount corresponding to each subtopic according to the first pull ratio, and obtain the actual pull amount of the messages pulled by each subtopic;

The calculation unit 606 is configured to accumulate the actual pull amount of messages pulled by each subtopic to obtain the actual pull amount of messages pulled by the topic to be consumed.

In the above solution, the determining unit 601 is further configured to obtain that the actual pull amount of messages pulled by one of the subtopics is 0 if the message accumulation amount corresponding to one of the multiple subtopics is equal to 0. ；

Correspondingly, the adjustment unit 602 is further configured to adjust the first pull ratio if the message accumulation amount corresponding to one of the subtopics is equal to 0, so as to obtain the second pull ratio assigned to each of the multiple subtopics .

In the above solution, referring to FIG. 7, the message pulling device 60 may further include a comparing unit 607; wherein,

The determining unit 601 is further configured to determine the preset quota corresponding to each of the multiple subtopics based on the preset rated pull amount and the first pull ratio;

The comparing unit 607 is configured to compare the actual pull amount of messages pulled by each subtopic of the plurality of subtopics with the preset quota corresponding to each subtopic; and if there are actual pulls in the plurality of subtopics; At least one subtopic whose amount is less than the preset quota, it is determined that the actual amount of messages pulled by the topic to be consumed is less than the preset rated amount of pulling.

In the above solution, the adjustment unit 602 is further configured to use a preset reduction mode to reduce the preset quota corresponding to the subtopic to be reduced among the plurality of subtopics, and allocate the obtained reduction amount to the next priority parameter corresponding The sub-themes of to achieve the adjustment of the first pull ratio; wherein, the sub-topics to be reduced refer to sub-topics of multiple sub-topics whose actual pull amount is less than a preset quota.

In the above solution, the comparison unit 607 is further configured to, if the actual pull amount corresponding to one of the multiple subtopics is less than the preset quota, use the one of the subtopics as the subtopic to be reduced, and compare The preset quota corresponding to the sub-topic to be reduced is reduced to obtain the remaining preset quota; and when the remaining preset quota is not lower than the preset minimum quota, the preset corresponding to the sub-topic to be reduced is calculated The difference between the quota and the remaining preset quota is the available quota;

The adjustment unit 602 is specifically configured to sequentially reduce the priority parameters, and use the subtopic corresponding to the reduced priority parameter as the first subtopic; and determine whether the preset quota corresponding to the first subtopic is greater than the maximum quota threshold and Whether the first sub-topic is in the full-speed processing state; and when the preset quota corresponding to the first sub-topic is not greater than the maximum quota threshold and the first sub-topic is in the full-speed processing state, allocating the available quota to The first sub-topic, and update the preset quota corresponding to the first sub-topic; and when the preset quota corresponding to the first sub-topic is greater than the maximum quota threshold and the first sub-topic is in a non-full-speed processing state , Continue to execute the step of lowering the priority parameters in order, and using the subtopic corresponding to the reduced priority parameter as the first subtopic, until the available quota is allocated to the subtopic corresponding to the lowest priority parameter to achieve Adjustment of the first pull ratio.

In the above solution, the adjustment unit 602 is further configured to collect preset quotas corresponding to the multiple subtopics by using a preset collection mode, and allocate the collected amount to the subtopics to be expanded, so as to realize the An adjustment of the pull ratio; wherein, the sub-topic to be expanded represents a sub-topic whose actual pull amount is equal to a preset quota among multiple sub-topics.

In the above solution, the comparing unit 607 is further configured to, if the actual pull amount corresponding to one of the multiple subtopics is equal to the preset quota, use the one of the subtopics as the subtopic to be expanded, according to the preset quota. Set the increase amount to increase the preset quota corresponding to the subtopic to be expanded;

The adjustment unit 602 is specifically configured to use the sub-topic corresponding to the lowest priority as the second sub-topic for the preset increase amount; and determine whether the second sub-topic is in a full-speed processing state; and when the second sub-topic When the subject is in a full-speed processing state, collect quotas from the second subtopic; and if the amount of quota collected is less than the preset increase amount, calculate the difference between the preset increase amount and the collected quota amount Value; and for the calculated difference, the priority parameter is increased in order, the subtopic corresponding to the increased priority parameter is taken as the second subtopic, and the judgment is continued to determine whether the second subtopic is in full-speed processing In the state step, until the solicited quota amount is equal to the preset increase amount, so as to realize the adjustment of the first pull ratio.

In the above solution, the adjustment unit 602 is further configured to increase the priority parameters in order when the second sub-topic is in the non-full-speed processing state, and use the sub-topic corresponding to the increased priority parameter as the second sub-topic , Continue to execute the step of determining whether the second sub-topic is in a full-speed processing state.

It can be understood that, in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a module, or it may also be non-modular. Moreover, the various components in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function module.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or It is said that the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment. The aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Therefore, this embodiment provides a computer storage medium that stores a message pull program that implements the method described in any one of the foregoing embodiments when the message pull program is executed by at least one processor.

Based on the composition of the message pulling device 60 and the computer storage medium described above, see FIG. 8, which shows an example of a specific hardware structure of the message pulling device 60 provided in an embodiment of the present application, which may include: a communication interface 801, a memory 802, and processing 803; The various components are coupled together through the bus system 804. It can be understood that the bus system 804 is used to implement connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear description, various buses are marked as the bus system 804 in FIG. 8. Among them, the communication interface 801 is used for receiving and sending signals in the process of sending and receiving information with other external network elements;

The memory 802 is configured to store a computer program that can run on the processor 803;

The processor 803 is configured to execute: when the computer program is running:

It can be understood that the memory 802 in the embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), and electrically available Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be a random access memory (Random Access Memory, RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced SDRAM, ESDRAM), Synchronous Link Dynamic Random Access Memory (Synchronous Link DRAM, SLDRAM) And Direct Rambus RAM (DRRAM). The memory 802 of the system and method described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

The processor 803 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method can be completed by an integrated logic circuit of hardware in the processor 803 or instructions in the form of software. The aforementioned processor 803 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other Programming logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 802, and the processor 803 reads the information in the memory 802, and completes the steps of the foregoing method in combination with its hardware.

It can be understood that the embodiments described in this application can be implemented by hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, and others for performing the functions described in this application Electronic unit or its combination.

For software implementation, the technology described in this application can be implemented through modules (for example, procedures, functions, etc.) that perform the functions described in this application. The software codes can be stored in the memory and executed by the processor. The memory can be implemented in the processor or external to the processor.

Optionally, as another embodiment, the processor 803 is further configured to execute the steps of the method described in any one of the foregoing embodiments when the computer program is running.

It should be noted that in this application, the terms "including", "including" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements , And also include other elements not explicitly listed, or elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or device that includes the element.

The serial numbers of the foregoing embodiments of the present application are for description only, and do not represent the superiority or inferiority of the embodiments.

The methods disclosed in the several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment or device embodiment.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Industrial applicability

In the embodiment of the present application, by determining the multiple subtopics included in the theme to be consumed and the first pull ratio assigned to each; wherein, different subtopics have different priority parameters; according to the first pull ratio, Determine the actual pull amount of messages pulled by the topic to be consumed; if the actual pull amount of messages pulled by the topic to be consumed is less than the preset rated pull amount, perform the first pull ratio Adjust to obtain a second pull ratio allocated to each of the multiple subtopics; and pull messages for the topic to be consumed according to the second pull ratio. In this way, dynamic flow control is carried out through the adaptive dynamic quota method. When there is no message accumulation in some subtopics, it can avoid the problem of the decrease in the calculation rate and the performance bottleneck of the Kafka cluster caused by the frequent small batches of messages pulled by the consumer group. Optimization of computing performance; In addition, in the case of a subtopic burst of large-flow message writing, it can also allocate and update its flow control rate in real time, thereby achieving optimal consumption performance.

Claims

A message pulling method, the method includes:

Determine the multiple sub-themes included in the theme to be consumed and the first pull ratio assigned to each; among them, different sub-themes have different priority parameters;

According to the first pull ratio, determine the actual pull amount of messages pulled by the topic to be consumed;

If the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, the first pull ratio is adjusted to obtain the second pull ratio assigned to each of the multiple subtopics ；

According to the second pulling ratio, message pulling is performed on the topic to be consumed.
The method according to claim 1, wherein the method further comprises:

Receiving a message to be consumed sent by a message producer, where the message to be consumed includes message content and priority parameters;

The received messages to be consumed are saved in the subtopics corresponding to the priority parameters, and the accumulation of messages corresponding to each subtopic is obtained.
The method according to claim 2, wherein the determining the actual pull amount of the message pulled by the topic to be consumed according to the first pull ratio comprises:

According to the first pull ratio, perform message pull on the message accumulation amount corresponding to each subtopic, and obtain the actual pull amount of messages pulled by each subtopic;

The actual pull amount of messages pulled by each subtopic is accumulated to obtain the actual pull amount of messages pulled by the topic to be consumed.
The method according to claim 3, wherein said obtaining the actual pull amount of the messages pulled by each subtopic comprises:

If the message accumulation amount corresponding to one of the multiple subtopics is equal to 0, it is obtained that the actual pull amount of the messages pulled by the one of the subtopics is 0;

Correspondingly, the method further includes:

If the message accumulation amount corresponding to one of the subtopics is equal to 0, the first pull ratio is adjusted to obtain the second pull ratio assigned to each of the multiple subtopics.
The method according to claim 1, wherein the method further comprises:

Determine the preset quota corresponding to each of the multiple subtopics based on the preset rated pull amount and the first pull ratio;

Comparing the actual pull amount of messages pulled by each subtopic of the plurality of subtopics with the preset quota corresponding to each subtopic;

If there is at least one subtopic whose actual pull amount is less than the preset quota among the multiple subtopics, it is determined that the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount.
The method according to claim 5, wherein said adjusting said first pull ratio comprises:

The preset reduction mode is used to reduce the preset quota corresponding to the sub-topic to be reduced among the multiple sub-topics, and the obtained reduction amount is allocated to the sub-topic corresponding to the next priority parameter to achieve the first reduction. Proportion adjustment; wherein, the sub-topics to be reduced represent sub-topics of the multiple sub-topics whose actual pull amount is less than a preset quota.
8. The method according to claim 6, wherein the preset reduction mode is used to reduce the preset quota corresponding to the subtopic to be reduced among the plurality of subtopics, and the obtained reduction amount is allocated to the next priority parameter The corresponding subtopics include:

If the actual pull amount corresponding to one of the multiple subtopics is less than the preset quota, then the one of the subtopics is regarded as the subtopic to be reduced, and the preset quota corresponding to the subtopic to be reduced is performed Reduce to get the remaining preset quota;

When the remaining preset quota is not lower than the preset minimum quota, calculating the difference between the preset quota corresponding to the subtopic to be reduced and the remaining preset quota to obtain the available quota;

Decrease the priority parameters in order, and use the subtopic corresponding to the reduced priority parameter as the first subtopic;

Determining whether the preset quota corresponding to the first sub-topic is greater than a maximum quota threshold and whether the first sub-topic is in a full-speed processing state;

When the preset quota corresponding to the first sub-topic is not greater than the maximum quota threshold and the first sub-topic is in a full-speed processing state, the available quota is allocated to the first sub-topic, and the first sub-topic is updated. The preset quota corresponding to the subtopic;

When the preset quota corresponding to the first sub-topic is greater than the maximum quota threshold and the first sub-topic is in a non-full-speed processing state, continue to execute the lower priority parameter in order, and change the child corresponding to the lower priority parameter The subject is used as the first sub-topic until the available quota is allocated to the sub-topic corresponding to the lowest priority parameter, so as to realize the adjustment of the first pull ratio.
The method according to claim 5, wherein said adjusting said first pull ratio comprises:

Use the preset collection mode to collect the preset quotas corresponding to the multiple subtopics, and allocate the collected amount to the subtopics to be expanded, so as to realize the adjustment of the first pull ratio; wherein, the waiting Expanding a sub-theme represents a sub-theme whose actual pull amount is equal to a preset quota among the multiple sub-themes.
The method according to claim 8, wherein said using a preset collection mode to collect preset quotas corresponding to the plurality of subtopics, and allocating the collected amount to the subtopics to be expanded, comprises:

If the actual pull amount corresponding to one of the multiple subtopics is equal to the preset quota, then the one of the subtopics is regarded as the subtopic to be expanded, and the corresponding amount of the subtopic to be expanded is increased according to the preset increase amount. Preset quota;

Regarding the preset increase amount, the sub-topic corresponding to the lowest priority is taken as the second sub-topic;

Determine whether the second sub-topic is in a full-speed processing state;

When the second sub-topic is in a full-speed processing state, perform quota collection from the second sub-topic;

If the collected quota amount is less than the preset increase amount, calculate the difference between the preset increase amount and the collected quota amount;

For the calculated difference, the priority parameters are increased in order, the subtopic corresponding to the increased priority parameter is regarded as the second subtopic, and the step of determining whether the second subtopic is in a full-speed processing state is continued. , Until the solicited quota amount is equal to the preset increase amount, so as to realize the adjustment of the first pull ratio.
The method according to claim 9, wherein after said determining whether the second sub-topic is in a full-speed processing state, the method further comprises:

When the second sub-topic is in the non-full-speed processing state, the priority parameters are sequentially increased, the sub-topic corresponding to the increased priority parameter is taken as the second sub-topic, and the judgment of the second sub-topic is continued. Whether it is in the full-speed processing state.
A message pulling device, the message pulling device includes a determining unit, an adjusting unit, and a pulling unit, wherein:

The determining unit is configured to determine a plurality of sub-themes included in the theme to be consumed and a first pull ratio allocated to each; wherein, different sub-themes have different priority parameters;

The determining unit is further configured to determine the actual pull amount of messages pulled by the topic to be consumed according to the first pull ratio;

The adjustment unit is configured to, if the actual pull amount of the message pulled by the topic to be consumed is less than the preset rated pull amount, adjust the first pull ratio to obtain each of the multiple subtopics The allocated second pull ratio;

The pulling unit is configured to pull messages of the topic to be consumed according to the second pulling ratio.
A message pulling device includes a memory and a processor, wherein,

The memory is used to store a computer program that can run on the processor;

The processor is configured to execute the method according to any one of claims 1 to 10 when the computer program is running.
A computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 10 is implemented.