CN115211092B

CN115211092B - Message pulling method and device and computer storage medium

Info

Publication number: CN115211092B
Application number: CN202080097907.8A
Authority: CN
Inventors: 郭子亮
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Priority date: 2020-03-02
Filing date: 2020-03-02
Publication date: 2023-12-22
Anticipated expiration: 2040-03-02
Also published as: CN115211092A; WO2021174382A1

Abstract

The embodiment of the application discloses a message pulling method, a message pulling device and a computer storage medium, wherein the method comprises the following steps: determining a plurality of sub-topics included in the topic to be consumed and a first pull proportion respectively distributed; wherein different sub-topics have different priority parameters; determining the actual pulling amount of the message pulled by the theme to be consumed according to the first pulling proportion; if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the first pulling proportion is adjusted to obtain second pulling proportions respectively distributed by the plurality of sub-themes; and according to the second pulling proportion, message pulling is carried out on the theme to be consumed.

Description

Message pulling method and device and computer storage medium

Technical Field

The embodiment of the application relates to the technical field of data application, in particular to a message pulling method, a message pulling device and a computer storage medium.

Background

Kafka is a high throughput distributed publish-subscribe messaging system, where each message published to the Kafka cluster has a Topic called Topic. Specifically, multiple sub-topics with different priorities can be created for the same Topic, each sub-Topic is configured with a consumption group, and the maximum pulling amount of different priorities can be calculated by different consumption groups according to the message number proportion corresponding to different priority levels configured by default or user definition.

However, considering that there may be some or several sub-topics among the plurality of sub-topics without message stacking, the amount of messages pulled by the consumer at this time does not meet the maximum amount of pulling allocated at initialization. If part of the sub-topics have message accumulation, but the messages are still pulled in a fixed proportion, the message pulling amount of the consumer is smaller than the maximum pulling amount, so that abnormal artifacts are caused to the user; in addition, when the lower priority has message accumulation and the higher priority has no message, if the lower priority still pulls the message in a low quota ratio, the consumer can be caused to frequently pull small batches of messages, so that the IO load of the disk is increased, the performance advantage brought by the pre-reading of an operating system is reduced, and the Kafka cluster has performance bottleneck.

Disclosure of Invention

The application provides a message pulling method, a message pulling device and a computer storage medium, wherein the flow control rate of each sub-theme can be distributed and updated in real time by a self-adaptive dynamic quota mode, and the problems of calculation rate reduction and Kafka cluster performance bottleneck caused by frequent small-batch message pulling of a consumption group can be avoided.

The technical scheme of the embodiment of the application can be realized as follows:

In a first aspect, an embodiment of the present application provides a message pulling method, where the method includes:

determining a plurality of sub-topics included in the topic to be consumed and a first pull proportion respectively distributed; wherein different sub-topics have different priority parameters;

determining the actual pulling amount of the message pulled by the theme to be consumed according to the first pulling proportion;

if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the first pulling proportion is adjusted to obtain second pulling proportions respectively distributed by the plurality of sub-themes;

and according to the second pulling proportion, message pulling is carried out on the theme to be consumed.

In a second aspect, embodiments of the present application provide a message pulling apparatus, which includes a determining unit, an adjusting unit, and a pulling unit, wherein,

the determining unit is configured to determine a plurality of sub-topics included in the topics to be consumed and the first pull proportion distributed by each sub-topic; wherein different sub-topics have different priority parameters;

the determining unit is further configured to determine an actual pull amount of the message pulled by the theme to be consumed according to the first pull proportion;

The adjusting unit is configured to adjust the first pulling proportion if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, so as to obtain second pulling proportions respectively distributed by the plurality of sub-themes;

and the pulling unit is configured to pull the message of the theme to be consumed according to the second pulling proportion.

In a third aspect, embodiments of the present application provide a message pulling apparatus, including a memory and a processor, wherein,

the memory is used for storing a computer program capable of running on the processor;

the processor is configured to perform the method according to the first aspect when the computer program is run.

In a fourth aspect, embodiments of the present application provide a computer storage medium storing a computer program which, when executed by at least one processor, implements a method according to the first aspect.

The embodiment of the application provides a message pulling method, a message pulling device and a computer storage medium, wherein a plurality of sub-topics included in a topic to be consumed and first pulling proportions distributed respectively are determined; wherein different sub-topics have different priority parameters; determining the actual pulling amount of the message pulled by the theme to be consumed according to the first pulling proportion; if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the first pulling proportion is adjusted to obtain second pulling proportions respectively distributed by the plurality of sub-themes; and according to the second pulling proportion, message pulling is carried out on the theme to be consumed. In this way, the dynamic flow control is performed in a self-adaptive dynamic quota mode, when part of sub-topics are not piled up, the problems of calculation rate reduction and Kafka cluster performance bottleneck caused by frequent small-batch message pulling of consumption groups can be avoided, and calculation performance optimization is realized; in addition, under the condition that a certain sub-theme bursts high-flow information writing, the flow control rate can be distributed and updated in real time, so that the optimal consumption performance is realized.

Drawings

Fig. 1 is a schematic flow diagram of a message pulling system according to a related technical scheme;

fig. 2 is a schematic diagram of a scenario of a pull message application provided in the related art;

fig. 3 is a flow chart of a message pulling method according to an embodiment of the present application;

fig. 4 is a schematic view of a scenario of a preset clipping mode application provided in an embodiment of the present application;

fig. 5 is a schematic view of a scenario of a preset feature set mode application provided in an embodiment of the present application;

fig. 6 is a schematic diagram of a composition structure of a message pulling device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of another message pulling device according to an embodiment of the present application;

fig. 8 is a schematic hardware structure of a message pulling device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the specific embodiments described herein are merely illustrative of the application and not limiting of the application. It should be noted that, for convenience of description, only a portion related to the related application is shown in the drawings.

Message middleware is software for inter-node communication that is widely used in distributed systems. In a large-scale high-concurrency background service architecture system, kafka is used as a common message middleware, and has very wide application.

Kafka is a message middleware developed and sourced by Linkedin corporation, a high throughput distributed publish-subscribe messaging system that can handle all action flow data of consumers in websites. The main objective of kafka development is to construct a data processing framework for processing massive logs, user behaviors, website operation statistics and the like, and under the condition of combining the requirements of data mining, behavior analysis, operation monitoring and the like, the requirements of various real-time online and batch offline processing applications on low delay and batch throughput performance are required to be met, and real-time messages can be provided through a cluster.

The Kafka cluster contains one or more servers, known as brookers, each message issued to the Kafka cluster has a topic, known as topic, each topic containing one or more sub-topics. Referring to fig. 1, an example of a flow block diagram of a message pulling system provided by a related technical solution is shown; as shown in fig. 1, the message pulling system 10 includes a message producer 101, a Kafka cluster 102, a client 103, and a consumer 104; the client 103 is typically a software development kit (Software Development Kit, SDK) provided by a third party developer, that is to say an adaptation client provided by the SDK for the developer to quickly access the middleware. Thus, after the message producer 101 generates a message (message), priority parameters of the message can be configured at the same time; for example: message1{ priority:2; data: A3615C }, message2{ priority:5; data: B36D4} etc.; i.e. having priority message logic encapsulated in the client 103, then simulating message writes with different priority parameters by creating sub-topics (topic-1, topic-2, topic-3, topic-4, topic-5, etc.) ending with different priorities for the same topic in the Kafka cluster 102; after the message is written into the sub-topics with the corresponding priority, the message is pulled from different sub-topics in the Kafka cluster 102 through the client 103, for example, the pulled message is message1 (priority: 2; data: A3615C) and message2 (priority: 5; data: B36D 4), and finally the pulled message is sent to the consumer 104, and the consumer 104 carries out relevant processing on the pulled message.

Specifically, a consumption group is configured for each sub-theme corresponding to each priority parameter, different consumption groups multiply rated pulling amounts of messages pulled by all priorities according to pulling ratios corresponding to different priority parameters configured by default or user definition, reference quota distributed by different priorities can be calculated, messages pulled by different sub-themes are collected through matched client SDKs, and a pulled batch message is returned to a user, wherein the batch message contains pulling cancellation amounts corresponding to different priority parameters, so that support of Kafka on the information pulled by different priorities is realized.

Considering that for the subtopics corresponding to the priority parameters, there may be no message accumulation of one or more subtopics at a specific moment, and the actual pulling amount of the message pulled by the consumer at this time does not meet the rated pulling amount allocated during initialization. Fig. 2 shows a schematic view of a scenario of a pull-to-cancel application provided by the related technical solution. In FIG. 2, the same theme includes five sub-themes of topic-1, topic-2, topic-3, topic-4 and topic-5, and the priority parameter includes 5 levels, where the priority parameter is equal to 5, indicating that the priority is highest; the priority parameter is equal to 1, indicating that the priority is lowest; and there is a message pile in topic-1 and topic-5, but there is no message pile in topic-2, topic-3 and topic-4; assuming that the default rated pull amount is 500, the pull ratios (from high priority to low priority) corresponding to the different priority parameters are respectively: 40%, 30%, 15%, 10%, 5%; specifically, a pull quota of topic-5 of 200, i.e., a sub-topic with a priority parameter equal to 5 can pull 200 messages at a time; while the pull quota for topic-4 is 150, the pull quota for topic-3 is 75, the pull quota for topic-2 is 50, and the pull quota for topic-1 is 25; that is, messages can be pulled only from topic-5 and topic-1, and a total of 225 messages are actually pulled. Thus, based on the scenario example shown in FIG. 2, in the case where there is a message pile-up for the sub-topics, the user expects to pull 500 messages at a time, but because of the pull rate limitation corresponding to the different priorities, the user actually pulls only 225 messages. Thus, there will be the following problems: on the one hand, when part of sub-topics have message accumulation, but the actual pulling total amount of the messages pulled by the SDK is smaller than the rated pulling amount set by a user, abnormal artifacts are caused to the user at the moment; on the other hand, if a lower priority presents message stacking while a higher priority does not have messages (in idle state), then the lower priority still pulls messages in a low quota proportion, potentially resulting in frequent pulling of small batches of messages by consumers; and frequent small-batch message pulling can lead to a large number of random reads, thereby leading to the increase of disk IO load, and simultaneously reducing the performance advantage brought by the pre-reading of an operating system, so that the Kafka cluster has performance bottleneck.

Based on the above, the embodiment of the application provides a message pulling method, which includes determining a plurality of sub-topics included in a topic to be consumed and a first pulling proportion respectively allocated; wherein different sub-topics have different priority parameters; determining the actual pulling amount of the message pulled by the theme to be consumed according to the first pulling proportion; if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the first pulling proportion is adjusted to obtain second pulling proportions respectively distributed by the plurality of sub-themes; and according to the second pulling proportion, message pulling is carried out on the theme to be consumed. In this way, the dynamic flow control is performed in a self-adaptive dynamic quota mode, when part of sub-topics are not piled up, the problems of calculation rate reduction and Kafka cluster performance bottleneck caused by frequent small-batch message pulling of consumption groups can be avoided, and calculation performance optimization is realized; in addition, under the condition that a certain sub-theme bursts high-flow information writing, the flow control rate can be distributed and updated in real time, so that the optimal consumption performance is realized.

Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

In an embodiment of the present application, referring to fig. 3, a flow diagram of a message pulling method provided in an embodiment of the present application is shown. As shown in fig. 3, the method may include:

s301: determining a plurality of sub-topics included in the topic to be consumed and a first pull proportion respectively distributed; wherein different sub-topics have different priority parameters;

it should be noted that the method is applied to a message pulling system. In the message pulling system, the SDK pulls messages to topics to be consumed within the kafka cluster. Wherein a same topic to be consumed may create multiple sub-topics with different priority parameters.

It should be further noted that, the first pull ratio is a default pull ratio preset according to different priority parameters; here, the first pull proportion represents a ratio of a message pull number preset for each sub-topic to a rated pull number preset for the whole topic to be consumed. For example, assume that a plurality of sub-topics may include five sub-topics of topic-5, topic-4, topic-3, topic-2, and topic-1; and the priority parameters are 5, 4, 3, 2, 1 and 5, the priority is highest, the priority is lowest, and the first pull ratio can be set to 40%, 30%, 15%, 10% and 5% respectively.

In addition, among the plurality of sub-topics, it is possible that a part of the sub-topics stores the message to be consumed, and a part of the sub-topics does not store the message to be consumed. For each sub-topic stored message to be consumed, the message is produced by the message producer and then placed into the corresponding sub-topic according to the priority parameters. Specifically, in some embodiments, prior to S301, the method may further include:

receiving a message to be consumed sent by a message producer, wherein the message to be consumed comprises message content and priority parameters;

and storing the received message to be consumed into the sub-topics corresponding to the priority parameters to obtain the message accumulation amount corresponding to each sub-topic.

That is, after the message producer produces the message to be consumed, the priority parameter corresponding to the message to be consumed is also configured; thus, after receiving the message to be consumed sent by the message producer through the kafka cluster, the received message to be consumed can be saved to the corresponding sub-topic according to the priority parameter, so as to obtain the message accumulation amount of each sub-topic.

If the message producer produces more messages with certain priority parameters, the message accumulation amount is larger in the subtopic corresponding to the priority parameters; if a message producer produces fewer or even no messages of a certain priority parameter, the message pile-up amount is smaller or even equal to 0 in the subtopic corresponding to the priority parameter.

S302: determining the actual pulling amount of the message pulled by the theme to be consumed according to the first pulling proportion;

it should be noted that after the first pull ratio is obtained, message pulling may be performed according to the first pull ratio, so as to determine an actual pulling amount of each sub-topic in the topic to be consumed, so as to determine an actual pulling amount of the message pulled by the topic to be consumed. Specifically, in some embodiments, for S302, the determining, according to the first pull ratio, an actual pull amount of the message pulled by the to-be-consumed theme may include:

message pulling is carried out on the message accumulation amount corresponding to each sub-theme according to the first pulling proportion, and the actual pulling amount of the message pulled by each sub-theme is obtained;

and accumulating the actual pulling amount of the message pulled by each sub-theme to obtain the actual pulling amount of the message pulled by the theme to be consumed.

That is, after the first pulling ratio is obtained, message pulling can be performed on the message accumulation amount corresponding to each sub-topic according to the first pulling ratio, so as to obtain the actual pulling amount of the message pulled by each sub-topic; and then accumulating the actual pulling amount of the message pulled by each sub-theme, thereby obtaining the actual pulling amount of the message pulled by the theme to be consumed.

S303: if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the first pulling proportion is adjusted to obtain second pulling proportions respectively distributed by the plurality of sub-themes;

it should be noted that, when the message stacking amount corresponding to one of the multiple sub-topics is equal to 0, the actual pulling amount of the message pulled by the sub-topic is 0; at this time, the actual pull amount of the message pulled by the subject to be consumed will be smaller than the preset rated pull amount, that is, the method may further include: and if the message accumulation amount corresponding to one of the sub-topics is equal to 0, adjusting the first pull proportion to obtain a second pull proportion respectively distributed by the plurality of sub-topics.

Further, in some embodiments, the method may further comprise:

determining a preset quota corresponding to each of the plurality of sub-topics based on the preset rated pulling amount and the first pulling proportion;

comparing the actual pulling amount of the message pulled by each sub-topic in the plurality of sub-topics with a preset quota corresponding to each sub-topic;

and if at least one sub-theme with the actual pulling amount smaller than the preset quota exists in the plurality of sub-themes, determining that the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount.

That is, according to the preset rated pulling amount and the first pulling ratio, a preset quota (which may also be referred to as a base station quota) corresponding to each of the plurality of sub-topics may be determined. Specifically, it may include: and multiplying the first pulling proportion corresponding to each sub-theme by the preset rated pulling amount respectively aiming at the plurality of sub-themes to obtain a preset quota corresponding to each sub-theme, thereby determining the preset quota corresponding to each sub-theme.

Here, the preset quota corresponding to each sub-topic indicates the preset number of pull messages of each sub-topic when the message is pulled. After the preset quota corresponding to each sub-topic is calculated, the actual pulling amount of the message pulled by each sub-topic can be compared with the preset quota corresponding to each sub-topic; and when at least one sub-theme with the actual pulling amount smaller than the preset quota exists in the plurality of sub-themes, determining that the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount.

In this way, when the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the fact that the first pulling proportion needs to be dynamically adjusted at the moment is indicated, so that the pulling quota distributed by the subtopic corresponding to each priority parameter is changed; and the second pulling proportion is the distribution proportion corresponding to the converted pulling quota, so that when the next message is pulled, the message can be pulled according to the second pulling proportion.

In other words, such adaptive adjustment of the pull ratio assigned to different priority parameters may be referred to as dynamic quota. Specifically, dynamic flow control can be realized by dynamically adjusting pull quotas (or pull proportions) with different priorities; that is, under a given total flow control rate, the flow control rate of each priority is dynamically allocated, for example, when a certain priority queue has no message accumulation or is called that the message accumulation amount is equal to 0, the flow control rate (i.e. pull quota) of the priority queue can be dynamically allocated to other priorities for use, that is, the pull ratio is adjusted, so that the total consumption rate can be optimized.

It should also be noted that the dynamic quota/dynamic flow control may be implemented using a preset cut mode and/or a preset collection mode. Here, the preset cut-down pattern may be regarded as an active cut-down, and the preset collection pattern may be regarded as a passive collection.

Optionally, in some embodiments, for S303, the adjusting the first pull ratio may include:

the preset quota corresponding to the subtopic to be cut in the plurality of subtopics is cut by using a preset cutting mode, and the obtained cutting quantity is distributed to the subtopic corresponding to the next priority parameter, so that the first pulling proportion is adjusted; the subtopic to be cut represents a subtopic with the actual pulling amount smaller than a preset quota in the plurality of subtopics.

performing collection on preset quotas corresponding to the multiple sub-topics by using a preset collection mode, and distributing the obtained collection quantity to the sub-topics to be expanded so as to realize adjustment of the first pull proportion; the sub-topics to be expanded represent sub-topics with actual pulling quantity equal to a preset quota in the plurality of sub-topics.

That is, in the preset clipping mode, when the actual pulling amount of the message pulled by the subtopic corresponding to a certain priority parameter is insufficient, the quota of the user is actively released to other priorities so as to realize the adjustment of the first pulling proportion; in the preset collection mode, when the actual pulling amount of the message pulled by the subtopic corresponding to a certain priority parameter meets a preset quota (benchmark allocation), if an idle quota exists at the time, for example, the quota amount actively reduced by the subtopic corresponding to the priority parameter in a back pressure (backstpress) state, the subtopic corresponding to the priority parameter can collect the part of quota at the moment so as to realize the adjustment of the first pulling proportion; the benchmark allocation represents a pulling proportion allocated according to different preset priority parameters by default or user definition, and the benchmark quota is obtained through calculation; the backPress state represents a flow control means that indicates that a sub-topic is in a backpressure state when the actual pull amount of a message pulled by that sub-topic is below a reference quota.

S304: and according to the second pulling proportion, message pulling is carried out on the theme to be consumed.

Thus, after the pulling proportion corresponding to the parameters of different priorities is adjusted, a second pulling proportion can be obtained. At this time, if the user triggers the SDK to pull the message again, the message may be pulled on the to-be-consumed theme according to the second pull proportion, so as to obtain the actual pull amount of the message pulled on the to-be-consumed theme again. As such, after S304, the method may further include:

if the actual pulling amount of the obtained message of the subject to be consumed is smaller than the preset rated pulling amount, continuing to adjust the second pulling proportion to obtain a third pulling proportion respectively distributed by the plurality of sub-subjects;

and according to the third pulling proportion, message pulling is carried out on the theme to be consumed.

That is, as long as the actual pulling amount of the message pulled by the obtained theme to be consumed is smaller than the preset rated pulling amount, the current pulling proportion needs to be continuously adjusted, and then when the user triggers the SDK to pull the message next time, the message is pulled according to the adjusted new pulling proportion; the dynamic adjustment of the pulling proportion is realized, so that the dynamic quota of a plurality of subtopics corresponding to different priority parameters is also realized.

It should be noted that, in the message pulling method in the embodiment of the present application, besides kafka, the dynamic quota or the dynamic flow control manner may also be applied to other middleware with dynamic flow control requirements, so as to obtain the optimal consumption performance.

The embodiment provides a message pulling method, which comprises the steps of determining a plurality of sub-topics included in a topic to be consumed and a first pulling proportion distributed by each sub-topic; wherein different sub-topics have different priority parameters; determining the actual pulling amount of the message pulled by the theme to be consumed according to the first pulling proportion; if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the first pulling proportion is adjusted to obtain second pulling proportions respectively distributed by the plurality of sub-themes; and according to the second pulling proportion, message pulling is carried out on the theme to be consumed. In this way, the dynamic flow control is performed in a self-adaptive dynamic quota mode, when part of sub-topics are not piled up, the problems of calculation rate reduction and Kafka cluster performance bottleneck caused by frequent small-batch message pulling of consumption groups can be avoided, and calculation performance optimization is realized; in addition, under the condition that a certain sub-theme bursts high-flow information writing, the flow control rate can be distributed and updated in real time, so that the optimal consumption performance is realized.

In another embodiment of the present application, taking a preset clipping mode as an example, for S303, the adjusting the first pull ratio may include:

if the actual pulling amount corresponding to one of the plurality of sub-topics is smaller than a preset quota, taking the one of the sub-topics as a sub-topic to be cut, and cutting the preset quota corresponding to the sub-topic to be cut to obtain a residual preset quota;

when the remaining preset quota is not lower than a preset minimum quota, calculating a difference value between the preset quota corresponding to the subtopic to be cut and the remaining preset quota to obtain an available quota;

sequentially reducing the priority parameters, and taking the subtopic corresponding to the reduced priority parameters as a first subtopic;

judging whether a preset quota corresponding to the first sub-theme is larger than a maximum quota threshold and whether the first sub-theme is in a full-speed processing state;

when the preset quota corresponding to the first sub-theme is not greater than the maximum quota threshold and the first sub-theme is in a full-speed processing state, the available quota is distributed to the first sub-theme, and the preset quota corresponding to the first sub-theme is updated;

And when the preset quota corresponding to the first sub-theme is greater than the maximum quota threshold and the first sub-theme is in a non-full-speed processing state, continuing to execute the steps of sequentially reducing the priority parameters and taking the sub-theme corresponding to the reduced priority parameter as the first sub-theme until the available quota is distributed to the sub-theme corresponding to the lowest priority parameter so as to realize the adjustment of the first pull proportion.

It should be noted that, the "order lowering priority parameter" indicates that the current sub-theme corresponding to priority 4, and after the order lowering priority parameter is executed, the sub-theme corresponding to the lowered priority parameter is the sub-theme corresponding to priority 3; after the priority parameters are reduced again in the execution sequence, the subtopic corresponding to the reduced priority parameters is the subtopic corresponding to the priority 2, and so on, and when all the priority parameters are in a non-full-speed processing state, the available quota is distributed to the subtopic corresponding to the lowest priority parameter.

It should be further noted that the maximum quota threshold may be represented by a highWaterMark limit, mainly to avoid that the preset rated pulling amount is totally and intensively distributed to the subtopic corresponding to the specific priority parameter due to the dynamic quota, and the maximum pulling amount limit needs to be configured for each priority parameter at this time. The Full-speed processing state can be represented by a Full Progress state, and the actual pulling amount after the dynamic quota of the current sub-theme is represented as being greater than or equal to the reference quota, namely the current sub-theme consumes the message at Full speed.

In this way, for the preset clipping mode, when the message is pulled each time, the self quota can be reduced for the subtopic corresponding to the priority parameter that the actual pulling amount is less than the preset allocation amount. Here, the reduced quota amount may be set according to the actual situation, such as reducing the quota amount by half; and then, carrying out difference calculation on the self quota of the sub-theme and the residual quota amount of the sub-theme, and distributing the obtained difference as an available quota, and simultaneously distributing the quota to the sub-theme corresponding to the next-level priority parameter, wherein if the self quota of the sub-theme corresponding to the next-level priority parameter reaches a maximum quota threshold (highWaterMark limit), the redundant quota can be continuously distributed to the sub-theme corresponding to the next-level priority parameter. In addition, the priority parameter under quota needs to be in Full-speed processing (Full Progress), if all priority parameters are not in Full-speed processing, the available quota is finally aggregated to the subtopic corresponding to the lowest priority parameter, and the lowest priority is not highWaterMark limited.

Further, the quota reduction aiming at the subtopic to be reduced is not always infinitely reduced, and when the dynamic quota corresponding to the subtopic to be reduced is lower than the preset minimum quota, the quota reduction of the subtopic to be reduced is stopped; when the dynamic quota corresponding to all the subtopics to be cut is lower than the preset minimum quota, the dynamic adjustment of the pull proportion is stopped. Thus, in some embodiments, the method may further comprise:

After message pulling is carried out on the theme to be consumed each time, the first pulling proportion is updated by utilizing the second pulling proportion;

message pulling is carried out on the theme to be consumed according to the updated first pulling proportion, and the updated actual pulling amount of the information pulled by the theme to be consumed is obtained;

if the actual pulling amount of the information of the subject to be consumed after updating is smaller than the preset rated pulling amount, continuing to adjust the first pulling proportion to obtain the second pulling proportion, and returning to the step of updating the first pulling proportion by using the second pulling proportion;

after the theme to be consumed is pulled for a plurality of times, stopping the step of adjusting the first pulling proportion when the dynamic quota corresponding to the subtopic to be cut reaches the preset minimum quota.

That is, after each message pull of the topic to be consumed, the first pull rate may be updated with the second pull rate; when the client SDK is triggered to pull the message next time, the updated first pull proportion can be utilized to pull the message, and then the updated actual pull amount of the message pulled by the subject to be consumed is obtained; if the updated actual pulling amount of the information of the to-be-consumed theme is smaller than the preset rated pulling amount, the first pulling proportion needs to be continuously adjusted.

Thus, after a plurality of message pulling processes, when the dynamic quota corresponding to the subtopic to be cut reaches the preset minimum quota, it is indicated that the dynamic quota corresponding to the subtopic to be cut has reached the low water level limit, at this time, the adjustment of the first pulling proportion needs to be stopped, and when the client SDK is triggered to pull the message again, the message pulling process can be performed according to the pulling proportion after the latest dynamic update. It should be further noted that, the preset minimum quota is mainly a quota lower limit set to avoid that a quota corresponding to a sub-topic without message accumulation actively cuts down to 0. Here, the preset minimum quota may be specifically set according to actual situations, and typically, one sixteenth of the preset quota (or referred to as a reference quota) configured by default may be used as the preset minimum quota, but is not specifically limited.

Referring to fig. 4, an application scenario diagram of a preset clipping mode provided in an embodiment of the present application is shown. In fig. 4, message pulling system 10 may include an internal state maintainer 105 in addition to Kafka cluster 102, clients 103, and consumers 104; here, the internal state maintainer 105 may be a state window (window). Specifically, among five sub-topics including topic-1, topic-2, topic-3, topic-4 and topic-5 included in the same topic, the priority parameter includes 5 levels, and the priority parameter is equal to 5, which indicates that the priority is highest; the priority parameter is equal to 1, indicating that the priority is lowest; and there is a message pile in topic-1 and topic-5, but there is no message pile in topic-2, topic-3 and topic-4; assuming that the default rated pull amount is 500, the preset pull ratios (from high priority to low priority) corresponding to different priority parameters are respectively: 40%, 30%, 15%, 10%, 5%; specifically, the preset quota for topic-5 is 200, the preset quota for topic-4 is 150, the preset quota for topic-3 is 75, the preset quota for topic-2 is 50, and the preset quota for topic-1 is 25; that is, messages can only be pulled from topic-5 and topic-1; at the moment, utilizing a preset cutting mode, wherein topic-2, topic-3 and topic-4 are used as subtopics to be cut; when the message is pulled each time, as the actual pulling amount is lower than the preset quota, one half of quota is actively reduced for one of the subtopics to be reduced, and the reduced quota is sequentially distributed to the subtopics corresponding to the secondary priority parameters; if the secondary priority parameters have no message accumulation, continuing to distribute the secondary priority parameters, and finally falling to the lowest priority so as to realize the adjustment of the pulling proportion; after the dynamic quota of the three subtopics to be cut is 1/16 of the default quota after the rest is pulled for multiple times, the updated quota for the subtopic-2, the topic-3 and the topic-4 are respectively as follows: 3. 4, 9; the quota released by the topic-2, the topic-3 and the topic-4 can be used as topic-1 and topic-5, namely can be used as the collectable quota of the topic-1 and the topic-5; here, the symptomatic quota for topic-1 and topic-5 is 259.

if the actual pulling amount corresponding to one of the multiple subtopics is equal to a preset quota, taking the one of the subtopics as a subtopic to be expanded, and increasing the preset quota corresponding to the subtopic to be expanded according to a preset increasing amount;

aiming at the preset increment, taking the subtopic corresponding to the lowest priority as a second subtopic;

judging whether the second sub-theme is in a full-speed processing state;

performing a quota collection from the second sub-topic when the second sub-topic is in a full-speed processing state;

if the collected quota amount is smaller than the preset increment amount, calculating a difference value between the preset increment amount and the collected quota amount;

and sequentially increasing the priority parameters according to the calculated difference values, taking the subtopic corresponding to the increased priority parameters as a second subtopic, and continuously executing the step of judging whether the second subtopic is in a full-speed processing state or not until the acquired quota amount is equal to the preset increment amount so as to realize the adjustment of the first pull proportion.

Further, after said determining whether said second sub-topic is in a full speed processing state, the method may further comprise:

and when the second sub-theme is in a non-full-speed processing state, sequentially increasing the priority parameters, taking the sub-theme corresponding to the increased priority parameters as the second sub-theme, and continuously executing the step of judging whether the second sub-theme is in the full-speed processing state.

It should be noted that, the "sequentially increasing priority parameter" indicates that the current sub-theme corresponding to the priority 2, and after the sequentially increasing priority parameter is executed, the sub-theme corresponding to the increased priority parameter is the sub-theme corresponding to the priority 3; after the priority parameters are sequentially increased again, the subtopics corresponding to the increased priority parameters are subtopics corresponding to the priority 4, and so on, and the acquired quota amount is equal to the preset increase amount.

It should be further noted that, performing quota collection from the second sub-topic, first, calculating the quota amount that can be collected by the second sub-topic, that is, adding the quota to the priority parameter because other priority parameters actively cut down; the preset quota can be subtracted from the current dynamic quota of the second sub-theme, and the difference value is the quota amount which can be collected by the second sub-theme; then comparing the quota amount which can be collected with a preset increment amount; when the quota amount of the collection is smaller than the preset increment amount, calculating a difference value between the preset increment amount and the quota amount of the collection, and carrying out collection on the difference value recursively from the subtopic corresponding to the parameter with the higher priority; when the amount of quota collected is not less than a preset increase, the preset increase can be collected from the sub-topics; and then continuing to carry out quota collection aiming at other sub-topics to be expanded.

In this way, in the case that there is no message accumulation for a specific priority parameter, then because of the message accumulation caused by message production, the subtopic corresponding to the priority parameter from which the quota was previously cut needs to be supplemented. Specifically, the actual pulling amount of the message pulled from the sub-theme corresponding to a certain priority parameter through the consumption group at this time is equal to the preset quota dynamically allocated for the message, so that the quota of the sub-theme can be updated to be doubled by the pulling quota, that is, the preset increasing amount can be the preset quota of the sub-theme at present; the preset increment (which can be represented by A) needs to be collected from the subtopic corresponding to other priority parameters, and the collection strategy is to collect the subtopic corresponding to the lowest priority parameter preferentially; for the subtopic corresponding to the lowest priority parameter, the quota expanding amount of the subtopic (i.e. the quota added to the present priority parameter because the other priority parameters actively clip) needs to be calculated, and if the current quota expanding amount B is lower than the preset increasing amount a, the rest (a-B) recursively increases from the higher priority value to the required preset increasing amount. Here, the subtopic corresponding to the high priority parameter of the recruited must be in Full Progress state, and the quota amount of the recruitable is limited not only by highWaterMark corresponding to the current priority parameter, but also by quota expansion amount corresponding to other priority parameters, and in summary, the total amount of dynamic quota needs to be kept consistent.

Further, quota expansion for the sub-theme to be expanded is not performed all the time in an infinite way, and when the dynamic quota corresponding to the sub-theme to be expanded is higher than the preset highest quota, quota expansion of the sub-theme to be expanded is stopped; when the dynamic quota corresponding to all the subtopics to be cut is higher than the preset minimum quota, the dynamic adjustment of the pull proportion is stopped. Thus, in some embodiments, the method may further comprise:

after the theme to be consumed is pulled for a plurality of times, stopping the step of adjusting the first pulling proportion when the dynamic quota corresponding to the subtopic to be expanded reaches the preset highest quota.

Thus, after a plurality of message pulling processes, when the dynamic quota corresponding to the sub-theme to be expanded reaches the preset highest quota, the dynamic quota corresponding to the sub-theme to be expanded is indicated to reach the high water level limit, at this time, the adjustment of the first pulling proportion needs to be stopped, and when the client SDK is triggered to pull the message again, the message pulling process can be performed according to the pulling proportion after the latest dynamic update. It should be further noted that, the preset highest quota is mainly set to avoid that all quota passive collection corresponding to a subtopic with message accumulation is concentrated on a subtopic corresponding to a specific priority parameter. Here, the preset highest quota may be specifically set according to the actual situation, and generally, for the lowest priority parameter, the preset highest quota may be one time of a preset quota (or referred to as a reference quota) configured by default; while other priority parameters, the preset highest quota may be eight times its own reference quota, but is not particularly limited.

Referring to fig. 5, an application scenario schematic diagram of a preset symptom set mode provided in an embodiment of the present application is shown. In fig. 5, message pulling system 10 includes an internal state maintainer 105 in addition to Kafka cluster 102, clients 103, and consumers 104. Specifically, among five sub-topics including topic-1, topic-2, topic-3, topic-4 and topic-5 included in the same topic, the priority parameter includes 5 levels, and the priority parameter is equal to 5, which indicates that the priority is highest; the priority parameter is equal to 1, indicating that the priority is lowest; and there is a message pile in topic-1 and topic-5, but there is no message pile in topic-2, topic-3 and topic-4; assuming that the default rated pull amount is 500, the preset pull ratios (from high priority to low priority) corresponding to different priority parameters are respectively: 40%, 30%, 15%, 10%, 5%; specifically, the preset quota for topic-5 is 200, the preset quota for topic-4 is 150, the preset quota for topic-3 is 75, the preset quota for topic-2 is 50, and the preset quota for topic-1 is 25; that is, messages can still only be pulled from topic-5 and topic-1; at this time, the preset collection mode is utilized, for topic-1, as the lowest priority parameter, the preset highest quota can be doubled as the preset quota of the default configuration, namely, the increased quota amount is 25, that is, the quota after the topic-1 is updated is 50; here, the increased quota amount may be solicited from topic-2; the symptomatic quota released by three priority parameters of topic-2, topic-3 and topic-4 is 259; then for topic-5, which is the highest priority parameter, from the collectable quota, an increasing quota amount of 234 can be obtained, that is, the updated quota for topic-5 is 434; in this case, when the message is pulled again by triggering the client SDK, the actual pull amount of the message pulled by the subject to be consumed is 484.

That is, based on the scenario example shown in fig. 5, topic-1 and topic-5 as sub-topics to be expanded, the quota released from the 3 priority parameters of topic-2, topic-3, topic-4, etc. can be solicited, and then the pull-to-break ratio and the high water limit of the high quota are corresponding to the high priority parameter, thereby ensuring that the high priority parameter can have more computing resources to process the message.

In the embodiment of the application, the dynamic quota and the dynamic flow control are realized through two strategies such as a preset cutting mode, a preset collection mode and the like, so that the problems of calculation rate reduction and Kafka cluster performance bottleneck caused by frequent small-batch message pulling of a consumption group when part of sub-topics are not piled up with messages existing when fixed proportion pulling is stopped can be avoided, and the optimization of calculation performance is realized; and the dynamic flow control realized by the dynamic quota can also be used for distributing and updating the flow control rate in real time under the condition that a certain sub-theme bursts the writing of the large-flow information so as to realize the optimal consumption performance. That is, in the implementation manner of the message pulling method of the embodiment of the present application, there is a status Window (Window) for collecting metadata, saving default quota, current quota, quota high water level, quota low water level, processing status of each priority, and the like; the Window may be called by a specific implementation policy (such as Load Balance) to implement allocation and collection of quota proportion, the actual pulling amount of each message pulled by the subtopic corresponding to each priority parameter may be written into the Window as metadata, the Window may update the processing state of each priority parameter in real time, then the Load Balance is called in a given time interval to trigger a dynamic quota and a dynamic flow control, the dynamic quota calculates the priority parameter with quota change and updates the actual pulling amount of each message pulled by the consumer in the Kafka cluster through reflection, and the dynamic flow control is the proportion of each priority parameter relative to a default preset quota after calculating the dynamic quota, thereby obtaining the flow control rate that each priority parameter should be allocated currently, and keeping the total amount of the dynamic quota consistent.

The embodiment provides a message pulling method, which elaborates the specific implementation of the foregoing embodiment through the foregoing embodiment, and it can be seen that, by implementing dynamic quota and dynamic flow control through two strategies, such as a preset cutting mode and a preset collection mode, when part of sub-topics have no message accumulation, the problems of reduced computation rate and Kafka cluster performance bottleneck caused by frequent small batch message pulling of a consumption group can be avoided, and optimization of computation performance is implemented; in addition, under the condition that a certain sub-theme bursts high-flow information writing, the flow control rate can be distributed and updated in real time, so that the optimal consumption performance is realized.

Based on the same inventive concept as the previous embodiments, referring to fig. 6, a schematic diagram of a composition structure of a message pulling device according to an embodiment of the present application is shown. As shown in fig. 6, the message pulling apparatus 60 may include a determining unit 601, an adjusting unit 602, and a pulling unit 603; wherein,

a determining unit 601, configured to determine a plurality of sub-topics included in the topic to be consumed and a first pull rate allocated to each of the sub-topics; wherein different sub-topics have different priority parameters;

A determining unit 601, further configured to determine, according to the first pull proportion, an actual pull amount of the message pulled by the topic to be consumed;

the adjusting unit 602 is configured to adjust the first pull ratio if the actual pull amount of the message pulled by the to-be-consumed theme is smaller than the preset rated pull amount, so as to obtain second pull ratios respectively allocated to the multiple sub-themes;

and the pulling unit 603 is configured to pull the message on the topic to be consumed according to the second pulling proportion.

In the above-described scheme, referring to fig. 7, the message pulling apparatus 60 may further include a receiving unit 604 and a storing unit 605; wherein,

an accepting unit 604, configured to receive a message to be consumed sent by a message producer, where the message to be consumed includes a message content and a priority parameter;

the storing unit 605 is configured to store the received message to be consumed in the sub-topics corresponding to the priority parameter, so as to obtain a message stacking amount corresponding to each sub-topic.

In the above-described aspect, referring to fig. 7, the message pulling apparatus 60 may further include a calculating unit 606; wherein,

the pulling unit 603 is further configured to pull the message according to the first pulling proportion, so as to obtain an actual pulling amount of the message pulled by each sub-topic;

The calculating unit 606 is configured to accumulate the actual pulling amount of the message pulled by each sub-topic, so as to obtain the actual pulling amount of the message pulled by the topic to be consumed.

In the above solution, the determining unit 601 is further configured to obtain, if the message stacking amount corresponding to one of the multiple sub-topics is equal to 0, an actual pull amount of the message pulled by the one of the sub-topics is 0;

correspondingly, the adjusting unit 602 is further configured to adjust the first pull ratio if the message stacking amount corresponding to one of the sub-topics is equal to 0, so as to obtain a second pull ratio allocated to each of the plurality of sub-topics.

In the above-described aspect, referring to fig. 7, the message pulling apparatus 60 may further include a comparing unit 607; wherein,

the determining unit 601 is further configured to determine a preset quota corresponding to each of the plurality of subtopics based on the preset rated pull amount and the first pull proportion;

a comparing unit 607, configured to compare the actual pulled amount of the message pulled by each of the multiple sub-topics with a preset quota corresponding to each sub-topic; and if at least one sub-theme with the actual pulling amount smaller than the preset quota exists in the plurality of sub-themes, determining that the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount.

In the above solution, the adjusting unit 602 is further configured to use a preset clipping mode to clip a preset quota corresponding to a sub-topic to be clipped in the multiple sub-topics, and allocate the clipping amount obtained to a sub-topic corresponding to a next priority parameter, so as to adjust the first pull ratio; the subtopic to be cut represents a subtopic with the actual pulling amount smaller than the preset quota in a plurality of subtopics.

In the above solution, the comparing unit 607 is further configured to, if an actual pull amount corresponding to one of the multiple sub-topics is smaller than a preset quota, take the one sub-topic as a sub-topic to be reduced, and reduce the preset quota corresponding to the sub-topic to be reduced to obtain a remaining preset quota; when the remaining preset quota is not lower than a preset minimum quota, calculating a difference value between the preset quota corresponding to the subtopic to be cut and the remaining preset quota to obtain an available quota;

the adjusting unit 602 is specifically configured to sequentially reduce the priority parameters, and takes the subtopic corresponding to the reduced priority parameters as a first subtopic; judging whether a preset quota corresponding to the first sub-theme is larger than a maximum quota threshold and whether the first sub-theme is in a full-speed processing state; when the preset quota corresponding to the first sub-theme is not greater than the maximum quota threshold and the first sub-theme is in a full-speed processing state, the available quota is distributed to the first sub-theme, and the preset quota corresponding to the first sub-theme is updated; and when the preset quota corresponding to the first sub-theme is greater than the maximum quota threshold and the first sub-theme is in a non-full-speed processing state, continuing to execute the steps of sequentially lowering the priority parameters, taking the sub-theme corresponding to the lowered priority parameter as the first sub-theme until the available quota is distributed to the sub-theme corresponding to the lowest priority parameter, so as to realize adjustment of the first pull proportion.

In the above solution, the adjusting unit 602 is further configured to perform a collection on preset quotas corresponding to the multiple sub-topics by using a preset collection mode, and allocate the obtained collection amount to the sub-topics to be expanded, so as to implement adjustment of the first pull proportion; the sub-topics to be expanded represent sub-topics with actual pulling quantity equal to a preset quota in a plurality of sub-topics.

In the above solution, the comparing unit 607 is further configured to, if an actual pull amount corresponding to one of the multiple sub-topics is equal to a preset quota, use the one sub-topic as the sub-topic to be expanded, and increase the preset quota corresponding to the sub-topic to be expanded according to a preset increase amount;

an adjusting unit 602, specifically configured to take, as a second sub-theme, a sub-theme corresponding to the lowest priority for the preset increment; and determining whether the second sub-topic is in a full-speed processing state; and performing a quota collection from the second sub-topic when the second sub-topic is in a full speed processing state; if the collected quota amount is smaller than the preset increment amount, calculating a difference value between the preset increment amount and the collected quota amount; and sequentially increasing the priority parameters according to the calculated difference values, taking the subtopic corresponding to the increased priority parameters as a second subtopic, and continuously executing the step of judging whether the second subtopic is in a full-speed processing state or not until the acquired quota amount is equal to the preset increment amount so as to realize the adjustment of the first pull proportion.

In the above solution, the adjusting unit 602 is further configured to sequentially raise the priority parameters when the second sub-theme is in the non-full-speed processing state, and use the sub-theme corresponding to the raised priority parameter as the second sub-theme, to continue to execute the step of determining whether the second sub-theme is in the full-speed processing state.

It will be appreciated that in this embodiment, the "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., and may of course be a module, or may be non-modular. Furthermore, the components in the present embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional modules.

The integrated units, if implemented in the form of software functional modules, may be stored in a computer-readable storage medium, if not sold or used as separate products, and based on such understanding, the technical solution of the present embodiment may be embodied essentially or partly in the form of a software product, which is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or processor to perform all or part of the steps of the method described in the present embodiment. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Accordingly, the present embodiment provides a computer storage medium storing a message pull program that when executed by at least one processor implements the method of any of the preceding embodiments.

Based on the above-described composition of the message pulling apparatus 60 and the computer storage medium, referring to fig. 8, which shows a specific hardware configuration example of the message pulling apparatus 60 provided in the embodiment of the present application, the method may include: a communication interface 801, a memory 802, and a processor 803; the various components are coupled together by a bus system 804. It is to be appreciated that the bus system 804 is employed to enable connected communications between these components. The bus system 804 includes a power bus, a control bus, and a status signal bus in addition to a data bus. But for clarity of illustration the various buses are labeled as bus system 804 in fig. 8. The communication interface 801 is configured to receive and send signals during the process of receiving and sending information with other external network elements;

a memory 802 for storing a computer program capable of running on the processor 803;

a processor 803 for executing, when running the computer program:

It is appreciated that the memory 802 in embodiments of the present application may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (Double Data Rate SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct memory bus RAM (DRRAM). The memory 802 of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

And the processor 803 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuitry of hardware or instructions in software form in the processor 803. The processor 803 may be a general purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 802, and the processor 803 reads information in the memory 802, and in combination with its hardware, performs the steps of the above method.

It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For a hardware implementation, the processing units may be implemented within one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing devices (DSP devices, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units configured to perform the functions described herein, or a combination thereof.

For a software implementation, the techniques described herein may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.

Optionally, as another embodiment, the processor 803 is further configured to perform the steps of the method of any of the preceding embodiments when the computer program is run.

It should be noted that, in this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

The methods disclosed in the several method embodiments provided in the present application may be arbitrarily combined without collision to obtain a new method embodiment.

The features disclosed in the several product embodiments provided in the present application may be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or apparatus embodiments provided in the present application may be arbitrarily combined without conflict to obtain new method embodiments or apparatus embodiments.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Industrial applicability

In the embodiment of the application, a plurality of sub-topics included in a topic to be consumed and a first pull proportion distributed by each sub-topic are determined; wherein different sub-topics have different priority parameters; determining the actual pulling amount of the message pulled by the theme to be consumed according to the first pulling proportion; if the actual pulling amount of the message pulled by the theme to be consumed is smaller than the preset rated pulling amount, the first pulling proportion is adjusted to obtain second pulling proportions respectively distributed by the plurality of sub-themes; and according to the second pulling proportion, message pulling is carried out on the theme to be consumed. In this way, the dynamic flow control is performed in a self-adaptive dynamic quota mode, when part of sub-topics are not piled up, the problems of calculation rate reduction and Kafka cluster performance bottleneck caused by frequent small-batch message pulling of consumption groups can be avoided, and calculation performance optimization is realized; in addition, under the condition that a certain sub-theme bursts high-flow information writing, the flow control rate can be distributed and updated in real time, so that the optimal consumption performance is realized.

Claims

1. A message pulling method, the method comprising:

2. The method of claim 1, wherein the method further comprises:

3. The method of claim 2, wherein the determining, from the first pull ratio, an actual pull amount of the message pulled by the topic to be consumed comprises:

4. The method of claim 3, wherein the obtaining the actual pull amount of the pulled message for each sub-topic comprises:

if the message accumulation amount corresponding to one of the multiple sub-topics is equal to 0, obtaining that the actual pulling amount of the message pulled by the one sub-topic is 0;

correspondingly, the method further comprises the steps of:

and if the message accumulation amount corresponding to one of the sub-topics is equal to 0, adjusting the first pull proportion to obtain a second pull proportion respectively distributed by the plurality of sub-topics.

5. The method of claim 1, wherein the method further comprises:

6. The method of claim 5, wherein the adjusting the first pull ratio comprises:

7. The method of claim 6, wherein the reducing the preset quota corresponding to the sub-topic to be reduced in the plurality of sub-topics using the preset reducing mode, and distributing the obtained reduction to the sub-topic corresponding to the next priority parameter, includes:

8. The method of claim 5, wherein the adjusting the first pull ratio comprises:

9. The method of claim 8, wherein the performing the soliciting on the preset quotas corresponding to the plurality of sub-topics using the preset solicitation pattern and distributing the obtained solicitation to the sub-topics to be expanded comprises:

judging whether the second sub-theme is in a full-speed processing state;

10. The method of claim 9, wherein after said determining whether said second sub-topic is in a full speed processing state, said method further comprises:

11. A message pulling device includes a determination unit, an adjustment unit, and a pulling unit, wherein,

12. A message pulling apparatus includes a memory and a processor, wherein,

the processor being adapted to perform the method of any of claims 1 to 10 when the computer program is run.

13. A computer storage medium storing a computer program which, when executed by at least one processor, implements the method of any one of claims 1 to 10.