CN113872888B - Method, system and medium for catch-up of Kafka backlog queue - Google Patents
Method, system and medium for catch-up of Kafka backlog queue Download PDFInfo
- Publication number
- CN113872888B CN113872888B CN202111133495.1A CN202111133495A CN113872888B CN 113872888 B CN113872888 B CN 113872888B CN 202111133495 A CN202111133495 A CN 202111133495A CN 113872888 B CN113872888 B CN 113872888B
- Authority
- CN
- China
- Prior art keywords
- consumer
- offset
- chaser
- state
- consumption
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/56—Queue scheduling implementing delay-aware scheduling
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a method, a system and a medium for catch up a Kafka backlog queue, wherein the method comprises the following steps: starting a consumer and adding the consumer to a consumer list, if the consumer is a catch-up consumer, starting to consume from the latest consumption offset, otherwise, consuming according to a normal consumption flow; when the consumer consumes normally, the consumer circularly pulls data from Kafka, calculates the difference value between the current consumer offset and the maximum offset of the message queue, sets the consumer to be in a delay state if the difference value is larger than a threshold value, and sets the consumer to be in a delay recovery state if the consumer already consumes the offset when the chaser starts and the consumer is in the delay state; starting a chaser for consumption by a consumer in a delay state; if the state of the consumer is found to be the delayed recovery state, the chaser is closed, the consumer continues to consume from the current consumption position of the chaser, and the consumer is set to be in a normal state. The invention realizes the full-automatic setting of the chaser and improves the data processing efficiency.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method, a system and a medium for catch up with a Kafka backlog queue.
Background
The existing method for catching up the message in the message queue is realized by manually designating the consumption starting position by using the existing operation and maintenance platform or API of the message queue middleware by an operation and maintenance personnel. At present, the mode of catch-up against the backlog queue is manually monitored and adjusted, so that the problems that the system can not automatically catch-up and stop catch-up according to the self-adaptive adjustment of the flow, the efficiency is lower and the like can not be achieved. In addition, the existing Kafka consumption is a single consumer consumption mode in a single theme group, and once consumer consumption capacity cannot keep up with producer production data capacity, relevant theme partitions can generate backlog data, so that the timeliness of the data cannot be guaranteed.
Disclosure of Invention
The invention aims to solve the technical problem of providing a method, a system and a medium for catch-up of a Kafka backlog queue, realizing flexible and automatic setting of a catch-up person and improving data processing efficiency and data timeliness.
In a first aspect, the present invention provides a method for Kafka backlog queue catch-up, the method comprising:
step S10, starting a consumer, adding the consumer to a consumer list, and transmitting a marking parameter to determine whether the consumer is a catch-up person, if so, starting consuming from the latest consumption offset and recording the offset, otherwise, consuming data by the consumer according to the normal consumption flow of the step S20;
step S20, starting a normal consumption flow, wherein a consumer starts to circularly pull data from Kafka, when the data is pulled, calculating a difference value between the current consumer offset and the maximum message queue offset, setting the consumer to a delay state if the difference value is larger than a threshold value, and setting the consumer to a delay recovery state if the consumer has consumed the offset when the chaser is started and the consumer state is the delay state; if the difference value is smaller than or equal to the threshold value, returning to the step S20 to continue execution;
step S30, a consumer list is scanned regularly, if the state of a consumer is found to be a delay state, a new consumer aiming at the same queue is started to be consumed as a chaser, and the consumer and the chaser corresponding to the consumer are stored in a map;
and S40, mapping between the consumer and the chaser is scanned regularly, if the state of the consumer is found to be in a delayed recovery state, the current consumption offset of the message queue partition of the chaser is obtained, the chaser is closed, the current consumption offset of the chaser is set to the consumer, the consumer continues to consume from the consumption position of the chaser, and the consumer is set to be in a normal state.
Further, the marking parameter is a boolean parameter, and the true is taken to represent that the marking parameter is a chaser, and vice versa.
Further, the latest consumption offset refers to an absolute value position of latest data of the current message queue.
Further, in the step S20, when the consumer starts to cycle to pull data from Kafka, it is first determined whether the production speed increase of the message queue data producer is greater than the production speed increase of the message queue consumer, if so, the difference between the current consumer offset and the maximum message queue offset is further calculated, otherwise, the difference is not calculated.
In a second aspect, the present invention provides a system for Kafka backlog queue catch-up, the system comprising:
the system starting module is used for starting the consumer, adding the consumer into the consumer list, and transmitting a marking parameter to determine whether the consumer is a catch-up person, if the consumer is the catch-up person, starting consuming from the latest consumption offset and recording the offset, otherwise, the consumer consumes data according to the normal consumption flow of the step S20;
the normal consumption module is used for starting a normal consumption flow, a consumer starts to circularly pull data from Kafka, when the data is pulled, the difference between the current consumer offset and the maximum message queue offset is calculated, if the difference is larger than a threshold value, the consumer is set to be in a delay state, and if the consumer has consumed the offset when the chaser starts and the consumer state is in the delay state, the consumer is set to be in a delay recovery state; if the difference value is smaller than or equal to the threshold value, returning to the normal consumption module to continue execution;
the catch-up person adding module is used for regularly scanning the consumer list, starting a new consumer aiming at the same queue to consume as a catch-up person if the state of the consumer is found to be a delay state, and storing the consumer and the corresponding catch-up person in a map; and
and the delay recovery module is used for regularly scanning the mapping between the consumer and the chaser, acquiring the current consumption offset of the message queue partition of the chaser if the state of the consumer is found to be the delay recovery state, closing the chaser, setting the current consumption offset of the chaser to the consumer, continuing to consume from the consumption position of the chaser by the consumer, and setting the consumer to be in a normal state.
Further, the marking parameter is a boolean parameter, and the true is taken to represent that the marking parameter is a chaser, and vice versa.
Further, the latest consumption offset refers to an absolute value position of latest data of the current message queue.
Further, in the normal consumption module, when the consumer starts to circularly pull data from Kafka, it is first determined whether the production speed increase of the message queue data producer is greater than the message queue consumer increase, if so, the difference between the current consumer offset and the maximum message queue offset is further calculated, otherwise, the difference is not calculated.
In a third aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the method of the first aspect.
The invention has the advantages that: automatically setting a catch-up person to consume the backlog queue, and setting an automatic stopping mechanism, so that manual operation is reduced, and timeliness of data processing is improved; the threshold value can be customized according to the system load, so that the flexibility is higher.
Drawings
The invention will be further described with reference to examples of embodiments with reference to the accompanying drawings.
FIG. 1 is a flow chart of a method of the present invention for Kafka backlog queue catch-up.
FIG. 2 is a block diagram of a system for Kafka backlog queue catch-up of the present invention.
FIG. 3 is a schematic diagram of the structure of the medium of the present invention.
FIG. 4 is a schematic diagram of a consumer structure of the present invention in a normal state.
FIG. 5 is a schematic diagram of a consumer architecture in a consumer delayed state according to the present invention.
FIG. 6 is a schematic diagram of the structure of the present invention when the consumer has consumed the catch-up position.
FIG. 7 is a schematic diagram of the consumer's consumption when the consumer goes from the delayed recovery state to the normal state.
FIG. 8 is a flowchart of the system start-up and consumer normal consumption execution of the present invention.
FIG. 9 is a flow chart of the present invention for timing scanning the delay status of a consumer.
FIG. 10 is a flow chart of the timing scan mapping of the present invention.
Detailed Description
Example 1
Referring to fig. 1 and fig. 4 to fig. 10, the present invention provides a method for chafka backlog queue catch-up, which includes:
step S10, starting a consumer C1, adding the consumer C1 to a consumer list, consuming data of a message queue Q1, and transmitting a flag parameter to determine whether the consumer is a chaser, if so, consuming from the latest consumption offset and recording the offset, otherwise, consuming the data by the consumer according to the normal consumption flow of the step S20;
step S20, starting a normal consumption flow, wherein a consumer starts to circularly pull data from Kafka, when the data is pulled, calculating the difference between the current consumer offset and the maximum offset of a message queue (namely, making a difference between the current consumer offset and the maximum offset of the message queue, taking the absolute value of the current consumer offset and the maximum offset of the message queue to obtain a delay size, namely, the difference), if the difference is larger than a threshold (which can be adjusted according to the need), setting the consumer into a delay state, and if the consumer already consumes the offset when the chaser starts and the consumer state is in the delay state, setting the consumer into a delay recovery state; if the difference value is smaller than or equal to the threshold value, returning to the step S20 to continue execution;
step S30, a consumer list is scanned regularly, if the state of a consumer is found to be a delay state, a new consumer aiming at the same queue is started to consume as a chaser, the chaser is marked as a chaser C2 by transmitting a Boolean parameter P1=true, at the moment, the chaser C2 begins to consume from the latest offset of the message queue, namely, the latest data is consumed, the consumer continues to consume according to the original flow, the consumer quotes as keys, the corresponding chaser is regarded as Value, and the Value is stored in a map;
step S40, mapping between the consumer and the chaser is scanned regularly, if the state of the consumer is found to be in a delayed recovery state, the current consumption offset of the message queue partition of the chaser C2 is obtained, the chaser is closed, the current consumption offset of the chaser is set to the consumer C1, the consumer C1 continues to consume from the consumption position of the chaser, and the consumer is set to be in a normal state.
Preferably, the marking parameter is a boolean parameter, and the true is taken to represent that the marking parameter is a chaser, and vice versa.
Preferably, the latest consumption offset refers to an absolute value position of latest data of the current message queue.
Preferably, in the step S20, when the consumer starts to cycle to pull data from Kafka, it is first determined whether the production speed increase of the message queue data producer is greater than the message queue consumer increase, if so, the difference between the current consumer offset and the maximum message queue offset is further calculated, otherwise, the difference is not calculated.
Based on the same inventive concept, the present application also provides a device corresponding to the method in the first embodiment, and details of the second embodiment are described in the following.
Example two
Referring to fig. 2 and fig. 4 to fig. 10, the present invention provides a system for Kafka backlog queue catch-up, the system comprising:
the system starting module is used for starting the consumer C1, adding the consumer C1 into a consumer list, consuming the data of the message queue Q1, transmitting a flag parameter to determine whether the consumer is a chaser, if the consumer is the chaser, consuming from the latest consumption offset and recording the offset, otherwise, the consumer consumes the data according to the normal consumption flow of the step S20;
the normal consumption module is used for starting a normal consumption flow, a consumer starts to circularly pull data from Kafka, when the data are pulled, the difference between the current consumer offset and the maximum offset of the message queue is calculated (namely, the difference is made between the current consumer offset and the maximum offset of the message queue, the absolute value of the current consumer offset and the maximum offset of the message queue is obtained, the absolute value of the current consumer offset and the maximum offset of the message queue are obtained, the current consumer offset and the maximum offset of the message queue are obtained to obtain a delay value, if the difference is larger than a threshold value (the current consumer offset can be adjusted according to the requirement), the consumer is set to be in a delay state, and if the consumer already consumes the offset of the catch-up consumer in the starting process and the consumer state is in the delay state, the consumer is set to be in a delay recovery state; if the difference value is smaller than or equal to the threshold value, returning to the normal consumption module to continue execution;
the chasing-person adding module is used for regularly scanning the consumer list, if the state of the consumer is found to be a delay state, starting a new consumer aiming at the same queue to consume as a chasing-person, marking the chasing-person as a chasing-person C2 by transmitting a Boolean parameter P1=true, wherein the chasing-person C2 begins to consume from the latest offset of the message queue, namely consumes the latest data, the consumer continues to consume according to the original flow, and takes the consumer reference as Key, and the corresponding chasing-person as Value to be stored in the map; and
and the delay recovery module is used for regularly scanning the mapping between the consumer and the chaser, if the state of the consumer is found to be the delay recovery state, acquiring the current consumption offset of the message queue partition of the chaser C2, closing the chaser, setting the current consumption offset of the chaser to the consumer C1, continuing to consume the consumer C1 from the consumption position of the chaser, and setting the consumer to be in a normal state.
Preferably, the marking parameter is a boolean parameter, and the true is taken to represent that the marking parameter is a chaser, and vice versa.
Preferably, the latest consumption offset refers to an absolute value position of latest data of the current message queue.
Preferably, in the normal consumption module, when the consumer starts to circularly pull data from Kafka, it is first determined whether the production speed increase of the message queue data producer is greater than the message queue consumer increase, if so, a difference between the current consumer offset and the maximum message queue offset is further calculated, otherwise, the difference is not calculated.
Since the device described in the second embodiment of the present invention is a device for implementing the method described in the first embodiment of the present invention, based on the method described in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and the deformation of the device, and thus the detailed description thereof is omitted herein. All devices used in the method according to the first embodiment of the present invention are within the scope of the present invention.
Based on the same inventive concept, the application provides a storage medium corresponding to the first embodiment, and the third embodiment is described in detail.
Example III
The present embodiment provides a computer readable storage medium, as shown in fig. 3, on which a computer program is stored, which when executed by a processor, can implement any implementation of the first embodiment.
The technical scheme provided in the embodiment of the application has at least the following technical effects or advantages: and setting a newly added consumption catch-up person for the backlog queue, so that the catch-up person pulls the message from the latest offset, and the consumer also continues to pull the message in the queue at the same time, thereby realizing automatic catch-up, accelerating the consumption of the backlog queue, and simultaneously setting an automatic recovery mechanism, relieving data conflict, improving the overall processing efficiency of the system and improving the timeliness of the data.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that the specific embodiments described are illustrative only and not intended to limit the scope of the invention, and that equivalent modifications and variations of the invention in light of the spirit of the invention will be covered by the claims of the present invention.
Claims (3)
1. A method for Kafka backlog queue catch-up, characterized by: the method comprises the following steps:
step S10, starting a consumer, adding the consumer to a consumer list, and transmitting a marking parameter to determine whether the consumer is a catch-up person, if so, starting consuming from the latest consumption offset and recording the offset, otherwise, consuming data by the consumer according to the normal consumption flow of the step S20;
step S20, starting a normal consumption flow, wherein a consumer starts to circularly pull data from Kafka, when the data is pulled, calculating a difference value between the current consumer offset and the maximum message queue offset, setting the consumer to a delay state if the difference value is larger than a threshold value, and setting the consumer to a delay recovery state if the consumer has consumed the offset when the chaser is started and the consumer state is the delay state; if the difference value is smaller than or equal to the threshold value, returning to the step S20 to continue execution;
step S30, a consumer list is scanned regularly, if the state of a consumer is found to be a delay state, a new consumer aiming at the same queue is started to be consumed as a chaser, and the consumer and the chaser corresponding to the consumer are stored in a map;
step S40, mapping between a consumer and a chaser is scanned regularly, if the state of the consumer is found to be a delayed recovery state, the current consumption offset of a message queue partition of the chaser is obtained, the chaser is closed, the current consumption offset of the chaser is set to the consumer, the consumer continues to consume from the consumption position of the chaser, and the consumer is set to be in a normal state;
the marking parameter is a Boolean parameter, and the true is taken to represent that the marker is a catch-up person, and the consumer is the opposite;
the latest consumption offset refers to the absolute value position of the latest data of the current message queue;
in the step S20, when the consumer starts to circularly pull data from Kafka, it is first determined whether the production speed increase of the message queue data producer is greater than the message queue consumer increase, if so, the difference between the current consumer offset and the message queue maximum offset is further calculated, otherwise, the difference is not calculated.
2. A system for Kafka backlog queue catch-up, characterized by: the system comprises:
the system starting module is used for starting the consumer, adding the consumer into the consumer list, and transmitting a marking parameter to determine whether the consumer is a catch-up person, if the consumer is the catch-up person, starting consuming from the latest consumption offset and recording the offset, otherwise, the consumer consumes data according to the normal consumption flow of the step S20;
the normal consumption module is used for starting a normal consumption flow, a consumer starts to circularly pull data from Kafka, when the data is pulled, the difference between the current consumer offset and the maximum message queue offset is calculated, if the difference is larger than a threshold value, the consumer is set to be in a delay state, and if the consumer has consumed the offset when the chaser starts and the consumer state is in the delay state, the consumer is set to be in a delay recovery state; if the difference value is smaller than or equal to the threshold value, returning to the normal consumption module to continue execution;
the catch-up person adding module is used for regularly scanning the consumer list, starting a new consumer aiming at the same queue to consume as a catch-up person if the state of the consumer is found to be a delay state, and storing the consumer and the corresponding catch-up person in a map; and
the delay recovery module is used for regularly scanning the mapping between the consumer and the chaser, if the state of the consumer is found to be the delay recovery state, acquiring the current consumption offset of the message queue partition of the chaser, closing the chaser, setting the current consumption offset of the chaser to the consumer, continuing to consume from the consumption position of the chaser by the consumer, and setting the consumer to be in a normal state;
the marking parameter is a Boolean parameter, and the true is taken to represent that the marker is a catch-up person, and the consumer is the opposite;
the latest consumption offset refers to the absolute value position of the latest data of the current message queue;
in the normal consumption module, when a consumer starts to circularly pull data from Kafka, firstly judging whether the production speed acceleration of a message queue data producer is larger than the production speed acceleration of the message queue consumer, if so, further calculating the difference between the current consumer offset and the maximum message queue offset, and if not, not calculating the difference.
3. A computer readable storage medium, on which a computer program is stored, which program, when being executed by a processor, implements the method according to claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111133495.1A CN113872888B (en) | 2021-09-27 | 2021-09-27 | Method, system and medium for catch-up of Kafka backlog queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111133495.1A CN113872888B (en) | 2021-09-27 | 2021-09-27 | Method, system and medium for catch-up of Kafka backlog queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113872888A CN113872888A (en) | 2021-12-31 |
| CN113872888B true CN113872888B (en) | 2023-05-19 |
Family
ID=78990859
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111133495.1A Active CN113872888B (en) | 2021-09-27 | 2021-09-27 | Method, system and medium for catch-up of Kafka backlog queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113872888B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115051963B (en) * | 2022-06-06 | 2024-01-26 | 阿里巴巴（中国）有限公司 | Message processing method and device, message queue system and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502402A (en) * | 2019-08-28 | 2019-11-26 | 中国联合网络通信集团有限公司 | Message treatment method, equipment and terminal device |
| CN112445626A (en) * | 2019-08-29 | 2021-03-05 | 北京京东振世信息技术有限公司 | Data processing method and device based on message middleware |
| WO2021129477A1 (en) * | 2019-12-27 | 2021-07-01 | 华为技术有限公司 | Data synchronization method and related device |
-
2021
- 2021-09-27 CN CN202111133495.1A patent/CN113872888B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502402A (en) * | 2019-08-28 | 2019-11-26 | 中国联合网络通信集团有限公司 | Message treatment method, equipment and terminal device |
| CN112445626A (en) * | 2019-08-29 | 2021-03-05 | 北京京东振世信息技术有限公司 | Data processing method and device based on message middleware |
| WO2021129477A1 (en) * | 2019-12-27 | 2021-07-01 | 华为技术有限公司 | Data synchronization method and related device |
Non-Patent Citations (2)
| Title |
|---|
| 海量数据存储和准实时查询系统设计与实现;齐方方;《中国优秀硕士学位论文全文数据库 信息科技辑》;全文 * |
| 高速公路收费及管理系统中分布式消息系统的应用;方瑜庆;;中国交通信息化(第01期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113872888A (en) | 2021-12-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2871809B1 (en) | Message processing method, device and system for internet of things | |
| US8139596B2 (en) | Communicating prioritized messages to a destination queue from multiple source queues using source-queue-specific priority values | |
| CN111314422A (en) | Kafka-based message processing method and system, storage medium and computer equipment | |
| CN113872888B (en) | Method, system and medium for catch-up of Kafka backlog queue | |
| US8001150B2 (en) | Device management method using nodes having additional attribute and device management client thereof | |
| CN107786623B (en) | Message asynchronous processing method and device | |
| US10417062B2 (en) | Method and apparatus of unloading out of memory processing flow to user space | |
| CN108667705A (en) | A kind of referee method of CANFD buses | |
| CN108008950B (en) | Method and device for realizing user interface updating | |
| CN111949417B (en) | Message transmission method and storage medium | |
| CN110532205A (en) | Data transmission method, device, computer equipment and computer readable storage medium | |
| US8132171B2 (en) | Method of controlling thread access to a synchronization object | |
| WO2023103865A1 (en) | Container pre-loading method, electronic device, and storage medium | |
| CN115001897B (en) | Communication method and device, electronic equipment and automatic driving vehicle | |
| US20140108642A1 (en) | Efficient Reliable Distributed Flow-controlled Event Propagation | |
| CN113031864A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114330722A (en) | Inference implementation method, network, electronic device and storage medium | |
| CN112069195A (en) | Database-based message transmission method and device, electronic equipment and storage medium | |
| US20190132105A1 (en) | Parent node device, terminal device for wireless network and data transmission method thereof | |
| CN113127179A (en) | Resource scheduling method and device, electronic equipment and computer readable medium | |
| CN113419437B (en) | Intelligent home data synchronization method and device based on MVVM (multifunction vehicle management model) framework and MQTT (message queuing time) protocol | |
| CN107832265A (en) | The cpu load Forecasting Methodology of desktop based on state aware | |
| EP4295297A1 (en) | Distributed virtual data tank for cross service quota management | |
| CN113239061B (en) | Intelligent court data updating method and system | |
| CN110109760B (en) | Memory resource control method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |