CN112488705A - Kafka low-order consumption program message backlog monitoring method and related equipment - Google Patents
Kafka low-order consumption program message backlog monitoring method and related equipment Download PDFInfo
- Publication number
- CN112488705A CN112488705A CN202011283068.7A CN202011283068A CN112488705A CN 112488705 A CN112488705 A CN 112488705A CN 202011283068 A CN202011283068 A CN 202011283068A CN 112488705 A CN112488705 A CN 112488705A
- Authority
- CN
- China
- Prior art keywords
- kafka
- message
- messages
- batch
- hash value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012544 monitoring process Methods 0.000 title claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 26
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000004590 computer program Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/389—Keeping log of transactions for guaranteeing non-repudiation of a transaction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Accounting & Taxation (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Finance (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the specification provides a Kafka low-order consumption program message backlog monitoring method and related equipment, which are used for monitoring and alarming message backlogs in a Kafka low-order consumption program and judging the repetition in the message processing process. Firstly, obtaining messages in batches according to a preset time interval in a Kafka consumption program, recording the maximum offset of the messages of one batch after the messages of the batch are consumed, obtaining the latest offset through a public interface, and subtracting the maximum offset from the latest offset to obtain the backlog of the messages after the messages of the batch are consumed; setting a backlog amount threshold, and sending an alarm mail to a technician by the consumption program when the calculated backlog amount exceeds the threshold. In the message processing process, calculating and storing the hash value of the processed message; when the message is processed, the hash value of the message to be processed is calculated and compared with the stored hash value, and the message to be processed is processed continuously when the hash value is inconsistent with the stored hash value.
Description
Technical Field
One or more embodiments of the present description relate to the field of Kafka consumer programs, and more particularly, to Kafka low-level consumer program message backlog monitoring in Kafka low-level consumer programs.
Background
The low-order Kafka consumption program maintains consumption records by the program itself, the current message backlog can be known by acquiring consumption information records and the maximum message records, the offset of the consumption information records is subtracted from the maximum message offset, and the obtained difference value is the current consumption backlog of the consumption program and is used for monitoring the running state of the program, and how to reasonably record consumption information becomes a consideration factor for program design and monitoring of the message backlog.
At present, consumption records are carried out once when some Kafka low-level consumption programs process one message, the consumption records are too frequent, and when the message amount is thousands or tens of thousands of messages per second, great pressure is caused to the programs and the recording equipment.
Disclosure of Invention
In view of the above, one or more embodiments of the present disclosure are directed to a method and related device for monitoring the backlog of Kafka low-level consumer program messages, so as to solve the problem that consumer records are too frequent and the program and recording device are stressed too much.
One or more embodiments of the present specification provide a Kafka low-order consumer program message backlog monitoring method, comprising:
obtaining Kafka messages in batches by a Kafka low-order consumption program, and obtaining a maximum message offset from a Kafka cluster through a Kafka public interface;
for each batch of Kafka messages captured, the following operations are performed by the Kafka low-level consuming program: recording the maximum offset of the batch of Kafka messages after the batch of Kafka messages is consumed;
taking the difference between the obtained maximum message offset and the recorded maximum offset as the backlog of real-time messages of the Kafka low-level consumption program;
and sending out alarm information when the backlog quantity of the real-time message exceeds a preset backlog quantity threshold value.
One or more embodiments of the present specification provide that the Kafka low-order consumer program message backlog monitoring method further includes recording a hash value of the processed message to perform deduplication:
when the acquired messages in each batch of Kafka messages are processed, calculating the hash value of the messages through a hash algorithm, and recording the hash value as the processing identification of the messages;
when a target Kafka message is to be processed, calculating a target hash value of the target Kafka message through a hash algorithm;
and comparing the target hash value with each recorded processing identifier, and determining not to process the target Kafka message if the comparison result indicates that the target hash value is consistent with at least one recorded processing identifier.
Based on the same inventive concept, one or more embodiments of the present specification further provide a Kafka low-order consumption program message backlog monitoring apparatus, which may be divided into the following modules according to the implemented functions:
the acquisition module acquires Kafka messages in batches and records the maximum offset of each batch of messages;
the monitoring module executes the following operations for each batch of Kafka messages acquired by the acquisition module: recording the maximum offset of the batch of Kafka messages after the batch of Kafka messages is consumed; taking the difference between the maximum message offset obtained by the obtaining module and the recorded maximum offset as the backlog of the real-time messages of the Kafka low-order consumption program; when the backlog amount of the real-time message exceeds a preset backlog amount threshold value, sending out alarm information;
the identification module is used for calculating a hash value of each batch of Kafka messages acquired by the acquisition module through a hash algorithm when the messages are processed, and recording the hash value as a processing identification of the messages;
and the judging module is used for calculating a target hash value of a target Kafka message through a hash algorithm when the target Kafka message is to be processed, comparing the target hash value with each processing identifier recorded by the identification module, and determining not to process the target Kafka message if the comparison result indicates that the target hash value is consistent with at least one recorded processing identifier.
Based on the same inventive concept, one or more embodiments of the present specification further provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the Kafka low-order consumption program message backlog monitoring method described above.
Based on the same inventive concept, one or more embodiments of the present specification further provide a storage medium, which is a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the Kafka low-order consumption program message backlog monitoring method described above.
From the above description, it can be seen that the Kafka low-level consumption program message backlog monitoring method and the related device provided by one or more embodiments of the present disclosure only need to perform consumption recording once in hundreds of thousands of messages per batch, thereby greatly reducing the pressure of the Kafka consumption program and the record source.
Drawings
In order to more clearly illustrate one or more embodiments or prior art solutions of the present specification, the drawings that are needed in the description of the embodiments or prior art will be briefly described below, and it is obvious that the drawings in the following description are only one or more embodiments of the present specification, and that other drawings may be obtained by those skilled in the art without inventive effort from these drawings.
FIG. 1 is a flow diagram of a Kafka low-level consumer program message backlog monitoring method implemented by one or more embodiments of the present disclosure;
FIG. 2 is a flow diagram illustrating how Kafka's low-order consumer processes messages in one or more embodiments of the present disclosure;
FIG. 3 is a schematic diagram of a Kafka low level consumer program message backlog monitor device in one or more embodiments of the present disclosure;
fig. 4 is a schematic diagram of an electronic device implementing Kafka low-level consumer program message backlog monitoring in one or more embodiments of the present disclosure.
Detailed Description
For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
It is to be noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this disclosure belongs. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items.
As described in the background, in the existing Kafka low-level consumption program, consumption records are performed once per processed message, and when the message volume reaches thousands or tens of thousands per second, great pressure is exerted on the Kafka consumption program itself and the message library for recording consumption records.
To this end, one or more embodiments of the present specification propose a Kafka low-order consumer program monitoring method that acquires Kafka messages in batches by setting a time interval; when a batch of Kafka messages are consumed, recording the offset of the last message in each batch of messages as the maximum offset of each batch of messages; subtracting the maximum offset of each batch of messages from the latest offset of the messages acquired through the Kafka public interface to obtain a difference value, namely the backlog quantity of each batch of messages after the consumption is finished; meanwhile, a backlog amount threshold is set by a technician, and when the backlog amount exceeds the backlog amount threshold, an alarm mail is actively sent to the technician by the Kafka consumption program. Meanwhile, because the number of the messages to be processed in each batch of messages is uncertain, screening needs to be performed according to a preset screening condition of service requirements, and the hash value of each processed message is recorded; and in the message processing process, calculating the hash value of the message to be processed and comparing the hash value of the processed message, wherein the inconsistency of the hash value of the message to be processed and the hash value of the processed message proves that the message to be processed is not processed yet, and the message to be processed can be processed continuously, otherwise, the message to be processed is skipped to verify and process the next message to be processed. This approach greatly reduces the pressure on the program and recording sources, and also meets the monitoring requirements for the Kafka consumption program. In addition, the hash value of the message being processed is compared with the hash value of the processed message, so that each message needing to be processed is guaranteed to be processed only once.
Referring to fig. 1, the steps of the technical solution expressed in one or more embodiments of the present specification are as follows.
Step S101, Kafka messages are acquired in batches by the Kafka program, and the maximum message offset is acquired from the Kafka cluster through the Kafka public interface.
In the step, the time interval for acquiring Kafka messages in batches can be set to be 1-10 seconds, so that the number of messages in each batch is controlled to be tens of thousands to hundreds of thousands; in the Kafka consumption procedure, the offsets of the messages are accumulated as the number of messages increases, one for each message. The maximum message offset that is obtained is the latest message offset.
Step S102, for each batch of acquired Kafka messages, performing the following operations by the Kafka low-order consumption program: after the batch of Kafka messages is consumed, the maximum offset for the batch of Kafka messages is recorded.
In this step, the maximum recorded offset is the offset of the last message in the batch of messages.
And step S103, taking the difference between the acquired maximum message offset and the recorded maximum offset as the real-time message backlog of the Kafka low-order consumption program.
And step S104, when the backlog amount of the real-time message exceeds a preset backlog amount threshold value, sending out alarm information.
In the step, a backlog threshold is set by a technician according to actual conditions, when the backlog of the real-time message exceeds the threshold, the technician is reminded to process in a mode of sending an alarm mail, and a general processing mode is to divide more computing resources for the Kafka low-order consumption program.
In the Kafka low-level consumption program monitoring method provided by one or more embodiments of the present specification, consumption records of a new batch of Kafka messages overlap consumption records of a previous batch.
One or more embodiments of the present specification further provide a method for re-judging a message to be processed, which is described below with reference to fig. 2.
Step S201, when the acquired messages in each batch of Kafka messages are processed, calculating a hash value of the messages by using a hash algorithm, and recording the hash value as a processing identifier of the messages.
In this step, according to the service requirement, a technician presets a screening condition to screen the Kafka messages of each batch to obtain the messages to be processed, and simultaneously calculates the hash value of the consumed messages and stores the hash value in a Set class in a remote dictionary service (Redis).
Step S202, when a target Kafka message is to be processed, calculating a target hash value of the target Kafka message through a hash algorithm.
Step S203, comparing the target hash value with each recorded processing identifier, and if the result of the comparison indicates that the target hash value is consistent with at least one recorded processing identifier, determining not to consume the target Kafka message.
The use scenario of the method is as follows, when a Kafka low-level consuming program suddenly stops consuming a certain batch of data, the process of processing the message is interrupted. When the program is restarted, the batch of data is re-screened to obtain the message to be processed; since the hash value of the processed message is recorded, the hash value of the message to be processed is compared with the hash value recorded in the Set class, so that the repeated processing of the processed message is avoided.
It should be noted that the method of one or more embodiments of the present disclosure may be performed by a single device, such as a computer or server. The method of the embodiment can also be applied to a distributed scene and completed by the mutual cooperation of a plurality of devices. In such a distributed scenario, one of the devices may perform only one or more steps of the method of one or more embodiments of the present disclosure, and the devices may interact with each other to complete the method.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Based on the same inventive concept, corresponding to any of the above embodiments, one or more embodiments of the present disclosure further provide a Kafka low-level consumer program message backlog monitoring apparatus, which can be divided into the following modules according to its functions, with reference to fig. 3.
The fetch module 301 fetches Kafka messages in batches and records the maximum offset for each batch of messages.
The monitoring module 302 performs the following operations for each batch of Kafka messages acquired by the acquisition module: recording the maximum offset of the batch of Kafka messages after the batch of Kafka messages is consumed; taking the difference between the maximum message offset obtained by the obtaining module and the recorded maximum offset as the backlog of the real-time messages of the Kafka low-order consumption program; and sending out alarm information when the backlog quantity of the real-time message exceeds a preset backlog quantity threshold value.
And the identification module 303, when the message in each batch of Kafka messages acquired by the acquisition module is processed, calculating a hash value of the message by using a hash algorithm, and recording the hash value as a processing identification of the message.
And the duplication decision module 304 is configured to, when a target Kafka message is to be processed, calculate a target hash value of the target Kafka message through a hash algorithm, compare the target hash value with each processing identifier recorded by the identifier module, and determine not to process the target Kafka message if the comparison result indicates that the target hash value is consistent with at least one recorded processing identifier.
For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, the functionality of the modules may be implemented in the same one or more software and/or hardware implementations in implementing one or more embodiments of the present description.
The apparatus in the foregoing embodiment is used to implement the corresponding Kafka low-order consumption program message backlog monitoring method in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Based on the same inventive concept, corresponding to any of the above embodiments, one or more embodiments of the present specification further provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor can implement the Kafka low-level consumer program message backlog monitoring method when executing the program.
Fig. 4 is a schematic diagram illustrating a more specific hardware structure of an electronic device according to this embodiment, where the electronic device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein the processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 are communicatively coupled to each other within the device via bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 1020 and called to be executed by the processor 1010.
The input/output interface 1030 is used for connecting an input/output module to input and output information. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 1040 is used for connecting a communication module (not shown in the drawings) to implement communication interaction between the present apparatus and other apparatuses. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
The electronic device in the foregoing embodiment is used to implement the corresponding Kafka low-order consumer program message backlog monitoring method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, one or more embodiments of the present specification further provide a storage medium, which is a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the Kafka low-order consumer program message backlog monitoring method described above.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, message structures, modules of a program, or other messages. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the Kafka low-order consumption program message backlog monitoring method according to any of the above embodiments, and have the beneficial effects of corresponding method embodiments, which are not described herein again.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the spirit of the present disclosure, features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present description as described above, which are not provided in detail for the sake of brevity.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures, for simplicity of illustration and discussion, and so as not to obscure one or more embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the understanding of one or more embodiments of the present description, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the one or more embodiments of the present description are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
It is intended that the one or more embodiments of the present specification embrace all such alternatives, modifications and variations as fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of one or more embodiments of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (10)
1. A Kafka low-order consumer program message backlog monitoring method comprises the following steps:
obtaining Kafka messages in batches by a Kafka low-order consumption program, and obtaining a maximum message offset from a Kafka cluster through a Kafka public interface;
for each batch of Kafka messages captured, the following operations are performed by the Kafka low-level consuming program: recording the maximum offset of the batch of Kafka messages after the batch of Kafka messages is consumed;
taking the difference between the obtained maximum message offset and the recorded maximum offset as the backlog of real-time messages of the Kafka low-level consumption program;
and sending out alarm information when the backlog quantity of the real-time message exceeds a preset backlog quantity threshold value.
2. The method of claim 1, wherein,
when the acquired messages in each batch of Kafka messages are processed, calculating the hash value of the messages through a hash algorithm, and recording the hash value as the processing identification of the messages;
when a target Kafka message is to be processed, calculating a target hash value of the target Kafka message through a hash algorithm;
and comparing the target hash value with each recorded processing identifier, and determining not to process the target Kafka message if the comparison result indicates that the target hash value is consistent with at least one recorded processing identifier.
3. The method of claim 1, wherein batch fetching of Kafka messages comprises: the Kafka messages are acquired in batches at time intervals of 1 to 10 seconds.
4. The method of any of claims 1 to 3, wherein the maximum offset of the batch of Kafka messages is the offset of the last message in the batch of Kafka messages.
5. The method of any one of claims 1 to 3, wherein the maximum offset for each batch of Kafka messages is recorded iteratively.
6. A Kafka low-order consumer program message backlog monitoring apparatus, comprising:
the acquisition module is configured to acquire Kafka messages in batches and acquire the maximum message offset from the Kafka cluster through a Kafka public interface;
a monitoring module configured to perform the following operations for each batch of Kafka messages acquired by the acquisition module: recording the maximum offset of the batch of Kafka messages after the batch of Kafka messages is consumed; taking the difference between the maximum message offset obtained by the obtaining module and the recorded maximum offset as the backlog of the real-time messages of the Kafka low-order consumption program; and sending out alarm information when the backlog quantity of the real-time message exceeds a preset backlog quantity threshold value.
7. The apparatus of claim 6, further comprising:
an identification module configured to: when the information in each batch of Kafka information acquired by the acquisition module is processed, calculating the Hash value of the information by a Hash algorithm, and recording the Hash value as the processing identification of the information;
a re-determination module configured to: when a target Kafka message is to be processed, calculating a target hash value of the target Kafka message through a hash algorithm, comparing the target hash value with each processing identifier recorded by the identifier module, and determining not to process the target Kafka message if the comparison result indicates that the target hash value is consistent with at least one recorded processing identifier.
8. The apparatus of claim 6 or 7, wherein the maximum offset of the batch of Kafka messages is the offset of the last message in the batch of Kafka messages.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, wherein the processor implements the method of any of claims 1 to 5 when executing the computer program.
10. A non-transitory computer readable storage medium having stored thereon computer instructions which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011283068.7A CN112488705A (en) | 2020-11-16 | 2020-11-16 | Kafka low-order consumption program message backlog monitoring method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011283068.7A CN112488705A (en) | 2020-11-16 | 2020-11-16 | Kafka low-order consumption program message backlog monitoring method and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112488705A true CN112488705A (en) | 2021-03-12 |
Family
ID=74931599
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011283068.7A Pending CN112488705A (en) | 2020-11-16 | 2020-11-16 | Kafka low-order consumption program message backlog monitoring method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112488705A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672459A (en) * | 2021-08-18 | 2021-11-19 | 北京沃东天骏信息技术有限公司 | Information processing method and device and storage medium |
| CN113886330A (en) * | 2021-10-13 | 2022-01-04 | 武汉达梦数据库股份有限公司 | Method and device for supporting dynamic reading of hive table data in stream processing mode |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109493076A (en) * | 2018-11-09 | 2019-03-19 | 武汉斗鱼网络科技有限公司 | A kind of unique consuming method of Kafka message, system, server and storage medium |
| CN110287038A (en) * | 2019-06-10 | 2019-09-27 | 天翼电子商务有限公司 | Promote the method and system of the data-handling efficiency of Spark Streaming frame |
| CN110535787A (en) * | 2019-07-25 | 2019-12-03 | 北京奇艺世纪科技有限公司 | Information consumption method, apparatus and readable storage medium storing program for executing |
| US10776441B1 (en) * | 2018-10-01 | 2020-09-15 | Splunk Inc. | Visual programming for iterative publish-subscribe message processing system |
| CN111897662A (en) * | 2020-07-01 | 2020-11-06 | 中国建设银行股份有限公司 | Offset processing method and device, electronic equipment and computer readable storage medium |
| CN111949418A (en) * | 2020-07-10 | 2020-11-17 | 北京思特奇信息技术股份有限公司 | Subscription message processing method and device |
-
2020
- 2020-11-16 CN CN202011283068.7A patent/CN112488705A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10776441B1 (en) * | 2018-10-01 | 2020-09-15 | Splunk Inc. | Visual programming for iterative publish-subscribe message processing system |
| CN109493076A (en) * | 2018-11-09 | 2019-03-19 | 武汉斗鱼网络科技有限公司 | A kind of unique consuming method of Kafka message, system, server and storage medium |
| CN110287038A (en) * | 2019-06-10 | 2019-09-27 | 天翼电子商务有限公司 | Promote the method and system of the data-handling efficiency of Spark Streaming frame |
| CN110535787A (en) * | 2019-07-25 | 2019-12-03 | 北京奇艺世纪科技有限公司 | Information consumption method, apparatus and readable storage medium storing program for executing |
| CN111897662A (en) * | 2020-07-01 | 2020-11-06 | 中国建设银行股份有限公司 | Offset processing method and device, electronic equipment and computer readable storage medium |
| CN111949418A (en) * | 2020-07-10 | 2020-11-17 | 北京思特奇信息技术股份有限公司 | Subscription message processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 高宗宝等: "Spark 平台中Kafka 偏移量的读取管理与设计", 软件, pages 118 - 122 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672459A (en) * | 2021-08-18 | 2021-11-19 | 北京沃东天骏信息技术有限公司 | Information processing method and device and storage medium |
| CN113886330A (en) * | 2021-10-13 | 2022-01-04 | 武汉达梦数据库股份有限公司 | Method and device for supporting dynamic reading of hive table data in stream processing mode |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TW202036356A (en) | Gradient boosting decision tree-based method and device for model training | |
| CN112488705A (en) | Kafka low-order consumption program message backlog monitoring method and related equipment | |
| US9436657B2 (en) | Computing device and method for analyzing acquisition values | |
| CN109669798B (en) | Crash analysis method, crash analysis device, electronic equipment and storage medium | |
| CN111198859A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN111612158A (en) | Model deployment method, device, equipment and storage medium | |
| CN111488170A (en) | Method, device and equipment for updating business processing model | |
| CN112672405B (en) | Power consumption calculation method, device, storage medium, electronic equipment and server | |
| CN108334429A (en) | Method, apparatus and system for investigating front end page problem | |
| CN105760280B (en) | Power consumption monitoring system and method | |
| CN111798263A (en) | Transaction trend prediction method and device | |
| US10311032B2 (en) | Recording medium, log management method, and log management apparatus | |
| CN111274104A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN111198853A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| JP6553650B2 (en) | Data processing method and system | |
| US20140335794A1 (en) | System and Method for Automated Testing of Mobile Computing Devices | |
| CN108712284B (en) | Fault service positioning method and device and service server | |
| CN114968696A (en) | Index monitoring method, electronic equipment and chip system | |
| CN111610732B (en) | Method, device and equipment for replacing configurable input/output module | |
| CN116962111A (en) | CAN bus log analysis and transmission method, related equipment and vehicle | |
| CN110032488B (en) | Monitoring system, method and device for specific nodes in cluster and service server | |
| CN115906368A (en) | Temperature signal real-time load spectrum calculation method and related equipment | |
| CN115357138A (en) | Screen debugging method, device and medium | |
| CN115984735A (en) | Video instance segmentation method and device, electronic equipment and storage medium | |
| CN114430190A (en) | Battery charging method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |