CN114546681A - Kafka-based message processing method, device, equipment and storage medium - Google Patents
Kafka-based message processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114546681A CN114546681A CN202210159014.2A CN202210159014A CN114546681A CN 114546681 A CN114546681 A CN 114546681A CN 202210159014 A CN202210159014 A CN 202210159014A CN 114546681 A CN114546681 A CN 114546681A
- Authority
- CN
- China
- Prior art keywords
- message
- delay
- kafka
- target
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/56—Queue scheduling implementing delay-aware scheduling
- H04L47/562—Attaching a time tag to queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application relates to the technical field of message queues, and provides a Kafka-based message processing method, device, equipment and storage medium, wherein the method comprises the following steps: when the consumption of the target message fails, generating a delay message corresponding to the target message, wherein the delay message comprises the message body content of the target message; sending the delay message to a pre-configured target Kafka delay message queue; when the delay message in the target Kafka delay message queue is delayed to expire, extracting the message body content in the delay message, and generating a retransmission message corresponding to the target message according to the message body content; and sending the retransmission message to a Kafka message queue so that a consumption end can obtain the retransmission message from the Kafka message queue for consumption, thereby realizing the improvement of the success rate of message consumption. The application also relates to blockchain technology, and the Kafka message queue can be stored in a blockchain node.
Description
Technical Field
The present application relates to the field of message queue technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing a message based on Kafka.
Background
Kafka is a distributed publish/subscribe schema-based message queuing system that handles all the action flow data of consumers in a web site. Kafka can be commonly applied to access logs, message services and the like. At present, the Kafka system only provides basic message receiving and message sending functions, and does not provide message processing functions in a message consumption failure scene.
Therefore, how to process the consumption failure message based on Kafka to improve the success rate of message consumption becomes an urgent problem to be solved.
Disclosure of Invention
The application provides a Kafka-based message processing method, device, equipment and storage medium, and aims to realize Kafka-based message processing of consumption failure so as to improve message consumption success rate.
In order to achieve the above object, the present application provides a message processing method based on Kafka, where the message processing method based on Kafka includes:
when the consumption of the target message fails, generating a delay message corresponding to the target message, wherein the delay message comprises the message body content of the target message;
sending the delay message to a pre-configured target Kafka delay message queue;
when the delay message in the target Kafka delay message queue is delayed to expire, extracting the message body content in the delay message, and generating a retransmission message corresponding to the target message according to the message body content;
and sending the retransmission message to a Kafka message queue so that a consumption end can obtain the retransmission message from the Kafka message queue for consumption.
In addition, to achieve the above object, the present application further provides a Kafka-based message processing apparatus, including:
the message generating module is used for generating a delay message corresponding to a target message when the consumption of the target message fails, wherein the delay message comprises the message body content of the target message;
the message sending module is used for sending the delay message to a pre-configured target Kafka delay message queue;
the message management module is used for extracting the message body content in the delayed message when the delay message in the target Kafka delayed message queue is delayed to expire, and generating a retransmission message corresponding to the target message according to the message body content;
and the message consumption module is used for sending the retransmission message to a Kafka message queue so that a consumption end can obtain the retransmission message from the Kafka message queue for consumption.
In addition, to achieve the above object, the present application also provides a computer device comprising a memory and a processor;
the memory for storing a computer program;
the processor is configured to execute the computer program and implement the Kafka-based message processing method as described above when the computer program is executed.
Further, to achieve the above object, the present application also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the above Kafka-based message processing method.
When the consumption of the target message fails, generating a delay message corresponding to the target message, wherein the delay message comprises the message body content of the target message, sending the delay message to a preset target Kafka delay message queue, and when the delay message in the target Kafka delay message queue expires, extracting the message body content in the delay message, generating a retransmission message corresponding to the target message, and sending the retransmission message to the Kafka message queue so that a consumption end can obtain the retransmission message from the Kafka message queue for consumption. The retransmission message corresponding to the target message is generated, and the retransmission message is consumed, namely the target message is consumed again, so that the success rate of target message consumption is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart illustrating steps of a message processing method based on Kafka according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a step of generating a delay message corresponding to the target message according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of steps for sending the delay message to a pre-configured target Kafka delay message queue according to an embodiment of the present application;
fig. 4 is a schematic diagram of a message processing flow based on Kafka according to an embodiment of the present application;
fig. 5 is a schematic block diagram of a Kafka-based message processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic block diagram of a structure of a computer device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
It is to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Embodiments of the present application provide a Kafka-based message processing method, apparatus, device, and storage medium, which are used to implement monitoring of a web page on a user perceivable layer.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a message processing method based on Kafka according to an embodiment of the present application. The method can be applied to computer equipment, and the application scene of the method is not limited in the application. The message processing method based on Kafka will be described in detail below by taking an example in which the message processing method based on Kafka is applied to a computer device.
As shown in fig. 1, the Kafka-based message processing method specifically includes steps S101 to S104.
S101, when the consumption of the target message fails, generating a delay message corresponding to the target message, wherein the delay message comprises the message body content of the target message.
Illustratively, the message producer sends each message to be consumed to the Kafka message queue, wherein the Kafka message queue includes at least one message. For example, the Kafka message queues include Topic a and Topic B, and each Kafka message queue buffers each message to be consumed.
And the consumption end pulls the target message to be consumed from the Kafka message queue and then performs service processing. For example, taking the target message C to be consumed in the Topic B as an example, the target message C is pulled from the Topic B, the target message C is sent to the consuming end, and the consuming end consumes the target message C to perform corresponding service processing.
When the consumption of the target message fails, the processing of the target message is finished, and other corresponding business processing can be continuously executed.
And when the consumption of the target message fails, generating a delay message corresponding to the target message, wherein the delay message comprises the message body content of the target message. Illustratively, a Kafka delay message manager is configured in advance, and a delay message corresponding to the target message is generated through the Kafka delay message manager.
In some embodiments, as shown in fig. 2, step S101 may include sub-step S1011 and sub-step S1012.
And S1011, obtaining delay information corresponding to the target message.
Illustratively, the delay information includes at least one of a delay level of the target message and a number of times the target message has been retransmitted.
Wherein different delay levels correspond to different delay durations. For example, assume that 3 delay levels are preconfigured: delay level 1, delay level 2, and delay level 3, with delay level 1 corresponding to a delay of 10 minutes, delay level 2 corresponding to a delay of 60 minutes, delay level 3 corresponding to a delay of 24 hours, and so on.
Illustratively, the delay information further includes key corresponding to the target message, Kafka message queue corresponding to the target message, delay message creation timestamp, and the like.
S1012, packaging the message body content of the target message and the delay information according to a preset delay message format, and generating the delay message.
Illustratively, the preconfigured delay message format is: { "key": key of source message "," Topic ": Topic of source message", "value" { \\_ delay _ message _ parameter \ "{ \\\\\\ \ createdDate \" } delayed message creation timestamp, \ "delaytimeLevel \ delay level, \\" retried Count \ the number of times retransmitted }, message body content of source message } "}.
Wherein, the source message is also the target message; "key" and "value" are a pair of key-value pairs, "key" is a key, and "value" is a value; "Topic" is a Kafka message queue, e.g., if the target message was previously cached in Kafka message queue Topic A, then "Topic of Source message" is Topic A.
And obtaining information such as the delay level of the target message, the number of times of retransmission of the target message, the key corresponding to the target message, the Kafka message queue corresponding to the target message, the delay message creation timestamp and the like through a Kafka delay message manager, and encapsulating the delay level and the key corresponding to the target message with the message body content of the target message according to the configured delay message format to generate the delay message corresponding to the target message.
The format of the delayed message may be other than the above-mentioned message format, and the format of the delayed message is not particularly limited herein.
And S102, sending the delay message to a pre-configured target Kafka delay message queue.
Illustratively, at least one Kafka delay message queue is preconfigured, and the Kafka delay message queue is used for buffering the generated delay messages.
After the delay message corresponding to the target message is generated, the Kafka delay message manager determines a target Kafka delay message queue corresponding to the delay message, and sends the delay message to the target Kafka delay message queue.
For example, if there is only one Kafka delay message queue, the target Kafka delay message queue is directly determined to be the Kafka delay message queue, and the delay message is sent to the target Kafka delay message queue.
For example, if the Kafka delay message queue includes a plurality of lines, as shown in fig. 3, step S102 may include sub-step S1021 and sub-step S1022.
S1021, selecting a Kafka delay message queue matched with the delay information from a plurality of pre-configured Kafka delay message queues, and determining the Kafka delay message queue as the target Kafka delay message queue;
s1022, sending the delay message to the target Kafka delay message queue.
For example, different Kafka delay message queues correspondingly buffer delay messages of different message types, based on the message type of the generated delay message, a Kafka delay message queue corresponding to the message type of the delay message is selected from the multiple Kafka delay message queues, and the Kafka delay message queue is determined as a target Kafka delay message queue.
After the target Kafka delayed message queue is determined, the delayed message is sent to the target Kafka delayed message queue through the Kafka delayed message manager, and the delayed message is buffered in the target Kafka delayed message queue.
In some embodiments, selecting a Kafka delayed message queue matching the delay information from a plurality of pre-configured Kafka delayed message queues, and determining as the target Kafka delayed message queue, may include:
acquiring a message retransmitted time range corresponding to each Kafka delay message queue and retransmitted times of the target message;
and when the number of times of retransmission of the target message is within the range of the number of times of retransmission of the message corresponding to the first Kafka delayed message queue, determining the first Kafka delayed message queue as the target Kafka delayed message queue.
When each Kafka delay message queue is configured, the message retransmitted time range corresponding to each Kafka delay message queue is configured. Wherein different Kafka delayed message queues correspond to different ranges of numbers of times that messages have been retransmitted.
For example, N Kafka delay message queues are preconfigured: delay _ Topic _1, Delay _ Topic _2, Delay _ Topic _3, Delay _ Topic _ N, wherein each Kafka delayed message queue corresponds to a respective range of times that a message has been retransmitted. For example, the range of the number of times of message retransmission is [0, 3] for the Delay message queue Delay _ Topic _1, the range of the number of times of message retransmission is (3, 6) for the Delay message queue Delay _ Topic _2, the range of the number of times of message retransmission is (6, 10] for the Delay message queue Delay _ Topic _3, and so on.
And after the delay message corresponding to the target message is generated, acquiring a message retransmitted time range corresponding to each Kafka delay message queue which is configured in advance, and determining the retransmitted time of the target message. And when the number of times of retransmission of the target message is within the range [0, 3] of the number of times of retransmission of the message corresponding to the Kafka delayed message queue Delay _ Topic _1, namely the number of times of retransmission of the target message is not more than 3, sending the generated delayed message corresponding to the target message to the Kafka delayed message queue Delay _ Topic _ 1.
And when the number of times of retransmission of the target message is within the range (3, 6) of the number of times of retransmission of the message corresponding to the Kafka delayed message queue Delay _ Topic _2, namely the number of times of retransmission of the target message exceeds 3 times and does not exceed 6 times, sending the generated delayed message corresponding to the target message to the Kafka delayed message queue Delay _ Topic _ 2.
And when the number of times of retransmission of the target message is within the range (6, 10) of the number of times of retransmission of the message corresponding to the Kafka delayed message queue Delay _ topoc _3, namely the number of times of retransmission of the target message exceeds 6 times and does not exceed 10 times, sending the generated delayed message corresponding to the target message to the Kafka delayed message queue Delay _ topoc _3.
In other embodiments, selecting a Kafka delayed message queue matching the delay information from a plurality of pre-configured Kafka delayed message queues, and determining as the target Kafka delayed message queue, may include:
obtaining a message delay grade corresponding to each Kafka delay message queue and a delay grade of the target message;
and when the delay level of the target message is consistent with the message delay level corresponding to the second Kafka delay message queue, determining the second Kafka delay message queue as the target Kafka delay message queue.
And when each Kafka delay message queue is configured, configuring the message delay level corresponding to each Kafka delay message queue. Wherein different Kafka delay message queues correspond to different message delay levels.
For example, the message Delay level corresponding to the Kafka delayed message queue Delay _ topo _1 is configured in advance to be Delay level 1, the message Delay level corresponding to the Kafka delayed message queue Delay _ topo _2 is configured to be Delay level 2, and the message Delay level corresponding to the Kafka delayed message queue Delay _ topo _3 is configured to be Delay level 3.
And after generating the delay message corresponding to the target message, acquiring the message delay level corresponding to each pre-configured Kafka delay message queue, and determining the delay level of the target message.
In some embodiments, the obtaining a message delay level corresponding to each Kafka delay message queue may include:
inquiring configuration information files corresponding to a plurality of Kafka delay message queues; the configuration information file comprises configuration information corresponding to each Kafka delay message queue;
and extracting the message delay grade corresponding to each Kafka delay message queue from the configuration information corresponding to each Kafka delay message queue.
Illustratively, when configuring the Kafka delay message queues, the configuration information corresponding to each Kafka delay message queue is configured. The configuration information corresponding to the Kafka delayed message queue includes at least one of a message retransmission time range corresponding to the Kafka delayed message queue and a message delay level corresponding to the Kafka delayed message queue. And summarizing the configuration information corresponding to each Kafka delay message queue, generating a corresponding configuration information file and storing the configuration information file.
After the delay message corresponding to the target message is generated, the stored configuration information file is inquired, the configuration information corresponding to each Kafka delay message queue is obtained, and the information such as the message retransmission time range corresponding to each Kafka delay message queue, the message delay grade corresponding to each Kafka delay message queue and the like is extracted from the configuration information corresponding to each Kafka delay message queue.
Illustratively, the mapping relationship between the retransmitted time range and the delay level of the message is preset, for example, the retransmitted time range [0, 3] is set to correspond to the delay level 1, the retransmitted time range (3, 6) is set to correspond to the delay level 2, and the retransmitted time range (6, 10) is set to correspond to the delay level 3.
After the number of times of retransmission of the target message is acquired, the range of the number of times of retransmission of the target message is determined based on the mapping relation between the range of the number of times of retransmission of the message and the delay level, and the delay level corresponding to the range of the number of times of retransmission is determined as the delay level of the target message.
For example, if the number of times of retransmission of the target message is within the range of the number of times of retransmission [0, 3], the delay level of the target message is determined to be delay level 1. And if the number of times of retransmission of the target message is within the range of the number of times of retransmission (3, 6), determining the delay level of the target message to be delay level 2, and if the number of times of retransmission of the target message is within the range of the number of times of retransmission (6, 10), determining the delay level of the target message to be delay level 3.
And when the Delay level of the target message is Delay level 1, sending the Delay message corresponding to the generated target message to a Kafka Delay message queue Delay _ Topic _ 1.
And when the Delay level of the target message is Delay level 2, sending the Delay message corresponding to the generated target message to a Kafka Delay message queue Delay _ Topic _ 2.
And when the Delay level of the target message is Delay level 3, sending the Delay message corresponding to the generated target message to a Kafka Delay message queue Delay _ Topic _3.
S103, when the delay message in the target Kafka delay message queue is delayed to expire, extracting the message body content in the delay message, and generating a retransmission message corresponding to the target message according to the message body content.
Illustratively, the delayed messages in each Kafka delayed message queue are managed by a Kafka delayed message manager to determine whether the delayed messages are delayed to expire. For example, when the Kafka delayed message manager currently pulls the delayed message corresponding to the target message in the target Kafka delayed message queue, it determines whether the delayed message is delayed to expire.
In some embodiments, when the delay message in the target Kafka delay message queue expires, before extracting the message body content in the delay message, the method may include:
adding the delay message creation timestamp to the delay duration corresponding to the delay level, and determining the delay expiration time corresponding to the delay message;
if the current time does not reach the delay expiration time, determining that the delay message does not delay expiration;
and if the current time reaches the delay expiration time, determining that the delay message is delayed to expire.
Illustratively, the Kafka delay message manager extracts information of a delay message creation timestamp, a delay level and the like in the delay message, and then adds the delay message creation timestamp to a delay duration corresponding to the delay level based on the delay duration corresponding to the delay level to determine a delay due time corresponding to the delay message.
For example, if the delay level corresponding to the delay message is delay level 1, and the delay duration corresponding to the delay level 1 is 10 minutes, it is determined that the delay due time corresponding to the delay message is the time corresponding to the delay message creation timestamp plus 10 minutes.
For another example, if the delay level corresponding to the delay message is delay level 2, and the delay duration corresponding to the delay level 2 is 60 minutes, it is determined that the delay due time corresponding to the delay message is the time corresponding to the sum of the delay message creation timestamp and 60 minutes.
For another example, if the delay level corresponding to the delay message is delay level 3, and the delay duration corresponding to the delay level 3 is 24 hours, it is determined that the delay due time corresponding to the delay message is the time corresponding to the sum of the delay message creation timestamp and the 24 hours.
And if the current time does not reach the delay expiration time corresponding to the delay message, determining that the delay message is not delayed to expire.
For example, if it is determined that the delay message has not been delayed to expire, it is determined whether the delay message is delayed to expire cyclically at preset periodic time intervals. For example, the preset period time is 1 second, and the determination of whether the delay message is delayed to expire is performed cyclically every 1 second.
It is understood that the preset cycle time can be flexibly set according to practical situations, and is not particularly limited herein.
And if the current time reaches the delay expiration time corresponding to the delay message, determining that the delay of the delay message expires. At this time, the Kafka delayed message manager extracts the message body content in the delayed message, and generates a retransmission message corresponding to the target message according to the message body content.
In some embodiments, the obtained message body content is encapsulated according to a preset retransmission message format, so as to generate a retransmission message corresponding to the target message.
Illustratively, the preconfigured retransmission message format is:
{ "_ delay _ message _ parameter _" { "createdate": delayed message creation timestamp, "delayTimeLevel": delay level, "retriedacount": number of times retransmitted }, message body content of source message }
And obtaining information such as the delay level of the target message, the number of times of retransmission of the target message, the delay message creation timestamp, the message body content of the target message and the like through a Kafka delay message manager, and encapsulating according to the configured retransmission message format to generate the retransmission message corresponding to the target message.
It should be noted that the format of the retransmission message may be other than the above-mentioned message format, and the format of the retransmission message is not particularly limited herein.
S104, sending the retransmission message to a Kafka message queue for a consuming end to obtain the retransmission message from the Kafka message queue for consumption.
After generating the retransmission message corresponding to the target message, the retransmission message is sent to the Kafka message queue through the Kafka delay message manager. Illustratively, the retransmission message is sent to the Kafka message queue corresponding to the target message, for example, if the target message is originally cached in the Kafka message queue Topic a, the retransmission message is sent to the Kafka message queue Topic a.
It should be noted that the retransmission message may be sent to another Kafka message queue other than the Kafka message queue corresponding to the target message, for example, if the target message is originally cached in the Kafka message queue Topic a, the retransmission message may be sent to the Kafka message queue Topic B.
The consumer end pulls the retransmission message from the Kafka message queue Topic a or Topic B, and then performs corresponding service processing. If the consumption of the retransmission message is successful, the processing of the target message is finished, and other corresponding business processing can be continuously executed. If the retransmission message consumption fails, the message processing procedure for the target message in the above embodiment is continuously circulated, and thus is not described in detail.
The retransmission message corresponding to the target message is generated, and the retransmission message is consumed, namely the previous target message is consumed again, so that the success rate of target message consumption is improved.
Illustratively, as shown in fig. 4, the Kafka-based message processing flow is as follows:
firstly, a message producer (a producer) sends a message a to a Kafka message queue Topic A or Topic B;
secondly, the message consumer (consumer) pulls the message a from the Topic A or the Topic B and then carries out business processing. If the consumption of the message a fails, executing the step three; if the message a is consumed successfully, executing the step (sixth);
and thirdly, the Kafka delayed message manager packages the message a into a delayed message and sends the delayed message to one of corresponding Kafka delayed message queues Delay _ Topic _1, …. For example, if the number of times the message a has been retransmitted does not exceed 3 times, the delayed message is sent to Delay _ Topic _ 1; if the number of times of the message a retransmission exceeds 3 and does not exceed 6, the delayed message is sent to Delay _ Topic _ 2;
fourthly, the Kafka delay message manager consumes the messages of the Kafka delay message queue and judges whether the delay messages are delayed to expire or not;
if not, circularly judging whether the delay message is delayed to expire or not at the interval of 1 second; if yes, pulling the delay message from the Kafka delay message queue, extracting the content of the message body, and sending the content to the Kafka message queue Topic A or Topic B;
sixthly, the consumption is successful, and the process is finished.
In the above embodiment, when the consumption of the target message fails, a delay message corresponding to the target message is generated, the delay message includes the message body content of the target message, and the delay message is sent to the preconfigured target Kafka delay message queue, when the delay message in the target Kafka delay message queue expires, the message body content in the delay message is extracted, a retransmission message corresponding to the target message is generated, and the retransmission message is sent to the Kafka message queue, so that the consumption end obtains the retransmission message from the Kafka message queue for consumption. The retransmission message corresponding to the target message is generated, and the retransmission message is consumed, namely the target message is consumed again, so that the success rate of target message consumption is improved.
Referring to fig. 5, fig. 5 is a schematic block diagram of a Kafka-based message processing apparatus according to an embodiment of the present application, where the Kafka-based message processing apparatus may be configured in a computer device, and is used to execute the Kafka-based message processing method.
As shown in fig. 5, the Kafka-based message processing apparatus 1000 includes: a message generation module 1001, a message transmission module 1002, a message management module 1003, and a message consumption module 1004.
A message generating module 1001, configured to generate a delay message corresponding to a target message when consumption of the target message fails, where the delay message includes a message body content of the target message;
a message sending module 1002, configured to send the delay message to a preconfigured target Kafka delay message queue;
a message management module 1003, configured to, when the delay message in the target Kafka delay message queue expires, extract the message body content in the delay message, and generate a retransmission message corresponding to the target message according to the message body content;
and the message consumption module 1004 is configured to send the retransmission message to a Kafka message queue, so that the consuming end obtains the retransmission message from the Kafka message queue for consumption.
In one embodiment, the message generating module 1001 is further configured to:
obtaining delay information corresponding to the target message;
and packaging the message body content of the target message and the delay information according to a preset delay message format to generate the delay message.
In one embodiment, the messaging module 1002 is further configured to:
selecting a Kafka delay message queue matched with the delay information from a plurality of pre-configured Kafka delay message queues, and determining the Kafka delay message queue as the target Kafka delay message queue;
and sending the delay message to the target Kafka delay message queue.
In one embodiment, the delay information includes the number of times the target message has been retransmitted, and the message sending module 1002 is further configured to:
acquiring a message retransmitted time range corresponding to each Kafka delay message queue and the retransmitted time of the target message;
and when the number of times of retransmission of the target message is within the range of the number of times of retransmission of the message corresponding to the first Kafka delayed message queue, determining the first Kafka delayed message queue as the target Kafka delayed message queue.
In one embodiment, the delay information includes a delay level of the target message, and the message sending module 1002 is further configured to:
obtaining a message delay grade corresponding to each Kafka delay message queue and a delay grade of the target message;
and when the delay level of the target message is consistent with the message delay level corresponding to the second Kafka delay message queue, determining the second Kafka delay message queue as the target Kafka delay message queue.
In one embodiment, the messaging module 1002 is further configured to:
inquiring configuration information files corresponding to a plurality of Kafka delay message queues; the configuration information file comprises configuration information corresponding to each Kafka delay message queue;
and extracting the message delay grade corresponding to each Kafka delay message queue from the configuration information corresponding to each Kafka delay message queue.
In one embodiment, the delay message includes delay levels and delay message creation timestamps, wherein different delay levels correspond to different delay durations; the message management module 1003 is further configured to:
adding the delay message creation timestamp to the delay duration corresponding to the delay level, and determining the delay expiration time corresponding to the delay message;
if the current time does not reach the delay expiration time, determining that the delay message does not delay expiration;
and if the current time reaches the delay expiration time, determining that the delay message is delayed to expire.
Each module in the Kafka-based message processing apparatus 1000 corresponds to each step in the Kafka-based message processing method embodiment, and the functions and implementation processes thereof are not described in detail here.
The methods, apparatus, and devices of the present application may be deployed in numerous general-purpose or special-purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
For example, the method and apparatus described above may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 6.
Referring to fig. 6, fig. 6 is a schematic block diagram of a computer device according to an embodiment of the present disclosure.
Referring to fig. 6, the computer device includes a processor and a memory connected by a system bus, wherein the memory may include a nonvolatile storage medium and an internal memory.
The processor is used for providing calculation and control capability and supporting the operation of the whole computer equipment.
The internal memory provides an environment for running a computer program in the non-volatile storage medium, which when executed by the processor causes the processor to perform any one of the Kafka-based message processing methods.
It should be understood that the Processor may be a Central Processing Unit (CPU), and the Processor may be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Wherein, in one embodiment, the processor is configured to execute a computer program stored in the memory to implement the steps of:
when the consumption of the target message fails, generating a delay message corresponding to the target message, wherein the delay message comprises the message body content of the target message;
sending the delay message to a pre-configured target Kafka delay message queue;
when the delay message in the target Kafka delay message queue is delayed to expire, extracting the message body content in the delay message, and generating a retransmission message corresponding to the target message according to the message body content;
and sending the retransmission message to a Kafka message queue so that a consumption end can obtain the retransmission message from the Kafka message queue for consumption.
In one embodiment, when the generating of the delayed message corresponding to the target message is implemented, the processor is configured to implement:
obtaining delay information corresponding to the target message;
and packaging the message body content of the target message and the delay information according to a preset delay message format to generate the delay message.
In one embodiment, the processor, in implementing the sending the delay message to the preconfigured target Kafka delay message queue, is configured to implement:
selecting a Kafka delay message queue matched with the delay information from a plurality of pre-configured Kafka delay message queues, and determining the Kafka delay message queue as the target Kafka delay message queue;
and sending the delay message to the target Kafka delay message queue.
In one embodiment, the delay information includes the number of times of retransmission of the target message, and the processor is configured to, when the selection of the Kafka delay message queue matching the delay information from the preconfigured plurality of Kafka delay message queues is performed, determine as the target Kafka delay message queue:
acquiring a message retransmitted time range corresponding to each Kafka delay message queue and the retransmitted time of the target message;
and when the number of times of retransmission of the target message is within the range of the number of times of retransmission of the message corresponding to the first Kafka delayed message queue, determining the first Kafka delayed message queue as the target Kafka delayed message queue.
In one embodiment, the delay information includes a delay level of the target message, and the processor is configured to, when implementing that the Kafka delayed message queue matching the delay information is selected from a plurality of pre-configured Kafka delayed message queues and determined as the target Kafka delayed message queue:
obtaining a message delay grade corresponding to each Kafka delay message queue and a delay grade of the target message;
and when the delay level of the target message is consistent with the message delay level corresponding to the second Kafka delay message queue, determining the second Kafka delay message queue as the target Kafka delay message queue.
In one embodiment, when implementing the obtaining of the message delay level corresponding to each Kafka delay message queue, the processor is configured to implement:
inquiring configuration information files corresponding to a plurality of Kafka delay message queues; the configuration information file comprises configuration information corresponding to each Kafka delay message queue;
and extracting the message delay level corresponding to each Kafka delay message queue from the configuration information corresponding to each Kafka delay message queue.
In one embodiment, the delay message includes delay levels and delay message creation timestamps, wherein different delay levels correspond to different delay durations;
said processor, prior to said extracting said message body content in said delayed message when said delayed message delay in said target Kafka delayed message queue expires, is configured to:
adding the delay message creation timestamp to the delay duration corresponding to the delay level, and determining the delay expiration time corresponding to the delay message;
if the current time does not reach the delay expiration time, determining that the delay message does not delay expiration;
and if the current time reaches the delay expiration time, determining that the delay message is delayed to expire.
The embodiment of the application also provides a computer readable storage medium.
The computer-readable storage medium of the present application has stored thereon a computer program that, when executed by a processor, implements the steps of the Kafka-based message processing method as described above.
The computer-readable storage medium may be an internal storage unit of the Kafka-based message processing apparatus or the computer device in the foregoing embodiments, for example, a hard disk or a memory of the Kafka-based message processing apparatus or the computer device. The computer-readable storage medium may also be an external storage device of the Kafka-based message processing apparatus or the computer device, such as a plug-in hard disk provided on the Kafka-based message processing apparatus or the computer device, a Smart Memory Card (SMC), a Secure Digital Card (Secure Digital Card, SD Card), a Flash memory Card (Flash Card), and the like.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain referred by the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or system in which the element is included.
While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention.
Claims (10)
1. A Kafka-based message processing method is characterized by comprising the following steps:
when the consumption of the target message fails, generating a delay message corresponding to the target message, wherein the delay message comprises the message body content of the target message;
sending the delay message to a pre-configured target Kafka delay message queue;
when the delay message in the target Kafka delay message queue is delayed to expire, extracting the message body content in the delay message, and generating a retransmission message corresponding to the target message according to the message body content;
and sending the retransmission message to a Kafka message queue so that a consumption end can obtain the retransmission message from the Kafka message queue for consumption.
2. The Kafka-based message processing method according to claim 1, wherein the generating the delay message corresponding to the target message includes:
obtaining delay information corresponding to the target message;
and packaging the message body content of the target message and the delay information according to a preset delay message format to generate the delay message.
3. The Kafka-based message processing method of claim 2, wherein sending the delay message to a preconfigured target Kafka delay message queue comprises:
selecting a Kafka delay message queue matched with the delay information from a plurality of pre-configured Kafka delay message queues, and determining the Kafka delay message queue as the target Kafka delay message queue;
and sending the delay message to the target Kafka delay message queue.
4. The Kafka-based message processing method according to claim 3, wherein the delay information includes the number of times the target message has been retransmitted, and wherein selecting a Kafka delay message queue matching the delay information from among a plurality of pre-configured Kafka delay message queues to determine as the target Kafka delay message queue comprises:
acquiring a message retransmitted time range corresponding to each Kafka delay message queue and the retransmitted time of the target message;
and when the number of times of retransmission of the target message is within the range of the number of times of retransmission of the message corresponding to the first Kafka delayed message queue, determining the first Kafka delayed message queue as the target Kafka delayed message queue.
5. The Kafka-based message processing method according to claim 3, wherein the delay information includes a delay level of the target message, and wherein selecting a Kafka delay message queue matching the delay information from among a plurality of pre-configured Kafka delay message queues to determine as the target Kafka delay message queue comprises:
obtaining a message delay grade corresponding to each Kafka delay message queue and a delay grade of the target message;
and when the delay level of the target message is consistent with the message delay level corresponding to the second Kafka delay message queue, determining the second Kafka delay message queue as the target Kafka delay message queue.
6. The Kafka-based message processing method of claim 5, wherein obtaining the message delay level corresponding to each Kafka-delayed message queue comprises:
inquiring configuration information files corresponding to a plurality of Kafka delay message queues; the configuration information file comprises configuration information corresponding to each Kafka delay message queue;
and extracting the message delay grade corresponding to each Kafka delay message queue from the configuration information corresponding to each Kafka delay message queue.
7. The Kafka-based message processing method according to any one of claims 1 to 6, wherein the delay message includes delay levels and delay message creation time stamps, and wherein different delay levels correspond to different delay durations;
when the delay message in the target Kafka delay message queue expires, before extracting the message body content in the delay message, the method includes:
adding the delay message creation timestamp to the delay duration corresponding to the delay level, and determining the delay expiration time corresponding to the delay message;
if the current time does not reach the delay expiration time, determining that the delay message does not delay expiration;
and if the current time reaches the delay expiration time, determining that the delay message is delayed to expire.
8. A Kafka-based message processing apparatus, comprising:
the message generating module is used for generating a delay message corresponding to a target message when the consumption of the target message fails, wherein the delay message comprises the message body content of the target message;
the message sending module is used for sending the delay message to a pre-configured target Kafka delay message queue;
the message management module is used for extracting the message body content in the delayed message when the delay message in the target Kafka delayed message queue is delayed to expire, and generating a retransmission message corresponding to the target message according to the message body content;
and the message consumption module is used for sending the retransmission message to a Kafka message queue so that a consumption end can obtain the retransmission message from the Kafka message queue for consumption.
9. A computer device, wherein the computer device comprises a memory and a processor;
the memory for storing a computer program;
the processor, configured to execute the computer program and to implement the Kafka-based message processing method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the steps of the Kafka-based message processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210159014.2A CN114546681A (en) | 2022-02-21 | 2022-02-21 | Kafka-based message processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210159014.2A CN114546681A (en) | 2022-02-21 | 2022-02-21 | Kafka-based message processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114546681A true CN114546681A (en) | 2022-05-27 |
Family
ID=81677210
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210159014.2A Pending CN114546681A (en) | 2022-02-21 | 2022-02-21 | Kafka-based message processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114546681A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115098889A (en) * | 2022-08-24 | 2022-09-23 | 广州市千钧网络科技有限公司 | Authority management method, device, equipment and storage medium |
| CN115550280A (en) * | 2022-11-24 | 2022-12-30 | 云账户技术(天津)有限公司 | Multi-level message queue implementation method, system, electronic device and readable storage medium |
| CN116094651A (en) * | 2022-10-31 | 2023-05-09 | 中国电信股份有限公司 | Message retry method, system, electronic device and storage medium |
| CN116155849A (en) * | 2022-12-28 | 2023-05-23 | 中国联合网络通信集团有限公司 | Message delivery method and device, electronic equipment and storage medium |
-
2022
- 2022-02-21 CN CN202210159014.2A patent/CN114546681A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115098889A (en) * | 2022-08-24 | 2022-09-23 | 广州市千钧网络科技有限公司 | Authority management method, device, equipment and storage medium |
| CN116094651A (en) * | 2022-10-31 | 2023-05-09 | 中国电信股份有限公司 | Message retry method, system, electronic device and storage medium |
| CN115550280A (en) * | 2022-11-24 | 2022-12-30 | 云账户技术(天津)有限公司 | Multi-level message queue implementation method, system, electronic device and readable storage medium |
| CN115550280B (en) * | 2022-11-24 | 2023-03-31 | 云账户技术(天津)有限公司 | Multi-level message queue implementation method, system, electronic device and readable storage medium |
| CN116155849A (en) * | 2022-12-28 | 2023-05-23 | 中国联合网络通信集团有限公司 | Message delivery method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114546681A (en) | Kafka-based message processing method, device, equipment and storage medium | |
| CN109344172B (en) | High-concurrency data processing method and device and client server | |
| US7702909B2 (en) | Method and system for validating timestamps | |
| CN115048658B (en) | System, method and computer readable medium for protecting distributed data using a latent name slicing technique | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| CN112486915B (en) | Data storage method and device | |
| WO2023071133A1 (en) | Tagged network information service generation and application method and apparatus, device and medium | |
| CN114041134A (en) | System and method for block chain based secure storage | |
| WO2019223979A1 (en) | Cryptographic key generation and storage | |
| CN110659124A (en) | Message processing method and device | |
| CN108376154B (en) | System base table synchronization method and device, computer equipment and storage medium | |
| CN112015815B (en) | Data synchronization method, device and computer readable storage medium | |
| CN112865927B (en) | Message delivery verification method, device, computer equipment and storage medium | |
| US11411726B2 (en) | Cryptographic key generation using multiple random sources | |
| CN114401239B (en) | Metadata transmission method, apparatus, computer device and storage medium | |
| CN112181307A (en) | Block chain based distributed data redundancy storage method and electronic equipment | |
| US11586644B2 (en) | System and methods for creating, distributing, analyzing and optimizing data-driven signals | |
| CN115169283A (en) | Planning plan distributed rapid supervision method, system, equipment and storage medium | |
| CN114048059A (en) | Method and device for adjusting timeout time of interface, computer equipment and storage medium | |
| CN114491453A (en) | Task authority management method, device, equipment and storage medium | |
| US8589605B2 (en) | Inbound message rate limit based on maximum queue times | |
| CN114615325A (en) | Message pushing method and device, computer equipment and storage medium | |
| CN115208872A (en) | Edge cloud mirror image data processing method and device, computer equipment and storage medium | |
| CN113760482A (en) | Task processing method, device and system | |
| CN113138862A (en) | Message processing method, server, device, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |