CN116560874A

CN116560874A - Kafka message consumption processing method, device, equipment and storage medium

Info

Publication number: CN116560874A
Application number: CN202310555150.8A
Authority: CN
Inventors: 汤波涛
Original assignee: Shanghai Weimeng Enterprise Development Co ltd
Current assignee: Shanghai Weimeng Enterprise Development Co ltd
Priority date: 2023-05-17
Filing date: 2023-05-17
Publication date: 2023-08-08

Abstract

The application discloses a Kafka message consumption processing method, a Kafka message consumption processing device, kafka message consumption processing equipment and a Kafka message storage medium, relates to the technical field of big data real-time processing, and comprises the following steps: matching one or more target MQ executors according to the category information of the Kafka message; judging whether the target MQ executor can process the Kafka message; if yes, calling a target interface to execute processing operation related to the Kafka message; determining whether the current call is successful or not, if so, saving the generated failure record related to the current call to a failure record table; scanning the failure records in the failure record table and determining whether a retry operation needs to be performed on the failure records; if so, re-executing the step of interface calling according to the identified failed retry strategy; and counting the failure records of the retry failure, and carrying out corresponding warning based on the counting result. The technical scheme provides a multi-consumption route function under the same consumption group for realizing a high-reliability consumption scheme, so that the consumption concurrency performance is improved.

Description

Kafka message consumption processing method, device, equipment and storage medium

Technical Field

The invention relates to the technical field of big data real-time processing, in particular to a Kafka message consumption processing method, a Kafka message consumption processing device, kafka message consumption processing equipment and a storage medium.

Background

Currently, kafka is one of the more popular message queuing schemes, is used in the industry more and is used for some log data transmission channels, and the service has no very high requirement on data reliability, but when the Kafka is used as an asynchronous processing scheme of service functions, the defects of Kafka are obvious, and the specific defects are as follows: firstly, kafka consumes data and then controls the consumption position by submitting a displacement point, and the consumption position has no retry function, so that the defects in a service use scene are more, and in practice, the service often needs to retry periodically or a more complex retry strategy after failure; secondly, each consuming service end of Kafka needs to newly start a packet, and the same Topic on the service often has a large number of downstream needs to be selectively consumed, such as a single message, and the effective message which is created by the same proportion of packets but is actually consumed is a selected part of the effective message, so that resource waste is caused; third, the number of threads consumed at the same Topic of Kafka is equivalent to the number of partitions, which limits the consumption capacity of the consumer, and increasing the number of partitions increases the resource overhead.

In view of this, it has been a great need for a person skilled in the art to provide a solution to the above-mentioned technical problems.

Disclosure of Invention

In view of the above, the present invention aims to provide a Kafka message consumption processing method, apparatus, device and storage medium, which can implement a highly reliable consumption scheme, and provide a multi-consumption routing function under the same consumption group, so as to improve consumption concurrency performance. The specific scheme is as follows:

in a first aspect, the present application discloses a Kafka message consumption processing method, including:

matching one or more target MQ executors according to the category information of the Kafka message;

judging whether the target MQ executor can process the Kafka message, and if the target MQ executor can process the Kafka message, calling a corresponding target interface to execute processing operation related to the Kafka message through the target interface;

determining whether the current call is successful or not, and if the current call fails, saving the generated failure record related to the current call to a failure record table;

scanning the failure records in the failure record table, and determining whether a corresponding retry operation is required to be performed on the failure records;

if the corresponding retry operation is required to be executed on the failure record, re-executing the step of calling the corresponding target interface according to the identified failure retry strategy corresponding to the failure record so as to execute the processing operation related to the Kafka message through the target interface;

and counting the failure records of the retry failure, and carrying out corresponding warning based on the counting result.

Optionally, before the matching of the one or more target MQ executors according to the category information of the Kafka message, the method further includes:

monitoring the configuration change of the MQ executor in the database to obtain corresponding change information;

and executing corresponding changing operation based on the changing information.

Optionally, the performing a corresponding change operation based on the change information includes:

if the change information indicates that a new MQ executor is added in the database, registering corresponding Kafka consumers to a Kafka cluster;

and if the change information indicates the change of the configuration of the MQ executor in the database, updating the configuration of the MQ executor in the database according to the change information.

Optionally, the determining whether the current call is successful includes:

and determining whether the current call is successful or not according to the call information returned by the target interface and the successful expression of the target MQ executor.

Optionally, the determining whether the target MQ executor is capable of processing the Kafka message includes:

and executing a trigger expression of the target MQ executor according to the specific content of the Kafka message to judge whether the target MQ executor can process the Kafka message.

Optionally, the calling a corresponding target interface to perform a processing operation related to the Kafka message through the target interface includes:

and calling a target interface according to the constructed request header and the request parameter body to execute processing operation related to the Kafka message through the target interface.

Optionally, after determining whether the corresponding retry operation needs to be performed on the failed record, the method further includes:

and if the corresponding retry operation is not required to be executed on the failure record or the retry on the failure record is successful, updating the state of the failure record to an expired state.

In a second aspect, the present application discloses a Kafka message consumption processing apparatus, including:

the executor matching module is used for matching one or more target MQ executors according to the category information of the Kafka message;

an executor judging module, configured to judge whether the target MQ executor is capable of processing the Kafka message;

an interface calling module, configured to call a corresponding target interface to perform a processing operation related to the Kafka message through the target interface if the target MQ executor is capable of processing the Kafka message;

the call judging module is used for determining whether the current call is successful or not, and if the current call fails, the generated failure record related to the current call is stored in the failure record table;

the retry judgment module is used for scanning the failure records in the failure record table and determining whether corresponding retry operation is required to be executed on the failure records;

a retry module, configured to re-execute, if the retry operation is required to be executed on the failure record, the step of calling the corresponding target interface according to the identified failure retry policy corresponding to the failure record to execute the processing operation related to the Kafka message through the target interface;

and the alarm module is used for counting the failure records of the retry failure and carrying out corresponding alarm based on the counting result.

In a third aspect, the present application discloses an electronic device comprising:

the memory is used for storing a big data real-time processing program;

and the processor is used for executing the big data real-time processing program to realize the steps of the Kafka message consumption processing method disclosed in the prior art.

In a fourth aspect, the present application discloses a big data real-time processing readable storage medium for storing a big data real-time processing program; wherein, the big data real-time processing program realizes the steps of the Kafka message consumption processing method disclosed above when being executed by a processor.

It can be seen that the present application provides a Kafka message consumption processing method, including: matching one or more target MQ executors according to the category information of the Kafka message; judging whether the target MQ executor can process the Kafka message, and if the target MQ executor can process the Kafka message, calling a corresponding target interface to execute processing operation related to the Kafka message through the target interface; determining whether the current call is successful or not, and if the current call fails, saving the generated failure record related to the current call to a failure record table; scanning the failure records in the failure record table, and determining whether a corresponding retry operation is required to be performed on the failure records; and if the retry operation is required to be executed on the failure record, re-executing the step of calling the corresponding target interface according to the identified failure retry strategy corresponding to the failure record so as to execute the processing operation related to the Kafka message through the target interface. And counting the failure records of the retry failure, and carrying out corresponding warning based on the counting result. Therefore, in the process of consumption processing of each Kafka message, one or more target MQ executors are matched according to the category information of the Kafka message, namely, a plurality of executors are configured to bind the same consumption group to realize concurrent consumption, when the target MQ executors can process the Kafka message, corresponding target interfaces are called to execute processing operations related to the Kafka message through the target interfaces, namely, various corresponding post-operations are carried out through the matched target MQ executors, so that one message can trigger a plurality of post-operations, then whether the calling is successful or not is determined, if the calling is failed, the generated failure record is subjected to strategic retry, and finally, the statistics result of the failure record is subjected to alarm based on the failure record, so that the reliability of consumption processing is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of a Kafka message consumption processing method disclosed in the present application;

FIG. 2 is a schematic diagram of a particular MQ actuator data model disclosed herein;

FIG. 3 is a schematic diagram of a message failure record data model according to one embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a specific Kafka message consumption processing method disclosed in the present application;

FIG. 5 is a schematic illustration of automatic recharge after payment of a telephone bill recharge order in accordance with the disclosure of the present application;

FIG. 6 is a schematic structural diagram of a Kafka message consumption processing apparatus disclosed in the present application;

fig. 7 is a block diagram of an electronic device disclosed in the present application.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Currently, when the currently popular Kafka message queue scheme is used as an asynchronous processing scheme of a service function, and under a service use scene, the Kafka has the problems of failed retry, resource waste, resource overhead increase and the like. Therefore, the application provides a Kafka message consumption processing scheme which can solve the problem of failed retry of the Kafka in a service use scene and provide a multi-consumption routing function under the same consumption group, so that the consumption performance can be improved.

The embodiment of the invention discloses a Kafka message consumption processing method, which is shown in fig. 1 and comprises the following steps:

step S11: one or more target MQ executors are matched according to the category information of the Kafka message.

It should be noted that, when the Kafka message is received, the message is put into a local consumption thread pool which is additionally arranged in advance, that is, the concurrent processing of the message is realized by a thread pool mode, and the internal consumption thread number can be configured. It should be noted that Kafka represents an implementation of a message queue.

In this embodiment, when the Kafka message is placed in the local consumption thread pool that is additionally provided in advance, each thread in the thread pool may process the corresponding message at the same time, that is, in the process of concurrent processing of each Kafka message in the local consumption thread pool, one or more target MQ executors are matched according to the category information of the Kafka message, specifically, one or more target MQ executors are matched according to the category information including Topic and/or GroupID in the Kafka message, that is, one message may be matched with one or more MQ executors.

In this embodiment, before the matching of the one or more target MQ executors according to the category information of the Kafka message, the method may further include: monitoring the configuration change of the MQ executor in the database to obtain corresponding change information; and executing corresponding changing operation based on the changing information. It will be appreciated that the developer periodically maintains the MQ executor before beginning concurrent processing of each of the Kafka messages, so MQ executor-related configuration may change. Therefore, the configuration change of the MQ executor in the database is monitored through the local task, and then corresponding operation is executed according to the monitored change information, namely if the change information indicates that a new MQ executor is added in the database, the corresponding Kafka consumer is registered to the Kafka cluster, so that the dynamic registration of the Kafka consumer is realized. If the change information indicates that the configuration of the MQ executor in the database is changed, the configuration of the MQ executor in the database is updated according to the change information, that is, if the change information indicates that a certain configuration in the MQ executor is changed, corresponding configuration information in a corresponding change cache comprises basic information, MQ access information, target call information, failure retry strategy and the like of the MQ executor, as shown in fig. 2, wherein the basic information part can comprise an executor ID, a name, a responsible person, a state and the like; the MQ access information part may include Topic, groupID, priority, etc.; the target call information part may include a call protocol type, a call interface address, a trigger expression, a judgment success expression, a service ID extraction expression, a request header construction configuration, and a parameter body construction configuration; the failed retry strategy comprises a maximum retry number, a retry delay strategy, a monitoring alarm threshold, a retry interval stepping strategy, and the like. It should be noted that, the MQ (Message Queue) represents a Message Queue, which is a "first-in first-out" basic data structure, and is generally used for solving the problems of application decoupling, asynchronous Message, traffic peak clipping, and the like, so as to implement a high-performance, high-availability, scalable and final consistency architecture. Topic represents the Topic of the message, and is used in Kafka to divide the category of the message, and generally one Topic is used for the same type of message. GroupID indicates a consumption group identification number of Topic, which may have multiple consumption groups, for consumption of the same message only once within a consumption group.

Step S12: judging whether the target MQ executor can process the Kafka message, and if the target MQ executor can process the Kafka message, calling a corresponding target interface to execute processing operation related to the Kafka message through the target interface.

In this embodiment, after one or more target MQ executors are matched, it is determined whether each target executor is capable of processing the Kafka message. Specifically, the trigger expression of the target MQ executor is executed according to the specific content of the Kafka message to determine whether the target MQ executor can process the Kafka message, and if the target MQ executor can process the Kafka message, a corresponding target interface is called to execute processing operation related to the Kafka message through the target interface. Specifically, according to the constructed request header and the request parameter body, a target interface is called to execute processing operation related to the Kafka message through the target interface.

Step S13: and determining whether the current call is successful, and if the current call fails, saving the generated failure record related to the current call to a failure record table.

In this embodiment, when the current target MQ executor can execute the message, a corresponding target interface is called to execute a processing operation related to the Kafka message, then whether the current call is successful is determined, specifically, whether the current call is successful is determined according to call information returned by the target interface and a success expression of the target MQ executor, if the current call fails, the generated failure record is saved in a failure record table in a MySQL database. The message failure record data model is shown in fig. 3, and the consumption failure record data model comprises a basic information, MQ message information, an executor information, execution result information, consumption retry information and the like, wherein the basic information comprises a detail ID, a creation time, an update time and the like, the MQ message information comprises Topic, groupID, a message body, an extracted service ID and the like, the executor information comprises an executor ID, an executor name, a responsible person, an executor snapshot information and the like, the execution result information comprises an execution result, a failure type, failure detail information, failure time and the like, the failure type is an abnormal call or a return failure, the consumption proving information comprises a retry state, a retry success time, an expiration reason, and the like, wherein the retry state comprises a to-be-retried state, a retry success, a retry failure, an expiration reason, and the expiration reason comprises an expiration reason that the executor has been deleted by hand.

Step S14: scanning the failure records in the failure record table and determining whether a corresponding retry operation needs to be performed on the failure records.

In this embodiment, the failure records in the failure record table are scanned periodically using the ElasticJob distributed timing task, and then it is determined whether the scanned failure records require retry. The elastic Job represents a distributed timing task middleware, and has the function of concurrent execution of fragments.

Step S15: and if the retry operation is required to be executed on the failure record, re-executing the step of calling the corresponding target interface according to the identified failure retry strategy corresponding to the failure record so as to execute the processing operation related to the Kafka message through the target interface.

In this embodiment, for a failed record that needs to be retried, the step of calling the corresponding target interface to execute the processing operation related to the Kafka message through the target interface is re-executed according to the identified failed retry policy corresponding to the failed record, when the retry is successful, it indicates that the current failed record has been terminated, and the terminated failed record may be periodically archived to ensure the scanning performance of the failed record table. And if the corresponding retry operation is not required to be executed on the failure record or the retry on the failure record is successful, updating the state of the failure record into an expired state, and if the failure record which is not required to be retried is updated into the expired state.

Step S16: and counting the failure records of the retry failure, and carrying out corresponding warning based on the counting result.

When the retry fails, the current failure record is not terminated, the failure record of the retry failure is recorded, the failure record of the retry failure is counted periodically, and the counting result is sent to the user terminal for alarming. It will be appreciated that the elastic job distributed timing task is used to periodically count the non-terminated failure records, such as counting the non-terminated failure records twice a day, and then sending the statistics result to the relevant responsible person through the touch-up tool to inform the responsible person that the failure records are retried for a plurality of times, and the touch-up tool needs to be retried manually, where the touch-up tool may include, but is not limited to, an enterprise micro robot, a nailing robot, a mail, a sms, etc.

In the process of consumption processing of each Kafka message, one or more target MQ executors are matched according to the category information of the Kafka message, namely, a plurality of executors are configured to bind the same consumption group to realize concurrent consumption, when the target MQ executors can process the Kafka message, corresponding target interfaces are called to execute processing operation related to the Kafka message through the target interfaces, namely, various corresponding post-operation is carried out through the matched target MQ executors, so that one message can trigger a plurality of post-operation, then whether the calling is successful or not is determined, if the calling is failed, the generated failure record is subjected to strategic retry, and finally, the statistics result of the failed retry is reported based on the failure record, so that the reliability of consumption processing is improved.

For example, referring to fig. 4, during maintenance of the MQ executor, a developer logs in the system background, configures the MQ executor, configures basic information, MQ access information, target call information, failure retry strategies and the like by querying, adding the MQ executor, modifying information and other operations, during processing of the Kafka message, a local timing task periodically scans the MQ executor configuration, dynamically registers corresponding Kafka consumers to the Kafka cluster based on configuration change information, and then receives the Kafka messages in batches by monitoring consumption, and puts the messages into a local consumption thread pool to process each message concurrently. Therefore, the method and the device can be used for configuring the number of internal consumption threads, so that the limit that only one consumer thread can exist in one partition set by Kafka is broken through, and a plurality of executors can be configured to bind the same consumption group to realize concurrent consumption, so that the limit that one consumption group set by Kafka corresponds to one business consumer is broken through, and the concurrent consumption performance is improved.

Referring to fig. 4, in the processing process of a single message, one or more target MQ executors are matched according to Topic, groupID information carried in the message, then a trigger expression of the target MQ executors is executed one by one according to specific content of the message, whether the current target MQ executors process the message is determined, if the current target MQ executors need to process the message, a request header and a parameter body are constructed, then a target interface is called according to a call protocol according to the request header and the parameter body, further whether the current call is successful is determined according to a success expression and call information returned by the target interface, if the current call is successful, the call is ended, and if the current call is failed, the generated failure record is saved or updated into a failure record table. And in the failure retry process, periodically scanning a failure record table through a distributed timing task, identifying a retry strategy, then determining whether the retry is required to be continued, and if the retry is required to be continued, re-executing the step of calling the target interface according to the identified retry strategy to execute the processing operation related to the Kafka message through the target interface. If the retry is not required, the update failure record state is an expired state. In the process of monitoring the alarm, statistics is carried out on failure records of retry failures at regular intervals through a distributed timing task, then information of a relevant executor responsible person is obtained, and then an alarm triggering system is called to remind the relevant responsible person of the failure records of multiple retry failures, for example, the relevant responsible person receives corresponding alarm information through an enterprise micro-system, namely, the responsible person is reminded of having an MQ message consume to retry failures for multiple times to inform the responsible person of timely processing, finally, intervention is carried out through a developer, and then the retry is successful.

In a specific embodiment, in an order, after-sales and performance system of an e-commerce center, each state change in the life cycle of the order of the e-commerce center needs to trigger various corresponding asynchronous operations, such as automatic delivery after virtual commodity payment, automatic refund after sale, ES heterogeneous data synchronization, and the like. That is, various complex corresponding post operations are performed by the matched MQ executors after each status message of the order is sent, and one message may trigger multiple post operations, each MQ executor also has a custom retry strategy, etc., where the order status message includes creation order message, order payment message, shipping message, etc. Therefore, when the e-commerce service center needs to consume a certain type of information, the operations of on-line configuration of the MQ executor, setting of Topic information, triggering of an expression, target interface calling information and the like are only needed, the configuration access can be realized through dynamic identification configuration and real-time effectiveness of the e-commerce service center, the service center only needs to provide RPC (Remote Procedure Call Protocol ) interfaces, such as protocols of http, dubbo and the like, and then the calling can be triggered through simple configuration, and any SDK (Software Development Kit ) of the service center is not needed to be injected in the technical scheme of the e-commerce service center, so that the asynchronous consumption design of the e-commerce service center can be simplified, and the functions of no code invasion, high-reliability consumption, multi-executor routing, dynamic start-stop and modification and the like are realized. And, the high-reliability consumption functions comprise calling failure recognition, failure information storage, rich retry strategies, monitoring alarm mechanisms and the like.

In another specific embodiment of automatic recharging after payment of the order for recharging telephone charges, the system needs to perform automatic recharging operation after payment of the order for recharging telephone charges, as shown in fig. 5, the service end designs and develops a service interface for realizing automatic recharging after payment of the order for recharging telephone charges, that is, develops a dubbo interface for automatic recharging, and takes in an order status message, wherein the order status message comprises an order ID, an order type, a status and the like, and invokes an operator interface for recharging telephone charges after performing necessary logic and anti-rechecking on the interface. The developer logs in the system to configure the MQ executor, for example, basic information is configured, for example, the name is that the telephone charge is automatically recharged after the order payment is carried out, and a responsible person is determined; configuring MQ access information, e.g., topic=order status change message (order_status_changed), groupid=order payment post-processing group (order_charge_process); configuring target call, for example, configuring an interface path and a method name of automatic recharging, calling a protocol to select dubbo, and triggering an expression to set semantics as follows: order type = virtual order and merchandise type = telephone charge and pre-change status = unpaid and post-change status = paid, the judgment success expression semantics are: return status = success, request argument set to: transmitting the message body; configuration retry strategy: the maximum retry number is set to 10, the timeout time is set to 5 seconds, and the retry delay ladder is: 3s,30s,2m,1h, i.e. 3 seconds apart for the first retry, 30 seconds apart for the second retry, and so on, 1 hour apart after more than 4. Referring to fig. 5, when a user selects 50-element face value charge order to purchase, place an order and pay goods in a small program mall, the order system receives a payment callback and then changes the order state and sends an order state change message.

Correspondingly, the embodiment of the application also discloses a Kafka message consumption processing device, which is shown in fig. 6, and comprises:

an actuator matching module 11, configured to match one or more target MQ actuators according to the category information of the Kafka message;

an actuator judgment module 12 for judging whether the target MQ actuator is capable of processing the Kafka message;

an interface calling module 13, configured to call a corresponding target interface to perform a processing operation related to the Kafka message through the target interface if the target MQ executor is capable of processing the Kafka message;

the call judging module 14 is configured to determine whether the current call is successful, and if the current call fails, save the generated failure record related to the current call to the failure record table;

a retry judgment module 15, configured to scan the failure records in the failure record table, and determine whether a corresponding retry operation needs to be performed on the failure records;

and a retry module 16, configured to re-execute, if the retry operation is required to be executed on the failure record, the step of calling the corresponding target interface according to the identified failure retry policy corresponding to the failure record to execute the processing operation related to the Kafka message through the target interface.

And the alarm module 17 is used for counting the failure records of the retry failure and carrying out corresponding alarm based on the counting result.

From the above, in the embodiment of the present application, in the process of consuming each Kafka message, one or more target MQ executors are matched according to the category information of the Kafka message, that is, multiple executors are configured to bind the same consuming group to implement concurrent consumption, when the target MQ executors can process the Kafka message, a corresponding target interface is called to execute processing operations related to the Kafka message through the target interface, that is, various corresponding post operations are performed through the matched target MQ executors, so that one message can trigger multiple post operations, then it is determined whether the call is successful, if the call is failed, a strategic retry is performed on the generated failure record, and finally, an alarm is performed based on the statistics result of the failed retry of the failure record, thereby improving the reliability of the consumption processing, that is, the above technical scheme of the present application provides a multi-consumption routing function under the same consuming group for implementing a highly reliable consumption scheme, thereby improving the concurrent performance of consumption.

In some specific embodiments, the Kafka message consumption processing apparatus may specifically further include:

the configuration monitoring module is used for monitoring the configuration change of the MQ executor in the database to obtain corresponding change information;

and the operation execution module is used for executing corresponding change operation based on the change information.

In some specific embodiments, the operation execution module may specifically include:

a consumer registration module, configured to register a corresponding Kafka consumer to a Kafka cluster if the change information indicates that a new MQ executor is added to the database;

and the configuration updating module is used for updating the configuration of the MQ executor in the database according to the change information if the change information indicates the change of the configuration of the MQ executor in the database.

In some specific embodiments, the call determination module 14 may specifically include:

and the call judging unit is used for determining whether the current call is successful or not according to the call information returned by the target interface and the successful expression of the target MQ executor.

In some specific embodiments, the actuator determination module 12 may specifically include:

and the judging unit is used for executing the trigger expression of the target MQ executor according to the specific content of the Kafka message so as to judge whether the target MQ executor can process the Kafka message.

In some specific embodiments, the interface calling module 13 may specifically include:

and the interface calling unit is used for calling a target interface according to the constructed request header and the request parameter body so as to execute processing operation related to the Kafka message through the target interface.

and the state updating module is used for updating the state of the failure record into an expired state if the corresponding retry operation is not required to be executed on the failure record or the retry on the failure record is successful.

Further, the embodiment of the application also provides electronic equipment. Fig. 7 is a block diagram of an electronic device 20, according to an exemplary embodiment, and the contents of the diagram should not be construed as limiting the scope of use of the present application in any way.

Fig. 7 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a big data real-time processing program, where the big data real-time processing program is loaded and executed by the processor 21, so as to implement relevant steps in the Kafka message consumption processing method disclosed in any one of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically process electronic big data in real time.

In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.

The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon may include an operating system 221, a big data real-time processing program 222, and the like, and the storage may be temporary storage or permanent storage.

The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and the big data real-time processing program 222, which may be Windows Server, netware, unix, linux, etc. The big data real-time processing program 222 may further include a big data real-time processing program that can be used to complete other specific works, in addition to the big data real-time processing program that can be used to complete the Kafka message consumption processing method performed by the electronic device 20 disclosed in any of the foregoing embodiments.

Further, the embodiment of the application also discloses a big data real-time processing readable storage medium, wherein the big data real-time processing readable storage medium stores a big data real-time processing program, and when the big data real-time processing program is loaded and executed by a processor, the steps of the Kafka message consumption processing method disclosed in any embodiment are realized.

In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.

Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above describes in detail a Kafka message consumption processing method, device, apparatus and storage medium provided by the present invention, and specific examples are applied to illustrate the principles and embodiments of the present invention, where the above examples are only used to help understand the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims

1. A Kafka message consumption processing method, comprising:

2. The Kafka message consumption processing method according to claim 1, further comprising, before said matching one or more target MQ executors according to the category information of the Kafka message:

3. The Kafka message consumption processing method according to claim 2, wherein said performing a corresponding change operation based on said change information comprises:

4. The Kafka message consumption processing method according to claim 1, wherein said determining whether the current call is successful comprises:

5. The Kafka message consumption processing method according to claim 1, wherein said determining whether said target MQ executor is capable of processing said Kafka message comprises:

6. The Kafka message consuming processing method of claim 1, wherein said invoking a corresponding target interface to perform processing operations associated with said Kafka message via said target interface comprises:

7. The Kafka message consumption processing method according to any one of claims 1 to 6, wherein after said determining whether or not a corresponding retry operation needs to be performed on said failure record, further comprising:

8. A Kafka message consumption processing apparatus, comprising:

9. An electronic device, comprising:

a memory for storing a computer program;

a processor for executing the computer program to implement the steps of the Kafka message consumption processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program; wherein the computer program when executed by a processor implements the steps of the Kafka message consumption processing method according to any one of claims 1 to 7.