CN115202898A - Message consumption method and device, computer equipment and storage medium - Google Patents
Message consumption method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN115202898A CN115202898A CN202110396874.3A CN202110396874A CN115202898A CN 115202898 A CN115202898 A CN 115202898A CN 202110396874 A CN202110396874 A CN 202110396874A CN 115202898 A CN115202898 A CN 115202898A
- Authority
- CN
- China
- Prior art keywords
- offset
- message
- consumed
- target
- coordination service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 230000002159 abnormal effect Effects 0.000 claims abstract description 39
- 238000005192 partition Methods 0.000 claims description 35
- 230000005856 abnormality Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 10
- 230000009471 action Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/465—Distributed object oriented systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The application relates to a message consumption method, a message consumption device, a computer device and a storage medium. The method comprises the following steps: after the message processing platform is recovered from the abnormal state to be normal, acquiring a first offset which is currently stored in the distributed coordination service cluster; acquiring a second offset corresponding to a target message queue in the distributed transaction cluster; acquiring and consuming messages to be consumed corresponding to each offset between a first offset and a second offset in a target message queue; and sending the second offset to the distributed coordination service cluster, and updating the currently stored offset into the second offset by the distributed coordination service cluster. After the message processing platform is recovered from the abnormal state, the consumption progress is obtained from the distributed coordination service cluster, so that the situation that resources are wasted to consume the consumed messages again or data loss is caused by missing the messages to be consumed which need to be consumed due to inaccurate positioning of the messages to be consumed is avoided.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a message consumption method, an apparatus, a computer device, and a storage medium.
Background
With the increasing real-time requirement of each large company on data, real-time storage, consumption and real-time calculation are very common in each large company, so that kafka is widely applied. Kafka is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site. These data are typically addressed by handling logs and log aggregations due to throughput requirements.
However, in the prior art, when consumption abnormality occurs in the process of consuming the kafka data through the sparkSteaming application program, the execution condition of the consumption task cannot be accurately known, and the data to be consumed cannot be accurately positioned after the abnormality is resolved, so that the data loss condition is caused.
Disclosure of Invention
In order to solve the technical problem, the application provides a message consumption method, a message consumption device, a computer device and a storage medium.
In a first aspect, the present application provides a message consumption method, including:
after the message processing platform is recovered from the abnormal state to be normal, acquiring a first offset currently stored in the distributed coordination service cluster, wherein the first offset is used for representing the consumption progress of the message processing platform to the message in the target message queue before the message processing platform is abnormal;
acquiring a second offset corresponding to the target message queue in the distributed transaction cluster, wherein the second offset is used for representing the last message to be consumed in the target message queue;
acquiring and consuming messages to be consumed corresponding to each offset between the first offset and the second offset in the target message queue;
and after the consumption of all messages to be consumed corresponding to the offsets between the first offset and the second offset is finished, sending the second offset to the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the currently stored offset to the second offset when receiving the second offset.
Optionally, after the message processing platform returns to normal from the abnormal state, the method further includes:
and sending a preset offset to the distributed coordination service cluster under the condition that the acquisition fails when the first offset stored currently is acquired from the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the offset stored currently to the preset offset under the condition that the preset offset is received.
Optionally, in a case that a message to be consumed corresponding to any offset between the first offset and the second offset is consumed abnormally, the method further includes:
detecting whether the state of the message processing platform is recovered to normal, wherein the second offset to be sent to the distributed coordination service cluster is cancelled when the message processing platform is abnormal;
and after the message processing platform is recovered from the abnormal state to be normal, executing the step of acquiring the first offset currently stored in the distributed coordination service cluster.
Optionally, the consuming an abnormal message to be consumed corresponding to any offset between the first offset and the second offset includes:
and in the process that the message to be consumed corresponding to any offset between the first offset and the second offset is consumed, determining that the message to be consumed is consumed abnormally under the condition that at least one of null pointer abnormality, message to be consumed analysis abnormality, message to be consumed reading abnormality and consumption task corresponding to the message to be consumed is forcibly ended.
Optionally, the distributed coordination service cluster includes a plurality of first message tags, where the first message tags correspond to a plurality of first partitions, each of the first partitions corresponds to one of the first offsets, and the obtaining of the first offset currently stored in the distributed coordination service cluster includes obtaining the first offset corresponding to each of the first message tags in the following manner:
acquiring a plurality of first target partitions corresponding to first target tags stored in the distributed coordination service cluster, wherein the first target tags are currently processed message tags in the plurality of first message tags;
and acquiring a first offset corresponding to each first target partition in the distributed coordination service cluster, wherein the first offset corresponding to each first target partition is used for representing the consumption progress of the message in a target message queue.
Optionally, the distributed transaction cluster includes a second message tag corresponding to the first message tag, where the second message tag corresponds to multiple message storage queues, each message storage queue corresponds to one first partition, each message storage queue includes multiple messages to be consumed, each message to be consumed carries a corresponding offset, and the obtaining of the second offset in the distributed transaction cluster includes obtaining each second offset corresponding to the first message tag in the following manner:
acquiring a plurality of target message queues corresponding to a second target tag in the distributed transaction cluster, wherein the second target tag is a second message tag corresponding to the first target tag in the distributed transaction cluster, and the target message queue is a message storage queue corresponding to the second target tag;
and selecting a second offset corresponding to each second target message queue from a plurality of offsets corresponding to each target message queue, wherein the second offset is used for representing the last message to be consumed in the second target message queue.
Optionally, the obtaining and consuming the to-be-consumed messages corresponding to the offsets between the first offset and the second offset in the target message queue includes:
and acquiring and consuming messages to be consumed corresponding to each offset between the first offset and the second offset in each target message queue corresponding to the second target label.
In a second aspect, the present application provides a message consumption apparatus comprising:
the first obtaining module is used for obtaining a first offset currently stored in the distributed coordination service cluster after the message processing platform returns to normal from an abnormal state, wherein the first offset is used for indicating the consumption progress of the message processing platform on a message in a target message queue before the message processing platform goes abnormal;
a second obtaining module, configured to obtain a second offset corresponding to the target message queue in the distributed transaction cluster, where the second offset is used to indicate a last message to be consumed in the target message queue;
the consumption module is used for acquiring and consuming messages to be consumed, which correspond to the offsets between the first offset and the second offset, in the target message queue;
and the consumption progress storage module is configured to send the second offset to the distributed coordination service cluster after consumption of all messages to be consumed corresponding to each offset between the first offset and the second offset is completed, where the distributed coordination service cluster is configured to update a currently stored offset to the second offset when the second offset is received.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
after the message processing platform is recovered from the abnormal state to be normal, acquiring a first offset currently stored in the distributed coordination service cluster, wherein the first offset is used for representing the consumption progress of the message processing platform to the message in the target message queue before the message processing platform is abnormal;
acquiring a second offset corresponding to the target message queue in the distributed transaction cluster, wherein the second offset is used for representing the last message to be consumed in the target message queue;
acquiring and consuming messages to be consumed corresponding to each offset between the first offset and the second offset in the target message queue;
and after the consumption of all messages to be consumed corresponding to each offset between the first offset and the second offset is finished, sending the second offset to the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the currently stored offset to the second offset when the second offset is received.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
after the message processing platform is recovered from the abnormal state to be normal, acquiring a first offset currently stored in the distributed coordination service cluster, wherein the first offset is used for representing the consumption progress of the message processing platform to the message in the target message queue before the message processing platform is abnormal;
acquiring a second offset corresponding to the target message queue in the distributed transaction cluster, wherein the second offset is used for representing the last message to be consumed in the target message queue;
acquiring and consuming messages to be consumed corresponding to each offset between the first offset and the second offset in the target message queue;
and after the consumption of all messages to be consumed corresponding to each offset between the first offset and the second offset is finished, sending the second offset to the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the currently stored offset to the second offset when the second offset is received.
Based on the method, after the message processing platform is recovered from the abnormal state to be normal, the message consumption progress of the message processing platform before the abnormality occurs is read from the distributed transaction cluster, so that the consumption starting point is accurately positioned, the newly received message to be consumed is obtained from the distributed transaction cluster, the consumption cut-off point is determined, all the messages to be consumed from the consumption starting point to the consumption cut-off point are consumed, after the consumption of all the messages to be consumed is finished, the message processing platform stores the second offset corresponding to the consumption cut-off point into the distributed coordination service cluster, the message processing platform is ensured to obtain the latest consumption progress from the distributed coordination service cluster, the execution progress of a consumption task cannot be accurately obtained under the condition that the message processing platform is abnormal in consumption is avoided, after the message processing platform is recovered from the abnormal state to be normal, the situation that resources consume the consumed messages again or data are lost due to the lost of the to-be-consumed messages due to inaccurate positioning of the messages to be consumed is avoided.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive labor.
FIG. 1 is a diagram of an application environment of a message consumption method in one embodiment;
FIG. 2 is a flow diagram that illustrates a method for message consumption in one embodiment;
FIG. 3 is a diagram illustrating a parameter relationship according to an embodiment;
FIG. 4 is a diagram illustrating a parameter relationship according to an embodiment;
FIG. 5 is a block diagram showing the structure of a message consuming apparatus according to one embodiment;
FIG. 6 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
FIG. 1 is a diagram of an application environment of a message consumption method in one embodiment. Referring to fig. 1, the message consumption method is applied to a message consumption system. The message consumption system includes a message processing platform 110, a distributed transaction cluster 120, and a distributed coordination service cluster 130. The message processing platform is used for acquiring and consuming messages in the distributed transaction cluster, and the message processing platform can be specifically but not limited to Storm, trident, spark SQL, spark Streaming, samza, flink and the like; the distributed transaction cluster is used for providing messages needing to be consumed for the message processing platform, and the distributed transaction cluster can be specifically, but not limited to RabbitMQ, redis, zeroMQ, activeMQ, kafka, rocketMQ and the like; the distributed coordination service cluster is used for storing the consumption progress of the message processing platform in real time, and the distributed coordination service cluster may be specifically, but not limited to, zookeeper or Etcd.
In one embodiment, fig. 2 is a flowchart illustrating a message consumption method according to an embodiment, and referring to fig. 2, a message consumption method is provided. The present embodiment is mainly exemplified by applying the method to the message processing platform 110 in fig. 1, where the message consumption method specifically includes the following steps:
step S210, after the message processing platform returns to normal from the abnormal state, acquiring a first offset currently stored in the distributed coordination service cluster.
In this embodiment, the message processing platform is specifically a SparkStreaming platform, the distributed coordination service cluster is specifically a Zookeeper cluster, the message processing platform cannot consume messages normally in an abnormal state, the state of the message processing platform is monitored in real time, after the message processing platform is restored to a normal state from the abnormal state, a first Offset which is currently stored is read from the distributed coordination service cluster, the first Offset is used for representing the consumption progress of the message processing platform on messages in a target message queue before the message processing platform is abnormal, the target message queue is used for representing a set of messages in the distributed transaction cluster, the distributed transaction is specifically a kafka cluster, each message carries a corresponding Offset, the Offset is recorded as Offset, a consumption node of the message processing platform before the message processing platform is abnormal can be known according to the first Offset, that messages corresponding to the Offset after the first Offset need to be consumed, and therefore, the consumption starting point of the message processing platform can be accurately located.
Step S220, obtaining a second offset corresponding to the target message queue in the distributed transaction cluster.
In this embodiment, the second offset is used to indicate a last message to be consumed in the target message queue, that is, indicate a latest received message in the target message queue, that is, the second offset is an offset corresponding to the latest received message in the target message queue, and the consumption cutoff point at which the message processing platform needs to consume the message is determined according to the second offset.
Step S230, obtaining and consuming the messages to be consumed corresponding to each offset between the first offset and the second offset in the target message queue.
In this embodiment, in the target message queue, messages corresponding to offsets between the first offset and the second offset according to the ordering are messages that the message processing platform needs to consume, and the consumption of the messages in the kafka cluster by the SparkStreaming platform is to read the messages from the kafka cluster and then cache the messages in the memory of the SparkStreaming platform for consumption processing at regular time.
Step S240, after the consumption of all messages to be consumed corresponding to each offset between the first offset and the second offset is finished, sending the second offset to the distributed coordination service cluster, where the distributed coordination service cluster is configured to update the currently stored offset to the second offset when receiving the second offset.
In this embodiment, after the message processing platform finishes consuming all the messages to be consumed, the second offset is stored in the distributed coordination service cluster, the first offset currently stored in the distributed coordination service cluster is updated to the second offset, the consumption progress of the message processing platform on the messages in the distributed transaction cluster is timely updated and recorded, the situation that the message processing platform in an abnormal state cannot accurately position the messages to be consumed is avoided, and the situation that resources are wasted to consume the consumed messages again due to inaccurate positioning of the messages to be consumed or the data loss is caused by missing the messages to be consumed which need to be consumed is avoided.
In one embodiment, after the message processing platform returns to normal from the abnormal state, the method further comprises: and sending a preset offset to the distributed coordination service cluster under the condition that the acquisition fails when the first offset stored currently is acquired from the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the offset stored currently to the preset offset under the condition that the preset offset is received.
In this embodiment, a failure to obtain the currently stored first offset from the distributed coordination service cluster indicates that the first offset does not exist in the distributed coordination service cluster, that is, the message processing platform does not consume the message before an exception occurs, and does not generate a consumption progress; and the message processing platform cannot normally consume the messages after the exception occurs, and cannot store the consumption progress of the consumption exception into the distributed coordination service cluster.
The message processing platform sends a preset offset to the distributed coordination service cluster under the condition that the first offset does not exist in the distributed coordination service cluster, the distributed coordination service cluster creates a first message label under the condition that the preset offset is received, the preset offset is set to be the offset corresponding to the first message label, the offset corresponding to the first message label is used as the currently stored offset, the preset offset is usually set to be 0, the subsequent message processing platform determines the consumption starting point of the message to be consumed according to the preset offset, and the message processing platform starts to consume the message which is received from the earliest in the target message queue.
In one embodiment, in the case that a message to be consumed corresponding to an arbitrary offset between the first offset and the second offset is consumed abnormally, the method further includes: detecting whether the state of the message processing platform is recovered to normal, wherein the second offset to be sent to the distributed coordination service cluster is cancelled when the message processing platform is abnormal; and after the message processing platform is recovered from the abnormal state to be normal, executing the step of acquiring the first offset currently stored in the distributed coordination service cluster.
In this embodiment, because the message to be consumed is consumed abnormally, all the message to be consumed with abnormal consumption is taken as a message not consumed, that is, the message processing platform does not change the consumption progress of the message in the distributed transaction cluster, the sending of the second offset to the distributed coordination service cluster is cancelled, the consumption progress stored in the distributed coordination service cluster is not updated, whether the message processing platform recovers the normal state is detected in real time, after the message processing platform recovers the normal state, the consumption starting point of the message to be consumed by the consumption processing platform is accurately positioned according to the distributed coordination service cluster again, and the consumption is performed according to the steps in the above embodiment.
In one embodiment, the consuming an exception of the message to be consumed corresponding to any offset between the first offset and the second offset includes: and in the process that the message to be consumed corresponding to any offset between the first offset and the second offset is consumed, determining that the message to be consumed is consumed abnormally under the condition that at least one of null pointer abnormality, message to be consumed analysis abnormality, message to be consumed reading abnormality and consumption task corresponding to the message to be consumed is forcibly ended.
In this embodiment, when at least one of an empty pointer exception, a message to be consumed analysis exception, a message to be consumed reading exception, a consumption task corresponding to the message to be consumed being forcibly ended, and the like occurs in a process of consuming the message to be consumed, it is determined that the message to be consumed cannot be normally consumed due to an exception of a state of the message processing platform. Null pointer exceptions are exceptions that are generated when an object is not present and its method is called, or when a field is accessed or modified for which an object is not present. The message analysis exception refers to that the analyzed data is not matched with preset data, and specifically includes, but is not limited to, format mismatch, the analyzed data contains illegal characters, type conversion mismatch, and the like.
In an embodiment, referring to fig. 3, the distributed coordination service cluster includes a plurality of first message tags, where the first message tags correspond to a plurality of first partitions, each of the first partitions corresponds to one of the first offsets, and the obtaining of the first offset currently stored in the distributed coordination service cluster includes obtaining the first offset corresponding to each of the first message tags in the following manner: acquiring a plurality of first target partitions corresponding to first target tags stored in the distributed coordination service cluster, wherein the first target tags are currently processed message tags in the plurality of first message tags; and acquiring a first offset corresponding to each first target partition in the distributed coordination service cluster, wherein the first offset corresponding to each first target partition is used for representing the consumption progress of the message in a target message queue.
In this embodiment, the message tag is denoted as Topic, and is used to indicate the category of the message, and the publisher publishes a message must specify Topic, and the subscriber can consume the message in this category by subscribing to Topic. The method comprises the steps that partitions are marked as Partition, each Topic corresponds to at least one Partition, a first message label is a message label in a distributed coordination service cluster, the first Partition is a Partition corresponding to the first message label, a message processing platform reads a first offset corresponding to each first target Partition under the first target label from the distributed coordination service cluster, the first target Partition is a first Partition corresponding to the first target label, and each first Partition records consumption progress of messages in a message storage queue, namely consumption progress of a plurality of target message queues is recorded corresponding to the first target label.
In an embodiment, referring to fig. 4, the distributed transaction cluster includes a second message tag corresponding to the first message tag, where the second message tag corresponds to a plurality of message storage queues, each message storage queue corresponds to one first partition, each message storage queue includes a plurality of messages to be consumed, each message to be consumed carries a corresponding offset, and the obtaining of the second offset in the distributed transaction cluster includes obtaining each second offset corresponding to the first message tag in the following manner: acquiring a plurality of target message queues corresponding to a second target tag in the distributed transaction cluster, wherein the second target tag is a second message tag corresponding to the first target tag in the distributed transaction cluster, and the target message queue is a message storage queue corresponding to the second target tag; and selecting a second offset corresponding to each second target message queue from a plurality of offsets corresponding to each target message queue, wherein the second offset is used for representing the last message to be consumed in the second target message queue.
In this embodiment, in a distributed transaction cluster, a second message tag corresponding to a first target message tag is used as a second target tag, a plurality of target message queues corresponding to the second target tag are obtained, each target message queue includes a plurality of messages to be consumed, each message to be consumed carries a corresponding offset, a second offset is selected from the plurality of offsets corresponding to the target message queues, the second offset is an offset corresponding to a message that is received most recently in the target message queues, for example, offset N in fig. 4 is an offset corresponding to a message that is received most recently, offset N is used as the second offset, only one message storage queue is exemplarily illustrated in fig. 4, corresponding relationships between the remaining message storage queues and the offsets are similar to message storage queue 01, each target message queue determines a message to be consumed that needs to execute a consumption task according to the corresponding first offset and second offset, and a message processing platform can simultaneously consume the message to be consumed in each target message queue, thereby improving consumption efficiency.
The effect of message storage capacity expansion can be realized by dividing the Topic into a plurality of partitions, and because the storage size of one file is limited, the number of messages which can be stored by one Topic can be greatly increased by storing a plurality of files in a cluster. The messages are stored through a plurality of partition files, so that a publisher and a consumer can read and write one Topic in parallel, and the effect of parallel reading and writing is achieved.
In an embodiment, the obtaining and consuming messages to be consumed corresponding to offsets between the first offset and the second offset in the target message queue includes: and acquiring and consuming messages to be consumed corresponding to each offset between the first offset and the second offset in each target message queue corresponding to the second target label.
In this embodiment, the message processing platform executes consumption tasks on a plurality of target message queues under the same message label at the same time, so that the consumption efficiency is improved, the target message queues determine a consumption starting point and a consumption stopping point according to the first offset and the second offset, a message to be consumed, which needs to be consumed, can be quickly and accurately positioned, resource waste caused by repeated consumption is avoided, and data loss caused by consumption omission is avoided.
Fig. 2 is a flow diagram illustrating a message consumption method in one embodiment. It should be understood that, although the steps in the flowchart of fig. 2 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 2 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 5, there is provided a message consumption apparatus including:
a first obtaining module 310, configured to obtain a first offset currently stored in a distributed coordination service cluster after a message processing platform returns to a normal state from an abnormal state, where the first offset is used to indicate a consumption progress of the message processing platform on a message in a target message queue before an abnormality occurs;
a second obtaining module 320, configured to obtain a second offset corresponding to the target message queue in the distributed transaction cluster, where the second offset is used to represent a last message to be consumed in the target message queue;
a consuming module 330, configured to obtain and consume to-be-consumed messages in the target message queue, where the to-be-consumed messages correspond to offsets between the first offset and the second offset;
the consumption progress storage module 340 is configured to send the second offset to the distributed coordination service cluster after the consumption of all to-be-consumed messages corresponding to each offset between the first offset and the second offset is completed, where the distributed coordination service cluster is configured to update a currently stored offset to the second offset when the second offset is received.
In one embodiment, after the message processing platform returns to normal from the abnormal state, the apparatus further comprises:
and the offset setting module is used for sending a preset offset to the distributed coordination service cluster under the condition that the acquisition fails when the first offset stored currently is acquired from the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the offset stored currently to the preset offset under the condition that the preset offset is received.
In one embodiment, in the case that the message to be consumed corresponding to any offset between the first offset and the second offset is consumed abnormally, the apparatus further includes a status monitoring module, where the status monitoring module is configured to:
detecting whether the state of the message processing platform is recovered to normal, wherein the second offset to be sent to the distributed coordination service cluster is cancelled when the message processing platform is abnormal;
and after the message processing platform is recovered from the abnormal state to be normal, executing the step of acquiring the first offset currently stored in the distributed coordination service cluster.
In one embodiment, the status monitoring module is further configured to:
and in the process that the message to be consumed corresponding to any offset between the first offset and the second offset is consumed, determining that the message to be consumed is consumed abnormally under the condition that at least one of null pointer abnormality, message to be consumed analysis abnormality, message to be consumed reading abnormality and consumption task corresponding to the message to be consumed is forcibly ended.
In an embodiment, the distributed coordination service cluster includes a plurality of first message tags, where the first message tags correspond to a plurality of first partitions, and each of the first partitions corresponds to one of the first offsets, and the first obtaining module 310 is further configured to obtain the first offset corresponding to each of the first message tags as follows:
acquiring a plurality of first target partitions corresponding to first target tags stored in the distributed coordination service cluster, wherein the first target tags are currently processed message tags in the plurality of first message tags;
and acquiring a first offset corresponding to each first target partition in the distributed coordination service cluster, wherein the first offset corresponding to each first target partition is used for representing the consumption progress of the message in a target message queue.
In an embodiment, the distributed transaction cluster includes a second message tag corresponding to the first message tag, where the second message tag corresponds to a plurality of message storage queues, each message storage queue corresponds to one first partition, each message storage queue includes a plurality of messages to be consumed, each message to be consumed carries a corresponding offset, and the second obtaining module 320 is further configured to obtain each second offset corresponding to the first message tag according to the following manner:
acquiring a plurality of target message queues corresponding to a second target tag in the distributed transaction cluster, wherein the second target tag is a second message tag corresponding to the first target tag in the distributed transaction cluster, and the target message queue is a message storage queue corresponding to the second target tag;
and selecting a second offset corresponding to each second target message queue from a plurality of offsets corresponding to each target message queue, wherein the second offset is used for representing the last message to be consumed in each second target message queue.
In one embodiment, the consumption module 330 is further configured to:
and acquiring and consuming messages to be consumed corresponding to each offset between the first offset and the second offset in each target message queue corresponding to the second target label.
FIG. 6 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may specifically be the terminal 110 (or the server 120) in fig. 1. As shown in fig. 6, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program that, when executed by the processor, causes the processor to implement a message consumption method. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a message consumption method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 6 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, the message consumption apparatus provided in the present application may be implemented in the form of a computer program that is executable on a computer device as shown in fig. 6. The memory of the computer device may store therein various program modules constituting the message consumption apparatus, such as a first acquisition module, a second acquisition module, a consumption module, and a consumption progress storage module shown in fig. 5. The computer program constituted by the respective program modules causes the processor to execute the steps in the message consumption method of the respective embodiments of the present application described in the present specification.
The computer device shown in fig. 6 may execute, by a first obtaining module in the message consuming apparatus shown in fig. 5, after the message processing platform returns to normal from the abnormal state, obtaining a first offset currently saved in the distributed coordination service cluster, where the first offset is used to indicate a consumption progress of the message processing platform on the message in the target message queue before the message processing platform goes abnormal. The computer device may execute, by a second obtaining module, obtaining a second offset corresponding to the target message queue in the distributed transaction cluster, where the second offset is used to represent a last message to be consumed in the target message queue. And the computer equipment can acquire and consume the messages to be consumed corresponding to the offsets between the first offset and the second offset in the target message queue through a consumption module. And the computer equipment can execute, by using a consumption progress storage module, after the consumption of all messages to be consumed corresponding to the offsets between the first offset and the second offset is finished, sending the second offset to the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the currently stored offset to the second offset when the second offset is received.
In one embodiment, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of any of the above embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the method of any of the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct bused dynamic RAM (DRDRAM), and bused dynamic RAM (RDRAM).
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A message consumption method, the method comprising:
after the message processing platform is recovered from the abnormal state to be normal, acquiring a first offset currently stored in the distributed coordination service cluster, wherein the first offset is used for representing the consumption progress of the message processing platform to the message in the target message queue before the message processing platform is abnormal;
acquiring a second offset corresponding to the target message queue in the distributed transaction cluster, wherein the second offset is used for representing the last message to be consumed in the target message queue;
acquiring and consuming messages to be consumed corresponding to each offset between the first offset and the second offset in the target message queue;
and after the consumption of all messages to be consumed corresponding to each offset between the first offset and the second offset is finished, sending the second offset to the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the currently stored offset to the second offset when the second offset is received.
2. The method of claim 1, wherein after the message processing platform returns to normal from the exception state, the method further comprises:
and sending a preset offset to the distributed coordination service cluster under the condition that the acquisition fails when the first offset stored currently is acquired from the distributed coordination service cluster, wherein the distributed coordination service cluster is used for updating the offset stored currently to the preset offset under the condition that the preset offset is received.
3. The method of claim 1, wherein in the event that a message to be consumed corresponding to any offset between the first offset and the second offset is consumed abnormally, the method further comprises:
detecting whether the state of the message processing platform is recovered to normal, wherein the second offset to be sent to the distributed coordination service cluster is cancelled when the message processing platform is abnormal;
and after the message processing platform is recovered from the abnormal state to be normal, executing the step of acquiring the first offset currently stored in the distributed coordination service cluster.
4. The method of claim 3, wherein the consuming the message to be consumed corresponding to any offset between the first offset and the second offset consumes abnormally, comprising:
and in the process that the message to be consumed corresponding to any offset between the first offset and the second offset is consumed, determining that the message to be consumed is consumed abnormally under the condition that at least one of null pointer abnormality, message to be consumed analysis abnormality, message to be consumed reading abnormality and consumption task corresponding to the message to be consumed is forcibly ended.
5. The method of claim 1, wherein the distributed coordination service cluster includes a plurality of first message tags, the first message tags correspond to a plurality of first partitions, each of the first partitions corresponds to one of the first offsets, and the obtaining of the first offset currently stored in the distributed coordination service cluster includes obtaining the first offset corresponding to each of the first message tags as follows:
acquiring a plurality of first target partitions corresponding to first target tags stored in the distributed coordination service cluster, wherein the first target tags are message tags currently processed in the plurality of first message tags;
and acquiring a first offset corresponding to each first target partition in the distributed coordination service cluster, wherein the first offset corresponding to each first target partition is used for indicating the consumption progress of the message in one target message queue.
6. The method of claim 5, wherein a second message tag corresponding to the first message tag is included in the distributed transaction cluster, the second message tag corresponds to a plurality of message storage queues, each message storage queue corresponds to one of the first partitions, each message storage queue includes a plurality of messages to be consumed, each message to be consumed carries a corresponding offset, and the obtaining the second offset in the distributed transaction cluster includes obtaining each second offset corresponding to the first message tag as follows:
acquiring a plurality of target message queues corresponding to a second target tag in the distributed transaction cluster, wherein the second target tag is a second message tag corresponding to the first target tag in the distributed transaction cluster, and the target message queues are message storage queues corresponding to the second target tag;
and selecting a second offset corresponding to each second target message queue from a plurality of offsets corresponding to each target message queue, wherein the second offset is used for representing the last message to be consumed in the second target message queue.
7. The method of claim 6, wherein the obtaining and consuming the messages to be consumed in the target message queue corresponding to each offset between the first offset and the second offset comprises:
and acquiring and consuming messages to be consumed corresponding to the offsets between the first offset and the second offset in each target message queue corresponding to the second target label.
8. A message consumption apparatus, the apparatus comprising:
the system comprises a first obtaining module, a second obtaining module and a third obtaining module, wherein the first obtaining module is used for obtaining a first offset which is currently stored in a distributed coordination service cluster after a message processing platform is recovered from an abnormal state, and the first offset is used for representing the consumption progress of the message processing platform to a message in a target message queue before the message processing platform is abnormal;
a second obtaining module, configured to obtain a second offset corresponding to the target message queue in the distributed transaction cluster, where the second offset is used to indicate a last message to be consumed in the target message queue;
the consumption module is used for acquiring and consuming messages to be consumed, which correspond to the offsets between the first offset and the second offset, in the target message queue;
and the consumption progress storage module is configured to send the second offset to the distributed coordination service cluster after consumption of all messages to be consumed corresponding to each offset between the first offset and the second offset is completed, where the distributed coordination service cluster is configured to update a currently stored offset to the second offset when the second offset is received.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110396874.3A CN115202898A (en) | 2021-04-13 | 2021-04-13 | Message consumption method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110396874.3A CN115202898A (en) | 2021-04-13 | 2021-04-13 | Message consumption method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115202898A true CN115202898A (en) | 2022-10-18 |
Family
ID=83571225
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110396874.3A Pending CN115202898A (en) | 2021-04-13 | 2021-04-13 | Message consumption method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115202898A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115658347A (en) * | 2022-11-04 | 2023-01-31 | 拉卡拉支付股份有限公司 | Data consumption method, device, electronic equipment, storage medium and program product |
-
2021
- 2021-04-13 CN CN202110396874.3A patent/CN115202898A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115658347A (en) * | 2022-11-04 | 2023-01-31 | 拉卡拉支付股份有限公司 | Data consumption method, device, electronic equipment, storage medium and program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107016480B (en) | Task scheduling method, device and system | |
| CN111026725A (en) | Data processing method and device for consumption data, computer equipment and storage medium | |
| CN110309109B (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN108255620B (en) | Service logic processing method, device, service server and system | |
| CN108446172B (en) | Data calling method and device, computer equipment and storage medium | |
| CN111190753B (en) | Distributed task processing method and device, storage medium and computer equipment | |
| CN110222535B (en) | Processing device, method and storage medium for block chain configuration file | |
| CN112948450B (en) | Method and device for Flink streaming processing engine for real-time recommendation and computer equipment | |
| CN112905323B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN106155877A (en) | A kind of fuzz testing method and system of Android application | |
| CN111475376A (en) | Method and device for processing test data, computer equipment and storage medium | |
| CN111768097A (en) | Method, device and system for monitoring task execution state and storage medium | |
| CN111949389B (en) | Slurm-based information acquisition method and device, server and computer-readable storage medium | |
| CN111400294A (en) | Data anomaly monitoring method, device and system | |
| CN115202898A (en) | Message consumption method and device, computer equipment and storage medium | |
| JP2011070364A (en) | Operation management server, roll back execution method, and roll back execution program | |
| CN115373886A (en) | Service group container shutdown method, device, computer equipment and storage medium | |
| CN115269252A (en) | Application program fault processing method, device, equipment and storage medium | |
| CN114661432A (en) | Task scheduling method, device, equipment and storage medium | |
| CN113176906A (en) | Model management method, model management device, computer equipment and storage medium | |
| CN112948096A (en) | Batch scheduling method, device and equipment | |
| CN113157569A (en) | Automatic testing method and device, computer equipment and storage medium | |
| CN114253825A (en) | Memory leak detection method and device, computer equipment and storage medium | |
| CN112257004B (en) | Method, device, server and storage medium for page backup | |
| CN115794209A (en) | Method and device for generating publishing scheme, computer equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |