CN110941502B - Message processing method, device, storage medium and equipment - Google Patents
Message processing method, device, storage medium and equipment Download PDFInfo
- Publication number
- CN110941502B CN110941502B CN201911295625.4A CN201911295625A CN110941502B CN 110941502 B CN110941502 B CN 110941502B CN 201911295625 A CN201911295625 A CN 201911295625A CN 110941502 B CN110941502 B CN 110941502B
- Authority
- CN
- China
- Prior art keywords
- message
- processing
- retry
- state
- consumption
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The embodiment of the invention discloses a message processing method, a message processing device, a storage medium and a device. Wherein the method comprises the following steps: and consuming the first message from the distributed message system through the first consumption instance, adding a state record corresponding to the first message into a preset state table in the memory storage system, wherein the state record comprises identification information and processing states corresponding to the message, updating the processing states in the state record corresponding to the first message into processing failures when detecting that the service processing corresponding to the first message fails, reading the state record corresponding to the first message from the preset state table through the first retry consumption instance, and consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure. The technical scheme provided by the embodiment of the invention can save the storage space and avoid repeated consumption and improve the message processing efficiency.
Description
Technical Field
Embodiments of the present invention relate to the field of computer technologies, and in particular, to a method, an apparatus, a storage medium, and a device for processing a message.
Background
Message middleware refers to platform-independent data communication with efficient and reliable messaging mechanisms and integration of distributed systems based on data communication. Currently, the mainstream message middleware is Kafka (Kafka), rabitmq, and the like. Taking Kafka as an example, kafka is a distributed message system based on publish/subscribe, has high throughput capability, fast read-write capability and load balancing capability, and is often applied to log processing, message distribution and the like. In the Kafka system, the message identity (id) of each message is determined by the topic (topic), partition (partition), and offset (offset). The producer is responsible for publishing the message to the kafka server, the kafka is responsible for hosting the message and load balancing of production and consumption, and the consumer pulls the subscribed message from the kafka server.
However, the current message processing schemes based on distributed message systems are still not perfect and need improvement.
Disclosure of Invention
The embodiment of the invention provides a message processing method, a device, equipment and a storage medium, which can optimize a message processing scheme based on a distributed message system.
In a first aspect, an embodiment of the present invention provides a message processing method, where the method includes:
Consuming a first message from a distributed message system through a first consumption example, and adding a state record corresponding to the first message in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message;
when detecting that the service processing corresponding to the first message fails, updating the processing state in the state record corresponding to the first message into processing failure;
and reading a state record corresponding to the first message from the preset state table through a first retry consumption example, and consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing under the condition that the processing state corresponding to the first message is determined to be processing failure.
In a second aspect, an embodiment of the present invention provides a message processing apparatus, including:
the system comprises a state record adding module, a state record generating module and a state record generating module, wherein the state record adding module is used for consuming a first message from a distributed message system through a first consumption example and adding a state record corresponding to the first message in a preset state table in a memory storage system, and the state record comprises identification information and a processing state corresponding to the message;
The state record updating module is used for updating the processing state in the state record corresponding to the first message into processing failure when detecting that the service processing corresponding to the first message fails;
and the service processing retry module is used for reading the state record corresponding to the first message from the preset state table through a first retry consumption example, and consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing under the condition that the processing state corresponding to the first message is determined to be processing failure.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a message processing method as provided by the embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor, where the processor implements a message processing method as provided in the embodiment of the present invention when the processor executes the computer program.
According to the message processing scheme provided by the embodiment of the invention, a first message is consumed from a distributed message system through a first consumption example, a state record corresponding to the first message is added in a preset state table in a memory storage system, wherein the state record contains identification information and a processing state corresponding to the message, when a service processing failure corresponding to the first message is detected, the processing state in the state record corresponding to the first message is updated to be the processing failure, the state record corresponding to the first message is read from the preset state table through a first retry consumption example, and the first message is consumed from the distributed message system according to the identification information in the state record corresponding to the first message, and the service processing is retried. By adopting the technical scheme, for the message consumed from the distributed message system, the corresponding identification information is recorded into the state table of the memory storage system, if the service processing of the message fails, the asynchronous consumption and the service processing retry of the consumer according to the identification information in the state table can be initiated, the storage space is saved, the repeated consumption is avoided, and the message processing efficiency is improved.
Drawings
Fig. 1 is a flow chart of a message processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another message processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a normal consumption flow provided in an embodiment of the present invention;
FIG. 4 is a schematic diagram of a retry consumption process according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an asynchronous service processing flow according to an embodiment of the present invention;
fig. 6 is a block diagram of a message processing apparatus according to an embodiment of the present invention;
fig. 7 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings. Furthermore, embodiments of the invention and features of the embodiments may be combined with each other without conflict.
In order to facilitate understanding of the technical solutions of the embodiments of the present invention, the following describes related arts. Taking Kafka as an example, a producer is responsible for publishing a message to Kafka (which may be a server corresponding to a Kafka system specifically), kafka is responsible for hosting the message and balancing the load of production and consumption, and a consumer pulls the subscribed message from Kafka and performs corresponding business processing. The consumer is typically a member of a subscription consumer group (consumer group), each consumer tags itself with a subscription consumer group name, and each record published to topic is passed to one consumer instance of each subscription consumer group, which may be in a separate process or on a separate machine. Typically, a consumer will advance the offset of the part of the topic that he reads sequentially as he reads the subscribed records, however, he can consume the records in any order. For example, it may be reset to some older offset to re-read past data, or jump to the latest record and use from the beginning of the present. The Kafka has higher throughput capacity, and needs to have high availability, high reliability, unique consumption and other performances when the message is consumed and business is processed. Wherein, the possibility of a system to be out of service is low, and the out of service time is short, which is usually described by high availability; each message of the high-reliability indicating system is processed as final as possible, and is not lost in the middle due to various anomalies; unique consumption means that each piece of information of the system is only and only once finally processed.
Fig. 1 is a flow chart of a message processing method according to an embodiment of the present invention, which may be performed by a message processing apparatus, where the apparatus may be implemented by software and/or hardware, and may be generally integrated into a computer device. As shown in fig. 1, the method includes:
Illustratively, the first consumption instance may be a consumer instance for performing a normal message consumption flow. The first consumption instance may be started upon detecting that a message handling event is triggered.
By way of example, the distributed messaging system may be Kafka and the memory storage system may be Redis or Codis. Redis is a data structure in memory that can be used as a database, cache, and message broker. The Codis is a distributed Redis solution, the application can use the Codis like the Redis using a single machine, the bottom layer of the Codis can process the forwarding of the Redis request, the work such as automatic load balancing, fault disaster tolerance, data migration without shutdown and the like can be realized, and compared with the Redis, the method is higher and more available. The preset state table can be designed according to the specific type and actual requirement of the memory storage system, and Redis or Codis is taken as an example, and the preset state table can be a state table of a sort type.
The first message may be, for example, any message in a distributed messaging system. After the first consumption example consumes the first message from the distributed message system, a state record corresponding to the first message is added in a preset state table in the memory storage system, wherein the state record comprises identification information corresponding to the message and a processing state, that is, the state record corresponding to the first message comprises the identification information corresponding to the first message and the processing state corresponding to the first message. The identification information may be information for representing a unique identity of a message, and may be referred to as a message id, and, in the example of kafka, a message identity (id) of each message is determined by a topic (topic), a partition (partition), and an offset (offset), and thus topic, partition and offset may be included in the identification information. Where topic represents the category of the message, a topic may have zero, one or more consumer groups to subscribe to the producer's data written to the topic, a topic containing one or more parts; the partition is an ordered, constant sequence of records, each record in the partition being assigned an offset; the offset is a sequential ID that uniquely identifies a record in the partition. Since the first message has been consumed from the distributed messaging system, the processing state corresponding to the first message may be in process at this time.
And 102, when detecting that the service processing corresponding to the first message fails, updating the processing state in the state record corresponding to the first message into processing failure.
The specific execution process of the service processing in the embodiment of the invention is not limited, and a synchronous processing or asynchronous processing mode can be adopted, and the service processing can be processed by a local machine or other equipment. When the failure of the service processing corresponding to the first message is detected, namely when the abnormality of the service processing process corresponding to the first message is detected, the processing state in the state record corresponding to the first message is updated to the processing failure, so that the processing state of the message is tracked in real time. In addition, the embodiment of the invention can greatly reduce the huge flow and the extra storage space caused by saving the message content by adopting the state table to record the processing state of the message.
The first retry consumption instance may be, for example, a consumer instance for performing a message retry consumption procedure. The first retry consumption instance may be initiated upon detecting that the message handling event is triggered; the first retry consumption instance can also be started when detecting that a state record of processing state which is processing failure exists in a preset state table; the first retry consumption instance may also be started at other occasions, which is not limited by the embodiment of the present invention. When the service processing corresponding to the first message fails, the first message is re-consumed through the first retry consumption instance, so that asynchronous performance of service processing failure retry can be realized, and the message processing efficiency is improved.
For example, there may be status records corresponding to a plurality of messages in the preset status table, and the first retry consumption instance may read the status records according to a set policy (e.g., read in order of adding time of the status records from front to back). When the state record corresponding to the first message is read, checking whether the processing state in the state record is processing failure, if so, finding the first message according to the identification information in the state record, then, re-messaging the first message, and retrying service processing.
According to the message processing scheme provided by the embodiment of the invention, a first message is consumed from a distributed message system through a first consumption example, a state record corresponding to the first message is added in a preset state table in a memory storage system, wherein the state record contains identification information and a processing state corresponding to the message, when a service processing failure corresponding to the first message is detected, the processing state in the state record corresponding to the first message is updated to be the processing failure, the state record corresponding to the first message is read from the preset state table through a first retry consumption example, and the first message is consumed from the distributed message system according to the identification information in the state record corresponding to the first message, and the service processing is retried. By adopting the technical scheme, for the message consumed from the distributed message system, the corresponding identification information is recorded into the state table of the memory storage system, if the service processing of the message fails, the asynchronous consumption and the service processing retry of the consumer according to the identification information in the state table can be initiated, the storage space is saved, the repeated consumption is avoided, and the message processing efficiency is improved.
In some embodiments, when it is detected that the service processing corresponding to the first message is successful, the state record corresponding to the first message is deleted. The method has the advantages that the preset state table is adopted to record the state information of the consumed message, the state record of the message which is successfully processed by the service is deleted in time, the state table can be simplified, the storage space is saved, the speed of reading the processing state of the first retry consumption instance into the state record of the processing failure can be accelerated, the message which is processed failure can be retried more quickly, and the message processing efficiency is further improved.
In some embodiments, the state record further includes a retry time corresponding to the message; and when consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the following steps: updating retry time in the state record corresponding to the first message according to a preset retry time rule; correspondingly, after consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the steps of: if the retry service processing fails, reading a state record corresponding to the first message from the preset state table through the first retry consumption instance again, and consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying the service processing under the condition that the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to the retry time corresponding to the first message. The retry time may be understood as the time for performing the retry next time, and the preset retry time rule may be set according to the actual requirement, for example, a preset duration is increased on the basis of the current time, and the preset duration may also be set according to the actual requirement, for example, 3 minutes. The method has the advantages that the next retry time can be reasonably planned, and the problem that the retry fails because of failure is not solved when the retry is performed in a short time can be avoided, so that the probability of success of the retry is improved. It should be noted that, if only one retry is supported, the retry time may be set when the state record is added, after the state record corresponding to the first message is read from the preset state table through the first retry consumption instance, if the processing state corresponding to the first message is determined to be a processing failure and the current time is greater than or equal to the retry time corresponding to the first message, the first message is consumed from the distributed message system according to the identification information corresponding to the first message, and the service processing is retried. At this time, since consumption is not retried any more, the retry time does not need to be updated.
In some embodiments, the state record further includes a remaining retry number corresponding to the message; and when consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the following steps: updating the residual retry times in the state record corresponding to the first message according to a preset retry time rule; correspondingly, after consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the steps of: if the retry service processing fails, reading a state record corresponding to the first message from the preset state table through the first retry consumption instance again, and consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying the service processing under the condition that the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to the retry time corresponding to the first message. The preset retry number rule may be set according to actual requirements, for example, different maximum retry numbers may be set for different types of messages, and a unified maximum retry number may also be set. The advantage of this is that the probability of a message being successfully processed can be guaranteed by reasonably setting the number of retries of the message processing.
In some embodiments, after updating the remaining retry times in the state record corresponding to the first message according to a preset retry time rule, the method further includes: and if the current residual retry number is 0, performing corresponding exception handling, and deleting the state record corresponding to the first message. The advantage of this arrangement is that it avoids wasting system resources due to a large number of retries while ensuring the probability of successful processing of the message, so that the message is processed in time. In addition, the state record corresponding to the message which cannot be retried is timely deleted, the state table can be simplified, the storage space is saved, and meanwhile, the speed of reading the state record with other processing states being processing failure by the first retry consumption instance can be accelerated, so that the other message with processing failure can be retried more quickly, and the message processing efficiency is further improved. The exception handling may be, for example, pushing a relevant notification to a monitoring center or a downstream device, etc.
In some embodiments, it may further comprise: and when the state record corresponding to the first message is updated, if the update failure is detected, writing the corresponding update content into a memory cache. Correspondingly, the reading, by the first retry consumption instance, the state record corresponding to the first message from the preset state table includes: reading a state record corresponding to the first message from the preset state table through a first retry consumption example, and reading the latest updated content corresponding to the first message from the memory cache, and correcting the state record corresponding to the first message according to the latest updated content when the latest updated content is inconsistent with the state record corresponding to the first message. The updating of the status record corresponding to the first message may include at least one of updating a processing status, updating a retry time, and updating a remaining number of retries. When the update failure is detected, the corresponding update content can be written into the memory cache, and when the state record needs to be read, the state record can be corrected according to the record in the memory cache, so that the correct state record is obtained, and the first retry consumption example can accurately process the message.
In some embodiments, before said consuming the first message from the distributed message system by the first consumption instance, further comprising: and simultaneously starting a plurality of consumption examples, determining a first consumption example serving as a main example through a competition mechanism corresponding to the memory storage system, wherein the examples except the first consumption example in the plurality of consumption examples become candidate consumption examples. The advantage of this arrangement is that when an exception occurs to the first consuming instance that is the primary instance, the candidate instance can be used to continue to perform the corresponding operation, improving the usability of the device.
In some embodiments, before the reading, by the first retry consumption instance, the state record corresponding to the first message from the preset state table, the method further includes: and simultaneously starting a plurality of retry consumption examples, determining a first retry consumption example serving as a main example through a competition mechanism corresponding to the memory storage system, wherein the examples except the first retry consumption example in the plurality of retry consumption examples become candidate retry consumption examples. The advantage of this arrangement is that when an exception occurs as the first retry consuming instance of the primary instance, the candidate instance can be utilized to continue performing the corresponding operation, improving the usability of the device.
In some embodiments, a processing service identification is also included in the status record. The method further comprises the steps of: and reading a state record corresponding to a second message from the preset state table through a first retry consumption example, and consuming the second message from the distributed message system according to the identification information corresponding to the second message through the first retry consumption example when the processing state corresponding to the second message is in processing and the processing service identification corresponding to the second message is different from the service identification corresponding to the first retry consumption example. Wherein the processing service identifier may be used to represent an identity of a service corresponding to an instance that processes the message. In an exemplary embodiment, the first consumption instance consumes the second message from the distributed message system, and adds a state record corresponding to the second message in a preset state table in the memory storage system, where a processing service identifier in the state record is set as a service identifier corresponding to the first consumption instance. If the service processing corresponding to the second message fails, the second message can be re-consumed through the second re-consumption instance, at this time, the processing state is that the processing service identifier is set to be the service identifier corresponding to the second re-consumption instance in the processing, when the equipment is restarted or the main instance is changed, the first re-try consumption instance becomes a new main instance, at this time, the processing service identifier in the state record of the second message is inconsistent with the service identifier corresponding to the first re-try consumption instance, and then the second message is re-consumed through the first re-try consumption instance and the service processing is re-tried. The advantage of this arrangement is that when a restart or change of master occurs, the consumer can be taken away from the message content corresponding to the message id that has not been successfully processed by the kafka consumption, and the problem of repeated consumption does not occur.
Optionally, when the processing state corresponding to the second message is in processing and the processing service identifier corresponding to the second message is the same as the service identifier corresponding to the first retry consumption instance, it may be considered that the second message is still in processing, and the retry time in the state record corresponding to the second message may be updated according to a preset retry time rule, and then the next state record is continuously read.
In some embodiments, after adding the state record corresponding to the first message in the preset state table in the memory storage system, the method further includes: and adding the first message into an asynchronous processing queue, and waiting for service processing. The advantage of the arrangement is that the asynchronous mode is adopted to process the information needing to be processed, so that the problem that the consumption speed of the information is lower than the generation speed due to abnormal conditions is avoided, and the information is backlogged in the distributed information system, and the consumption delay problem can be solved.
Fig. 2 is a flow chart of another message processing method according to an embodiment of the present invention, taking a distributed message system as Kafka and a memory storage system as Codis as an example, as shown in fig. 2, the method includes:
Wherein an instance of the plurality of consumption instances other than the first consumption instance becomes a candidate consumption instance.
The status record includes identification information, processing status, retry time, remaining retry times and processing service identification corresponding to the message, where the identification information may be a message id, i.e. topic, partition and offset corresponding to the first message. For example, the status record may be expressed as "message id+processing service sid+processing status+number of remaining retries+retry time". The message id is a fixed value, other items are updated according to actual conditions, and for the first addition, the processing service sed can be the sed corresponding to the first consumption instance, the processing state is the processing, the remaining retry times can be the preset maximum retry times or blank, and the retry time is the time or blank of the next retry.
Alternatively, the query rate per second qps limit and concurrency limit of message processing may also be checked before consuming the first message, which may be consumed if neither is limited. Optionally, when the first message is consumed, whether repeated consumption exists or not can be checked, and if so, the next message is regarded as the first message to be consumed.
The preset state table is in a sort structure, and the advantage of this arrangement is that, in this structure, the members in the set are sorted by taking the time as a score (score), so that the retry time can be taken as a score, and the state records in the preset state table can be sorted, so that the retry consumption instance can read the state records in a more reasonable order.
And 203, adding the first message into an asynchronous processing queue to wait for service processing.
Illustratively, this step may be performed by the message processing apparatus, or by the first consumption instance.
Wherein an instance of the plurality of retry consumption instances other than the first retry consumption instance becomes a candidate retry consumption instance.
It should be noted that the execution timing of step 204 may be any position before step 207, which is only schematically illustrated herein.
For the case of failure of the first processing, if the number of remaining retries is blank or the retry time is blank, the number of remaining retries and the next retry time can be updated to the maximum number of remaining retries respectively when the first processing fails.
Illustratively, the first message may still be added to the asynchronous processing queue while the traffic processing is retried, awaiting the traffic processing.
In the embodiment of the invention, when the state record is updated, if the update failure is detected, the corresponding update content is written into the memory cache. In this step, the state record corresponding to the first message is read from the preset state table through the first retry consumption instance, and the latest updated content corresponding to the first message is read from the memory cache, where the latest updated content is inconsistent with the state record corresponding to the first message, and the state record corresponding to the first message is corrected according to the latest updated content.
When the processing state corresponding to the first message is in processing and the processing service identifier corresponding to the first message is different from the service identifier corresponding to the first retry consumption instance, the first message is consumed from Kafka through the first retry consumption instance according to the identification information corresponding to the first message, and service processing is restarted; when the processing state corresponding to the first message is in processing, and the processing service identifier corresponding to the first message is the same as the service identifier corresponding to the first retry consumption instance, the first message can be considered to be still in processing, and the retry time in the state record corresponding to the first message can be updated according to the preset retry time rule, and then the next state record can be continuously read.
Illustratively, this step may be performed prior to retrying the traffic processing, i.e., prior to adding the first message to the asynchronous processing queue.
The message processing method provided by the embodiment of the invention only records the message id of the message consumed by Kafka into the state table, asynchronously processes the message, and when the retry fails, can start the message content from the consumer to the Kafka to consume the appointed message id according to the message id, thereby greatly reducing the extra storage space, avoiding the problem of lengthening the link of message consumption under normal conditions, improving the availability of the Codis, and correcting the Codis state table by using the process cache, thereby avoiding the problem of unavailability caused by network abnormality between the process and the Codis.
In order to further explain the technical scheme provided by the embodiment of the invention, the following is divided into three parts for specific description. The three parts are a normal consumption flow, a retry consumption flow and an asynchronous business processing flow respectively.
FIG. 3 is a schematic diagram of a normal consumption flow provided by an embodiment of the present invention, in which multiple instances are started simultaneously, and masters are selected by Codis competition. The main instance starts the flow as shown in fig. 3:
(1) The qps limit and concurrency limit of message processing are checked, if none are limited, one message is consumed (condume) from Kafka, whether repetition is performed, and the next message is consumed after repetition.
(2) The message id of the message, the processing service sed, the status of the message is the number of times of retry remaining in processing, the next time of retry, the Codis status table sortedset is written, and the offset is submitted to the Kafka server.
(3) And (5) delivering to asynchronous business processing. (i.e., the source of data for asynchronous business process flows)
Messages consumed from Kafka only have their message ids written into the Codis state table. When the retry fails, the consumer can be taken up from Kafka to consume the message content of the message id, so that the huge flow and extra storage space caused by writing the message content into Redis can be greatly reduced, and Codis is higher in availability than Redis.
FIG. 4 is a schematic diagram of a retry consumption flow provided by an embodiment of the present invention, in which multiple instances are started simultaneously, and masters are selected by Codis contention. The main instance starts the flow as shown in fig. 4:
(1) checking qps limitation and concurrency limitation of message processing, if the limitation is not met, reading a message to be retried from the Codis, taking out 1 record of which the retry time is less than or equal to the current time, correcting the state if the process cache also has the record, and then processing; b. if the state is in process and the process service sild-! =current service sid, considered restarted/changed, then go to flow (2); c. if the status is processing failure, flow (2) is entered.
(2) Further, kafka retries the consumption group, seek (find) to the location of the message id, and reads the message content corresponding to the message id from Kafka.
(3) And updating the state table, wherein the message id+the processing service sid+the state is the in-process+the residual retry number+the next retry time, and if the updating fails, the updating content is written into the memory.
(4) And (5) delivering to asynchronous business processing. (i.e., the source of data for the process asynchronous business process flow)
When the main retry is restarted/cut, the message content corresponding to the message id which is not processed yet can be taken out from the consumer to Kafka, and the repeated consumption is avoided.
Fig. 5 is a schematic diagram of an asynchronous service processing flow provided in an embodiment of the present invention, as shown in fig. 5:
(1) and outputting the asynchronous queue for service processing. (if the message is restarted after the asynchronous queue has not been fetched, a retry is initiated by part (1) of the retry consumption flow according to rule b)
(2) Updating a state table: if the processing is successful, deleting the message id from the state table; if the processing fails and the number of the remaining retries is not 0, updating the codis state table, and updating the state of the message id to be the processing failure and the next retry time; if the processing fails and the number of remaining retries is 0, then exception handling (e.g., push to monitoring center, push to downstream, etc.) is performed, which is also considered to be the final processing, the message id is deleted from the state table. (if the message is restarted during asynchronous processing, a retry is initiated by part (1) of the retry consumption flow according to rule b)
By adopting the scheme, consumption, processing and retry can be performed asynchronously, so that the problem of consumption delay can be effectively solved, and the message processing efficiency is improved.
Fig. 6 is a block diagram of a message processing apparatus according to an embodiment of the present invention, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in a computer device, and may perform message processing by executing a message processing method. As shown in fig. 6, the apparatus includes:
a state record adding module 601, configured to consume a first message from a distributed message system through a first consumption instance, and add a state record corresponding to the first message in a preset state table in a memory storage system, where the state record includes identification information and a processing state corresponding to the message;
a state record updating module 602, configured to update, when detecting that the service corresponding to the first message fails to process, a processing state in a state record corresponding to the first message to a processing failure;
the service processing retry module 603 is configured to read, by using a first retry consumption instance, a state record corresponding to the first message from the preset state table, and, if determining that the processing state corresponding to the first message is a processing failure, consume the first message from the distributed message system according to the identification information corresponding to the first message, and retry service processing.
The message processing device provided by the embodiment of the invention consumes the first message from the distributed message system through the first consumption instance, and adds the state record corresponding to the first message in the preset state table in the memory storage system, wherein the state record contains the identification information and the processing state corresponding to the message, when the processing failure of the service corresponding to the first message is detected, the processing state in the state record corresponding to the first message is updated to the processing failure, the state record corresponding to the first message is read from the preset state table through the first retry consumption instance, and the first message is consumed from the distributed message system according to the identification information in the state record corresponding to the first message, and the service processing is retried. By adopting the technical scheme, for the message consumed from the distributed message system, the corresponding identification information is recorded into the state table of the memory storage system, if the service processing of the message fails, the asynchronous consumption retry of the consumer according to the identification information in the state table can be started, the storage space is saved, the repeated consumption is avoided, and the message processing efficiency is improved.
In some embodiments, the apparatus may further comprise:
And the first state record deleting module is used for deleting the state record corresponding to the first message when the service corresponding to the first message is detected to be successfully processed.
In some embodiments, the state record further includes a retry time corresponding to the message; the device may further comprise:
the retry time updating module is used for updating the retry time in the state record corresponding to the first message according to a preset retry time rule while consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing;
accordingly, the service processing retry module may be further configured to:
and after the first message is consumed from the distributed message system according to the identification information corresponding to the first message and the service processing is retried, if the service processing is retried to fail, reading the state record corresponding to the first message from the preset state table again through the first retry consumption instance, and when the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to the retry time corresponding to the first message, consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying the service processing.
In some embodiments, the state record further includes a remaining retry number corresponding to the message; the device may further comprise:
the retry number updating module is used for updating the residual retry number in the state record corresponding to the first message according to a preset retry number rule while consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing;
accordingly, the service processing retry module may be further configured to:
after consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the following steps: if the retry service processing fails, reading a state record corresponding to the first message from the preset state table through the first retry consumption instance again, and consuming the first message from the distributed message system according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure and the residual retry number corresponding to the first message is greater than or equal to 1.
In some embodiments, the apparatus further comprises:
and the second state record deleting module is used for carrying out corresponding exception processing and deleting the state record corresponding to the first message if the current residual retry number is 0 after updating the residual retry number in the state record corresponding to the first message according to a preset retry number rule.
In some embodiments, the apparatus may further comprise:
the cache writing module is used for writing corresponding update content into the memory cache if the update failure is detected when the state record corresponding to the first message is updated;
correspondingly, reading the state record corresponding to the first message from the preset state table through the first retry consumption example comprises the following steps:
reading a state record corresponding to the first message from the preset state table through a first retry consumption example, and reading the latest updated content corresponding to the first message from the memory cache, and correcting the state record corresponding to the first message according to the latest updated content when the latest updated content is inconsistent with the state record corresponding to the first message.
In some embodiments, the apparatus further comprises:
and the first instance starting module is used for starting a plurality of consumption instances simultaneously before the first message is consumed from the distributed message system through the first consumption instance, determining the first consumption instance serving as a main instance through a competition mechanism corresponding to the memory storage system, wherein the instances except the first consumption instance in the plurality of consumption instances become candidate consumption instances.
In some embodiments, the apparatus further comprises:
and the second instance starting module is used for simultaneously starting a plurality of retry consumption instances before the state record corresponding to the first message is read from the preset state table through the first retry consumption instance, determining the first retry consumption instance serving as a main instance through a competition mechanism corresponding to the memory storage system, wherein the instances except the first retry consumption instance in the plurality of retry consumption instances become candidate retry consumption instances.
In some embodiments, the status record further includes a processing service identifier; the apparatus further comprises:
the service processing retry module is further configured to: and reading a state record corresponding to a second message from the preset state table through a first retry consumption example, and consuming the second message from the distributed message system according to the identification information corresponding to the second message through the first retry consumption example when the processing state corresponding to the second message is in processing and the processing service identification corresponding to the second message is different from the service identification corresponding to the first retry consumption example.
In some embodiments, the apparatus further comprises:
the asynchronous queue joining module is configured to add a state record corresponding to the first message to a preset state table in the memory storage system, and further includes: and adding the first message into an asynchronous processing queue, and waiting for service processing.
In some embodiments, the distributed messaging system is Kafka.
In some embodiments, the memory storage system is Redis or Codis.
The embodiments of the present invention also provide a storage medium containing computer-executable instructions which, when executed by a computer processor, are used to perform the message processing method provided by the embodiments of the present invention.
Storage media-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk or tape devices; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDORAM, rambus (Rambus) RAM, etc.; nonvolatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a second, different computer system connected to the first computer system through a network such as the internet. The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected by a network). The storage medium may store program instructions (e.g., embodied as a computer program) executable by one or more processors.
The embodiment of the invention provides a computer device, and the message processing device provided by the embodiment of the invention can be integrated in the computer device. Fig. 7 is a block diagram of a computer device according to an embodiment of the present invention. The computer device 700 comprises a memory 701, a processor 702 and a computer program stored in the memory 701 and executable on the processor 702, wherein the processor 702 implements the message processing method provided by the embodiment of the invention when executing the computer program.
The message processing device, the storage medium and the computer device provided in the above embodiments can execute the message processing method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the method. Technical details not described in detail in the above embodiments may be referred to the message processing method provided in any embodiment of the present invention.
Note that the above is only a preferred embodiment of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (14)
1. A method of message processing comprising:
consuming a first message from a distributed message system through a first consumption example, and adding a state record corresponding to the first message in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message;
when detecting that the service processing corresponding to the first message fails, updating the processing state in the state record corresponding to the first message into processing failure;
reading a state record corresponding to the first message from the preset state table through a first retry consumption example, and consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing under the condition that the processing state corresponding to the first message is determined to be processing failure;
the state record also comprises a processing service identifier, wherein the processing service identifier is a service identifier corresponding to a consumption instance for processing the message; the method further comprises the steps of:
and reading a state record corresponding to a second message from the preset state table through a first retry consumption example, and consuming the second message from the distributed message system according to the identification information corresponding to the second message through the first retry consumption example when the processing state corresponding to the second message is in processing and the processing service identification corresponding to the second message is different from the service identification corresponding to the first retry consumption example.
2. The method as recited in claim 1, further comprising:
and deleting the state record corresponding to the first message when the service processing corresponding to the first message is detected to be successful.
3. The method of claim 1, wherein the status record further includes a retry time corresponding to the message; and when consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the following steps:
updating retry time in the state record corresponding to the first message according to a preset retry time rule;
correspondingly, after consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the steps of:
if the retry service processing fails, reading a state record corresponding to the first message from the preset state table through the first retry consumption instance again, and consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying the service processing under the condition that the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to the retry time corresponding to the first message.
4. The method of claim 1, wherein the status record further includes a remaining number of retries corresponding to the message; and when consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the following steps:
updating the residual retry times in the state record corresponding to the first message according to a preset retry time rule;
correspondingly, after consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing, the method further comprises the steps of:
if the retry service processing fails, reading a state record corresponding to the first message from the preset state table through the first retry consumption instance again, and consuming the first message from the distributed message system according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure and the residual retry number corresponding to the first message is greater than or equal to 1.
5. The method of claim 4, further comprising, after updating the remaining number of retries in the state record corresponding to the first message according to a preset number of retries rule:
And if the current residual retry number is 0, performing corresponding exception handling, and deleting the state record corresponding to the first message.
6. The method of any one of claims 3-5, further comprising:
when the state record corresponding to the first message is updated, if the update failure is detected, writing the corresponding update content into a memory cache;
correspondingly, reading the state record corresponding to the first message from the preset state table through the first retry consumption example comprises the following steps:
reading a state record corresponding to the first message from the preset state table through a first retry consumption example, and reading the latest updated content corresponding to the first message from the memory cache, and correcting the state record corresponding to the first message according to the latest updated content when the latest updated content is inconsistent with the state record corresponding to the first message.
7. The method of claim 1, further comprising, prior to said consuming the first message from the distributed messaging system via the first consumption instance:
and simultaneously starting a plurality of consumption examples, determining a first consumption example serving as a main example through a competition mechanism corresponding to the memory storage system, wherein the examples except the first consumption example in the plurality of consumption examples become candidate consumption examples.
8. The method of claim 1, further comprising, prior to the reading, by the first retry consuming instance, a state record corresponding to the first message from the preset state table:
and simultaneously starting a plurality of retry consumption examples, determining a first retry consumption example serving as a main example through a competition mechanism corresponding to the memory storage system, wherein the examples except the first retry consumption example in the plurality of retry consumption examples become candidate retry consumption examples.
9. The method of claim 1, further comprising, after adding the state record corresponding to the first message to a preset state table in the memory storage system:
and adding the first message into an asynchronous processing queue, and waiting for service processing.
10. The method of claim 1, wherein the distributed messaging system is Kafka.
11. The method of claim 1, wherein the memory storage system is a Redis or a Codis.
12. A message processing apparatus, comprising:
the system comprises a state record adding module, a state record generating module and a state record generating module, wherein the state record adding module is used for consuming a first message from a distributed message system through a first consumption example and adding a state record corresponding to the first message in a preset state table in a memory storage system, and the state record comprises identification information and a processing state corresponding to the message;
The state record updating module is used for updating the processing state in the state record corresponding to the first message into processing failure when detecting that the service processing corresponding to the first message fails;
the service processing retry module is used for reading the state record corresponding to the first message from the preset state table through a first retry consumption instance, and consuming the first message from the distributed message system according to the identification information corresponding to the first message and retrying service processing under the condition that the processing state corresponding to the first message is determined to be processing failure;
the state record also comprises a processing service identifier, wherein the processing service identifier is a service identifier corresponding to a consumption instance for processing the message;
and the service processing retry module is further configured to read, by using a first retry consumption instance, a state record corresponding to a second message from the preset state table, and consume, by using the first retry consumption instance, the second message from the distributed message system according to the identification information corresponding to the second message and restart service processing when the processing state corresponding to the second message is in processing and the processing service identification corresponding to the second message is different from the service identification corresponding to the first retry consumption instance.
13. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any one of claims 1-11.
14. A message processing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-11 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911295625.4A CN110941502B (en) | 2019-12-16 | 2019-12-16 | Message processing method, device, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911295625.4A CN110941502B (en) | 2019-12-16 | 2019-12-16 | Message processing method, device, storage medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110941502A CN110941502A (en) | 2020-03-31 |
| CN110941502B true CN110941502B (en) | 2023-06-23 |
Family
ID=69911472
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911295625.4A Active CN110941502B (en) | 2019-12-16 | 2019-12-16 | Message processing method, device, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110941502B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111522672B (en) * | 2020-04-24 | 2023-02-28 | 北京思特奇信息技术股份有限公司 | Method and system for repeated consumption of fusing data |
| CN111949497B (en) * | 2020-06-23 | 2024-04-12 | 贝壳技术有限公司 | Message queue system and message processing method based on same |
| CN111884769B (en) * | 2020-08-06 | 2023-10-27 | 中国银行股份有限公司 | Message synchronization method and device |
| CN112463407A (en) * | 2020-11-25 | 2021-03-09 | 北京神州数字科技有限公司 | Message transmission and consumption method |
| CN112559217A (en) * | 2020-12-22 | 2021-03-26 | 鲁班(北京)电子商务科技有限公司 | Message reliability consumption idempotent processing method and processing device |
| CN112732460A (en) * | 2021-01-05 | 2021-04-30 | 卓望数码技术（深圳）有限公司 | Data state management method, system and storage medium |
| CN113452602B (en) * | 2021-06-21 | 2023-03-24 | 网易（杭州）网络有限公司 | Message transmission method and device, electronic equipment and storage medium |
| CN113608896B (en) * | 2021-08-12 | 2023-09-08 | 重庆紫光华山智安科技有限公司 | Method, system, medium and terminal for dynamically switching data streams |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10819648B2 (en) * | 2017-06-27 | 2020-10-27 | Atlassian Pty Ltd. | Retry handling in messaging queues |
| CN108647083B (en) * | 2018-04-28 | 2020-03-31 | 京东数字科技控股有限公司 | Task execution method, device, system, electronic equipment and computer readable medium |
| CN110532066A (en) * | 2018-05-23 | 2019-12-03 | 中国移动通信集团浙江有限公司 | A kind of realization method and system of transaction consistency |
| CN109451032B (en) * | 2018-11-20 | 2021-11-23 | 上海联寓智能科技有限公司 | Message transmission system |
-
2019
- 2019-12-16 CN CN201911295625.4A patent/CN110941502B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110941502A (en) | 2020-03-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110941502B (en) | Message processing method, device, storage medium and equipment | |
| US11210185B2 (en) | Method and system for data recovery in a data system | |
| US8132043B2 (en) | Multistage system recovery framework | |
| EP2474919B1 (en) | System and method for data replication between heterogeneous databases | |
| JPH10214199A (en) | Process restarting method, and system for realizing process restart | |
| CN110807062A (en) | Data synchronization method and device and database host | |
| CN110955506A (en) | Distributed job scheduling processing method | |
| CN113722277A (en) | Data import method, device, service platform and storage medium | |
| US20130318059A1 (en) | Transfer of data from transactional data sources to partitioned databases in restartable environment | |
| JP2000047894A (en) | Computer system | |
| CN116501783A (en) | Distributed database data importing method and system | |
| EP2416526B1 (en) | Task switching method, server node and cluster system | |
| US20230315713A1 (en) | Operation request processing method, apparatus, device, readable storage medium, and system | |
| CN113779149A (en) | Message processing method and device, electronic equipment and readable storage medium | |
| CN112015595B (en) | Master-slave database switching method, computing device and storage medium | |
| JPH11259326A (en) | Hot standby system, automatic re-execution method for the same and storage medium therefor | |
| US6944618B2 (en) | Method, computer program product, and system for unloading a hierarchical database utilizing segment specific selection criteria | |
| CN115687491A (en) | Data analysis task scheduling system based on relational database | |
| US10341434B2 (en) | Method and system for high availability topology for master-slave data systems with low write traffic | |
| US7856507B2 (en) | Data transmission process | |
| JPH11353284A (en) | Job re-executing method | |
| US20090187608A1 (en) | Handling transfer of bad data to database partitions in restartable environments | |
| JP3494788B2 (en) | Program execution management system and program execution management method | |
| CN112711382B (en) | Data storage method and device based on distributed system and storage node | |
| CN112732414B (en) | Distributed transaction processing method and system in OLTP mode and related components |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231008 Address after: 31a, 15th floor, building 30, maple commercial city, bangrang Road, Brazil Patentee after: Baiguoyuan Technology (Singapore) Co.,Ltd. Address before: 511400 floor 5-13, West Tower, building C, 274 Xingtai Road, Shiqiao street, Panyu District, Guangzhou City, Guangdong Province Patentee before: GUANGZHOU BAIGUOYUAN INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |