CN110941502A - Message processing method, device, storage medium and equipment - Google Patents
Message processing method, device, storage medium and equipment Download PDFInfo
- Publication number
- CN110941502A CN110941502A CN201911295625.4A CN201911295625A CN110941502A CN 110941502 A CN110941502 A CN 110941502A CN 201911295625 A CN201911295625 A CN 201911295625A CN 110941502 A CN110941502 A CN 110941502A
- Authority
- CN
- China
- Prior art keywords
- message
- processing
- retry
- state
- instance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The embodiment of the invention discloses a message processing method, a message processing device, a storage medium and equipment. Wherein, the method comprises the following steps: the method comprises the steps of consuming a first message from a distributed message system through a first consumption instance, adding a state record corresponding to the first message in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message, updating the processing state in the state record corresponding to the first message to be processing failure when the processing failure of a service corresponding to the first message is detected, reading the state record corresponding to the first message from the preset state table through a first retry consumption instance, and consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be the processing failure. The technical scheme provided by the embodiment of the invention can improve the message processing efficiency while saving the storage space and avoiding repeated consumption.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a message processing method, a message processing device, a message processing storage medium and message processing equipment.
Background
The message middleware is used for carrying out platform-independent data communication by using an efficient and reliable message transfer mechanism and carrying out integration of a distribution system based on data communication. Currently, the mainstream message middleware includes Kafka (Kafka) and RabbitMQ. Taking Kafka as an example, Kafka is a distributed message system based on publish/subscribe, has high throughput capability, fast read-write capability and load balancing capability, and is often applied to log processing, message distribution and the like. In the Kafka system, the message identity (id) of each message is determined by the topic (topic), partition (partition), and offset (offset). The producer is responsible for publishing messages to the kafka server, kafka is responsible for hosting messages and load balancing of production and consumption, and the consumer pulls subscribed messages from the kafka server.
However, the current message processing scheme based on the distributed message system is still not perfect, and needs to be improved.
Disclosure of Invention
The embodiment of the invention provides a message processing method, a message processing device, message processing equipment and a storage medium, which can optimize a message processing scheme based on a distributed message system.
In a first aspect, an embodiment of the present invention provides a message processing method, where the method includes:
consuming a first message from a distributed message system through a first consumption instance, and adding a state record corresponding to the first message in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message;
when the business processing failure corresponding to the first message is detected, updating the processing state in the state record corresponding to the first message into the processing failure;
and reading a state record corresponding to the first message from the preset state table through a first retry consumption instance, and consuming the first message from the distributed message system and retrying service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure.
In a second aspect, an embodiment of the present invention provides a message processing apparatus, where the apparatus includes:
the state record adding module is used for consuming a first message from the distributed message system through a first consumption instance and adding a state record corresponding to the first message in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message;
a status record updating module, configured to update a processing status in a status record corresponding to the first message to a processing failure when it is detected that the service processing corresponding to the first message fails;
and the service processing retry module is configured to read a state record corresponding to the first message from the preset state table by using a first retry consumption instance, and consume the first message from the distributed message system and retry service processing according to the identification information corresponding to the first message when it is determined that the processing state corresponding to the first message is processing failure.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a message processing method according to an embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the message processing method according to an embodiment of the present invention is implemented.
According to the message processing scheme provided by the embodiment of the invention, a first message is consumed from a distributed message system through a first consumption instance, a state record corresponding to the first message is added in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message, when the failure of service processing corresponding to the first message is detected, the processing state in the state record corresponding to the first message is updated to be processing failure, the state record corresponding to the first message is read from the preset state table through a first retry consumption instance, and the first message is consumed from the distributed message system and the service processing is retried according to the identification information in the state record corresponding to the first message. By adopting the technical scheme, the corresponding identification information of the message consumed from the distributed message system is recorded into the state table of the memory storage system, and if the service processing of the message fails, the asynchronous consumption and service processing retry can be performed by the consumer according to the identification information in the state table, so that the storage space is saved, the repeated consumption is avoided, and the message processing efficiency is improved.
Drawings
Fig. 1 is a schematic flowchart of a message processing method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of another message processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a normal consumption process according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a retry consuming process according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an asynchronous service processing flow according to an embodiment of the present invention;
fig. 6 is a block diagram of a message processing apparatus according to an embodiment of the present invention;
fig. 7 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures. In addition, the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.
In order to facilitate understanding of the technical solutions of the embodiments of the present invention, the following description will be made of related technologies. Taking Kafka as an example, the producer is responsible for publishing the message to Kafka (specifically, a server corresponding to the Kafka system), Kafka is responsible for hosting the message and load balancing of production and consumption, and the consumer pulls the subscribed message from Kafka and performs corresponding business processing. The consumer is typically a member of a group of subscribing consumers (consumer groups), each consumer tagging itself with a subscribing consumer group name, and each record published to topic is passed to one consumer instance in each subscribing consumer group, which may be in a separate process or on a separate machine. Typically, a consumer will advance the offsets of the partitions of their topic read sequentially when reading the subscribed records, however, it can consume the records in any order. For example, it may be reset to some offset older to reread past data, or jump to the newest record and use from now on. Kafka has a high throughput capability, and needs to have high availability, high reliability, unique consumption and other performances when consuming and processing the messages in Kafka. Wherein high availability generally describes a system with a low probability of out of service, short out of service time; the high reliability means that each message of the system is finally processed as much as possible and cannot be lost in the midway due to various exceptions; unique consumption means that each piece of information of the system is only and can only be finally processed once.
Fig. 1 is a flowchart illustrating a message processing method according to an embodiment of the present invention, where the method may be executed by a message processing apparatus, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in a computer device. As shown in fig. 1, the method includes:
Illustratively, the first consumption instance may be a consumer instance for performing a normal message consumption flow. The first consumption instance may be initiated upon detecting that a message processing event is triggered.
Illustratively, the distributed messaging system may be, for example, Kafka, and the memory storage system may be, for example, Redis or Codis. Redis is a data structure store in memory that can be used as a database, cache, and message broker, among other things. The Codis is a distributed Redis solution, the application can use the Codis like the Redis using a single machine, the bottom layer of the Codis can process the forwarding of Redis requests, the work of automatic load balancing, fault disaster tolerance, data migration without halt and the like can be realized, and the distributed Redis solution is more highly available compared with the Redis. The preset state table may be designed according to the specific type and actual requirements of the memory storage system, and taking Redis or Codis as an example, the preset state table may be a sortedset type state table.
Illustratively, the first message may be any one of the messages in the distributed messaging system. After the first message is consumed from the distributed message system by the first consumption instance, a state record corresponding to the first message is added to a preset state table in the memory storage system, where the state record includes identification information and a processing state corresponding to the message, that is, the state record corresponding to the first message includes the identification information and the processing state corresponding to the first message. The identification information may be information for indicating a unique identity of the message, which may be referred to as a message id, and in kafka for example, the message identity (id) of each message is determined by a topic (topic), a partition (partition), and an offset (offset), and therefore, the topic, the partition, and the offset may be included in the identification information. Wherein, topic represents the category of the message, a topic may have zero, one or more consumer groups to subscribe the data written into the topic by the producer, and a topic contains one or more partitions; partition is an ordered, invariant sequence of records, each record in a partition being assigned an offset; offset is a sequential ID that uniquely identifies a record in a partition. Since the first message has been consumed from the distributed message system, the processing state corresponding to the first message may be in-process at this time.
And 102, when the service processing failure corresponding to the first message is detected, updating the processing state in the state record corresponding to the first message into the processing failure.
The specific execution process of the service processing in the embodiment of the present invention is not limited, and a synchronous processing or asynchronous processing mode may be adopted, and the service processing may be performed by a local device or by other devices. When the service processing failure corresponding to the first message is detected, namely when the abnormality of the service processing process corresponding to the first message is detected, the processing state in the state record corresponding to the first message is updated to be the processing failure, so that the processing state of the message is tracked in real time. In addition, the embodiment of the invention records the processing state of the message by adopting the state table, thereby greatly reducing huge flow and extra storage space caused by additionally storing the message content.
Illustratively, the first retry consumption instance may be a consumer instance for performing a message retry consumption procedure. A first retry consumption instance may be initiated upon detecting that a message processing event is triggered; or starting a first retry consuming instance when detecting that the processing state is a state record of processing failure in the preset state table; the first retry consuming instance may also be initiated at other times, and the embodiment of the present invention is not limited. When the service processing corresponding to the first message fails, the first message is re-consumed through the first retry consumption instance, so that the asynchronous retry of the service processing failure can be realized, and the message processing efficiency is improved.
For example, there may be a plurality of status records corresponding to messages in the preset status table, and the first retry consuming instance may read the status records according to a set policy (e.g., read in order from the front to the back of the status record adding time). And when the state record corresponding to the first message is read, checking whether the processing state in the state record is processing failure, if the processing state is processing failure, finding the first message in the distributed message system according to the identification information in the state record, re-sending the first message, and re-trying the service processing.
According to the message processing scheme provided by the embodiment of the invention, a first message is consumed from a distributed message system through a first consumption instance, a state record corresponding to the first message is added in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message, when the failure of service processing corresponding to the first message is detected, the processing state in the state record corresponding to the first message is updated to be processing failure, the state record corresponding to the first message is read from the preset state table through a first retry consumption instance, and the first message is consumed from the distributed message system and the service processing is retried according to the identification information in the state record corresponding to the first message. By adopting the technical scheme, the corresponding identification information of the message consumed from the distributed message system is recorded into the state table of the memory storage system, and if the service processing of the message fails, the asynchronous consumption and service processing retry can be performed by the consumer according to the identification information in the state table, so that the storage space is saved, the repeated consumption is avoided, and the message processing efficiency is improved.
In some embodiments, when it is detected that the service processing corresponding to the first message is successful, the status record corresponding to the first message is deleted. The method has the advantages that the preset state table is used for recording the state information of the consumed message, the state record of the message which is successfully processed by the service is deleted in time, the state table can be simplified, the storage space is saved, the speed of reading the state record of which the processing state is the processing failure by the first retry consumption instance can be increased, the message which is processed by the processing failure can be retried more quickly, and the message processing efficiency is further improved.
In some embodiments, the status record further includes a retry time corresponding to the message; when the consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes: updating the retry time in the state record corresponding to the first message according to a preset retry time rule; correspondingly, after consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes: and if the retry service processing fails, reading the state record corresponding to the first message from the preset state table again through the first retry consumption instance, and consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to the retry time corresponding to the first message. The retry time may be understood as the time for the next retry, and the preset retry time rule may be set according to an actual requirement, for example, a preset time duration is added based on the current time, and the preset time duration may also be set according to the actual requirement, for example, 3 minutes. The advantage of setting up like this is that can rationally plan to retry the time next, can avoid when retrying again in the short time because of leading to the problem of failure failing not being solved and again fail to improve the probability of retrying the success. It should be noted that, if only one retry is supported, a retry time may be set when a state record is added, and after a state record corresponding to the first message is read from the preset state table by a first retry consuming instance, if it is determined that the processing state corresponding to the first message is processing failure and the current time is greater than or equal to the retry time corresponding to the first message, the first message is consumed from the distributed message system according to the identification information corresponding to the first message and service processing is retried. At this time, since consumption is not retried any more, it is not necessary to update the retry time.
In some embodiments, the status record further includes a remaining number of retries corresponding to the message; when the consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes: updating the residual retry times in the state record corresponding to the first message according to a preset retry time rule; correspondingly, after consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes: and if the retry service processing fails, reading the state record corresponding to the first message from the preset state table again through the first retry consumption instance, and consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to the retry time corresponding to the first message. The preset retry number rule may be set according to actual requirements, for example, different maximum retry numbers may be set for different types of messages, or a uniform maximum retry number may also be set. The advantage of this is that the probability of successful processing of the message can be guaranteed by reasonably setting the number of retries of the message processing.
In some embodiments, after updating the remaining retry number in the status record corresponding to the first message according to a preset retry number rule, the method further includes: and if the current residual retry number is 0, performing corresponding exception handling, and deleting the state record corresponding to the first message. The advantage of this arrangement is that while ensuring the probability of successful processing of the message, it avoids the waste of system resources caused by a large number of retries, so that the message is finally processed in time. In addition, the state record corresponding to the message which cannot be retried any more is deleted in time, so that the state table can be simplified, the storage space is saved, and meanwhile, the speed of reading the state record of which the other processing state is the processing failure by the first retry consuming instance can be increased, so that the other processing failure messages can be retried more quickly, and the message processing efficiency is further improved. The exception handling may be, for example, pushing a relevant notification to a monitoring center or a downstream device.
In some embodiments, it may further include: and when the state record corresponding to the first message is updated, if the update failure is detected, writing the corresponding update content into a memory cache. Correspondingly, the reading, by the first retry consuming instance, the state record corresponding to the first message from the preset state table includes: and reading a state record corresponding to the first message from the preset state table through a first retry consumption instance, reading the latest updated content corresponding to the first message from the memory cache, and correcting the state record corresponding to the first message according to the latest updated content when the latest updated content is inconsistent with the state record corresponding to the first message. The updating the status record corresponding to the first message may include at least one of updating a processing status, updating a retry time, and updating a remaining number of retries. When the update failure is detected, the corresponding update content can be written into the memory cache, and when the state record needs to be read, the correct state record can be obtained by correcting according to the record in the memory cache, so that the first retry consumption instance can be ensured to accurately process the message.
In some embodiments, prior to said consuming, by the first consuming instance, the first message from the distributed messaging system, further comprising: and simultaneously starting a plurality of consumption instances, determining a first consumption instance serving as a main instance through a competition mechanism corresponding to the memory storage system, wherein the instances except the first consumption instance in the plurality of consumption instances become candidate consumption instances. The advantage of this arrangement is that when the first consumption instance as the main instance is abnormal, the candidate instance can be used to continue to perform corresponding operations, thereby improving the usability of the device.
In some embodiments, before the reading, by the first retry consuming instance, the status record corresponding to the first message from the preset status table, the method further includes: and simultaneously starting a plurality of retry consumption instances, determining a first retry consumption instance serving as a main instance through a competition mechanism corresponding to the memory storage system, wherein the instances except the first retry consumption instance in the retry consumption instances become candidate retry consumption instances. The advantage of this arrangement is that when the first retry consuming instance as the main instance is abnormal, the candidate instance can be used to continue to perform corresponding operations, thereby improving the usability of the device.
In some embodiments, a processing service identification is also included in the status record. The method further comprises the following steps: and reading a state record corresponding to a second message from the preset state table through a first retry consuming instance, and consuming the second message from the distributed message system and restarting service processing through the first retry consuming instance according to identification information corresponding to the second message when a processing state corresponding to the second message is in processing and a processing service identification corresponding to the second message is different from a service identification corresponding to the first retry consuming instance. Wherein, the processing service identifier can be used to represent the identity of the service corresponding to the instance processing the message. Illustratively, the second message is consumed from the distributed message system through the first consumption instance, and the state record corresponding to the second message is added to the preset state table in the memory storage system, at this time, the processing service identifier in the state record is set as the service identifier corresponding to the first consumption instance. If the service processing corresponding to the second message fails, the second message can be re-consumed through the second re-consumption instance, at this time, the processing state is in processing, the processing service identifier is set to be the service identifier corresponding to the second re-consumption instance, when an event of equipment restart or owner change (master instance change) exists, the first re-consumption instance becomes a new master instance, at this time, the processing service identifier in the state record of the second message is not consistent with the service identifier corresponding to the first re-consumption instance, the second message is re-consumed through the first re-consumption instance, and the service processing is re-performed. The advantage of this arrangement is that when the restart or master change occurs, the message content corresponding to the message id which is not successfully processed by the consumer can be consumed by kafka, and the problem of repeated consumption can not be generated.
Optionally, when the processing state corresponding to the second message is in processing, and the processing service identifier corresponding to the second message is the same as the service identifier corresponding to the first retry consumption instance, it may be considered that the second message is still in processing, the retry time in the state record corresponding to the second message may be updated according to a preset retry time rule, and then the next state record is continuously read.
In some embodiments, after adding the state record corresponding to the first message in the preset state table in the memory storage system, the method further includes: and adding the first message into an asynchronous processing queue to wait for service processing. The method has the advantages that the message needing to be subjected to service processing is processed in an asynchronous mode, the phenomenon that the message consumption speed is lower than the generation speed due to abnormal conditions is avoided, the message is backlogged in a distributed message system, and the problem of consumption delay can be solved.
Fig. 2 is a schematic flow chart of another message processing method according to an embodiment of the present invention, taking a distributed message system as Kafka and a memory storage system as Codis as an example, as shown in fig. 2, the method includes:
Wherein an instance of the plurality of consuming instances other than the first consuming instance becomes a candidate consuming instance.
The status record includes identification information, a processing status, a retry time, a remaining retry number, and a processing service identifier corresponding to the message, where the identification information may be a message id, that is, a topic, a partition, and an offset corresponding to the first message. Illustratively, the state record may be expressed as "message id + processing service sid + processing state + number of remaining retries + retry time". The message id is a fixed value, other items are updated according to actual conditions, when adding for the first time, the processing service sid can be a sid corresponding to the first consumption instance, the processing state is in processing, the remaining retry number can be a preset maximum retry number or a blank, and the retry time is a time or a blank for next retry.
Optionally, before consuming the first message, the query rate per second qps limit and concurrency limit for message processing may also be checked, and if neither is limited, the first message may be consumed. Optionally, when the first message is consumed, it may be further checked whether there is a repeated consumption condition, and if so, the next message is considered as the first message to be consumed.
Illustratively, the preset state table is a sortedset structure, which is advantageous in that, since the members in the set are sorted by time as a score (score), the retry time can be used as a score to sort the state records in the preset state table, thereby enabling the retry consuming instance to read the state records in a more reasonable order.
This step may be performed, for example, by the message processing device or by the first consuming instance.
Wherein an instance of the plurality of retry consumption instances other than the first retry consumption instance becomes a candidate retry consumption instance.
It should be noted that the execution timing of step 204 may be anywhere before step 207, and is only described here as an illustrative example.
In the case where the first process fails, if the remaining retry number is blank or the retry time is blank, the maximum remaining retry number and the next retry time may be updated when the first process fails.
And step 207, reading a state record corresponding to the first message from the preset state table through the first retry consuming instance, and consuming the first message from the kafka and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure, the current time is greater than or equal to the corresponding retry time, and the remaining retry number is greater than or equal to 1.
Illustratively, when a transaction is retried, the first message may still be added to the asynchronous processing queue, waiting for the transaction.
In the embodiment of the invention, when the state record is updated, if the update failure is detected, the corresponding update content is written into the memory cache. In this step, the state record corresponding to the first message is read from the preset state table by the first retry consuming instance, and the latest update content corresponding to the first message is read from the memory cache.
When the processing state corresponding to the first message is in processing and the processing service identifier corresponding to the first message is different from the service identifier corresponding to the first retry consumption instance, indicating that restarting or master changing occurs, consuming the first message from Kafka through the first retry consumption instance according to the identifier information corresponding to the first message and restarting service processing; when the processing state corresponding to the first message is in processing, and the processing service identifier corresponding to the first message is the same as the service identifier corresponding to the first retry consumption instance, it may be considered that the first message is still in processing, the retry time in the state record corresponding to the first message may be updated according to the preset retry time rule, and then the next state record is continuously read.
This step may be performed, for example, before retrying the transaction, i.e., before adding the first message to the asynchronous processing queue.
And step 210, deleting the state record corresponding to the first message.
The message processing method provided by the embodiment of the invention only records the message id of the message consumed from Kafka into the state table, then processes the message asynchronously, and when the retry fails, the message content of the specified message id can be consumed from a consumer to Kafka according to the message id, so that the extra storage space is greatly reduced, the problem of chain lengthening of message consumption under normal conditions is avoided, Codis is more highly available, and the problem of unavailability caused by network abnormity between the process and the Codis can be avoided by correcting the Codis state table by using the process cache.
In order to further explain the technical scheme provided by the embodiment of the invention, the following is divided into three parts for specific description. The three parts are respectively a normal consumption flow, a retry consumption flow and an asynchronous business processing flow.
Fig. 3 is a schematic diagram of a normal consumption process according to an embodiment of the present invention, where multiple instances are simultaneously started and domination is selected through Codis competition. The main example starts the flow shown in fig. 3:
① check qps restriction and concurrency restriction of message processing, if none is restricted, consume one message from Kafka, check if it is repeated, and consume the next message if it is repeated.
② writes the status "message id + processing service sid + of the message as in-process + number of remaining retries + time of next retries" into the Codis status table sortedset, and submits offset to Kafka server.
③ to asynchronous business processes (i.e., data sources for asynchronous business process flows)
The messages consumed from Kafka have only their message id entered in the Codis status table. When the retry fails, the consumer can consume the message content of the message id from Kafka, so that the huge flow and the extra storage space caused by writing the message content into Redis can be greatly reduced, and the Codis is higher in availability than the Redis.
Fig. 4 is a schematic diagram of a retry consuming process according to an embodiment of the present invention, where multiple instances are simultaneously started and owner selection is performed through Codis competition. The main example starts the flow shown in fig. 4:
① checks the qps restriction and concurrency restriction of message processing, if none is restricted, reads the message to be retried from Codis, fetches 1 record with retry time less than or equal to the current time, if the process cache also has records, then amends the state, then processes a. if the state is in processing and the processing service sid is current service sid, then updates its next retry time, then goes back to ①, b. if the state is in processing and the processing service sid |, is current service sid, it is considered to be restarted/replaced, then goes to flow ②, c. if the state is processing failure, then goes to flow ②.
② retry consuming group by Kafka, seek to the position of message id, and read the message content corresponding to the message id from Kafka.
③, updating the state table, the state of 'message id + processing service sid + is in-process + number of remaining retries + time of next retries', if the updating fails, writing the updated content into the memory.
④ to asynchronous business processes (i.e., data sources for the asynchronous business process flows of the process)
When restarting/switching the master for retry, the message content corresponding to the message id which is not processed by the consumer until Kafka consumption can be taken up, and the situation of repeated consumption does not exist.
Fig. 5 is a schematic diagram of an asynchronous service processing flow provided in an embodiment of the present invention, as shown in fig. 5:
① out of the asynchronous queue for transaction processing (if the message is restarted without being taken out of the asynchronous queue, a retry is initiated by part ① of the retry consumption flow according to rule b)
② update state table, if the process is successful, delete the message id from the state table, if the process is failed and the number of remaining retries is not 0, update the codis state table, update the message id 'state as process failure, next retry time', if the process is failed and the number of remaining retries is 0, perform exception handling (such as pushing to the monitoring center, pushing downstream, etc.), also consider that the final process is obtained, delete the message id from the state table (if the message is restarted in the asynchronous process, initiate retry by ① part of the retry consuming process according to b rule)
By adopting the scheme, consumption, processing and retry can be asynchronously carried out, the consumption delay problem can be effectively solved, and the message processing efficiency is improved.
Fig. 6 is a block diagram of a message processing apparatus according to an embodiment of the present invention, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in a computer device, and may perform message processing by executing a message processing method. As shown in fig. 6, the apparatus includes:
a state record adding module 601, configured to consume a first message from a distributed message system through a first consumption instance, and add a state record corresponding to the first message in a preset state table in a memory storage system, where the state record includes identification information and a processing state corresponding to the message;
a status record updating module 602, configured to update a processing status in a status record corresponding to the first message to a processing failure when it is detected that the service processing corresponding to the first message fails;
a service processing retry module 603, configured to read a state record corresponding to the first message from the preset state table by using a first retry consumption instance, and consume the first message from the distributed message system and retry service processing according to the identification information corresponding to the first message when it is determined that the processing state corresponding to the first message is processing failure.
The message processing apparatus provided in the embodiment of the present invention consumes a first message from a distributed message system by using a first consumption instance, and adds a state record corresponding to the first message in a preset state table in a memory storage system, where the state record includes identification information and a processing state corresponding to the message, and when a service processing failure corresponding to the first message is detected, updates the processing state in the state record corresponding to the first message to a processing failure, reads the state record corresponding to the first message from the preset state table by using a first retry consumption instance, and consumes the first message from the distributed message system and retries the service processing according to the identification information in the state record corresponding to the first message. By adopting the technical scheme, the corresponding identification information of the message consumed from the distributed message system is recorded into the state table of the memory storage system, and if the service processing of the message fails, the asynchronous consumption retry can be performed by the consumer according to the identification information in the state table, so that the storage space is saved, the repeated consumption is avoided, and the message processing efficiency is improved.
In some embodiments, the apparatus may further comprise:
and the first state record deleting module is used for deleting the state record corresponding to the first message when the service processing corresponding to the first message is detected to be successful.
In some embodiments, the status record further includes a retry time corresponding to the message; the apparatus may further comprise:
a retry time updating module, configured to update a retry time in a status record corresponding to the first message according to a preset retry time rule while consuming the first message from the distributed message system and retrying service processing according to the identifier information corresponding to the first message;
correspondingly, the traffic processing retry module is further configured to:
after the first message is consumed from the distributed message system according to the identification information corresponding to the first message and service processing is retried, if service processing is retried, reading the state record corresponding to the first message from the preset state table again through the first retry consumption instance, and consuming the first message from the distributed message system and retrying service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to retry time corresponding to the first message.
In some embodiments, the status record further includes a remaining number of retries corresponding to the message; the apparatus may further comprise:
a retry number updating module, configured to update a remaining retry number in a status record corresponding to the first message according to a preset retry number rule while consuming the first message from the distributed message system according to the identifier information corresponding to the first message and retrying service processing;
correspondingly, the traffic processing retry module is further configured to:
after the consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes: and if the retry service processing fails, reading the state record corresponding to the first message from the preset state table again through the first retry consumption instance, and consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined as the processing failure and the remaining retry number corresponding to the first message is greater than or equal to 1.
In some embodiments, the apparatus further comprises:
and the second state record deleting module is used for performing corresponding exception handling and deleting the state record corresponding to the first message if the current remaining retry number is 0 after updating the remaining retry number in the state record corresponding to the first message according to a preset retry number rule.
In some embodiments, the apparatus may further comprise:
a cache write-in module, configured to, when updating a status record corresponding to the first message, if an update failure is detected, write corresponding update content into a memory cache;
correspondingly, reading the state record corresponding to the first message from the preset state table by the first retry consuming instance, including:
and reading a state record corresponding to the first message from the preset state table through a first retry consumption instance, reading the latest updated content corresponding to the first message from the memory cache, and correcting the state record corresponding to the first message according to the latest updated content when the latest updated content is inconsistent with the state record corresponding to the first message.
In some embodiments, the apparatus further comprises:
a first instance starting module, configured to start multiple consuming instances simultaneously before the first message is consumed from the distributed message system by the first consuming instance, and determine, by a contention mechanism corresponding to the memory storage system, the first consuming instance as a main instance, where an instance other than the first consuming instance among the multiple consuming instances becomes a candidate consuming instance.
In some embodiments, the apparatus further comprises:
a second instance starting module, configured to start multiple retry consuming instances simultaneously before the state record corresponding to the first message is read from the preset state table by the first retry consuming instance, and determine, by using a contention mechanism corresponding to the memory storage system, a first retry consuming instance as a main instance, where an instance other than the first retry consuming instance among the multiple retry consuming instances becomes a candidate retry consuming instance.
In some embodiments, the status record further includes a processing service identification; the device also includes:
the service processing retry module is further configured to: and reading a state record corresponding to a second message from the preset state table through a first retry consuming instance, and consuming the second message from the distributed message system and restarting service processing through the first retry consuming instance according to identification information corresponding to the second message when a processing state corresponding to the second message is in processing and a processing service identification corresponding to the second message is different from a service identification corresponding to the first retry consuming instance.
In some embodiments, the apparatus further comprises:
the asynchronous queue adding module is configured to, after adding the state record corresponding to the first message in a preset state table in the memory storage system, further include: and adding the first message into an asynchronous processing queue to wait for service processing.
In some embodiments, the distributed messaging system is Kafka.
In some embodiments, the memory storage system is a Redis or a Codis.
Embodiments of the present invention also provide a storage medium containing computer-executable instructions, which are used to execute the message processing method provided by the embodiments of the present invention when executed by a computer processor.
Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDORAM, Lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations, such as in different computer systems that are connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
The embodiment of the invention provides computer equipment, wherein the message processing device provided by the embodiment of the invention can be integrated in the computer equipment. Fig. 7 is a block diagram of a computer device according to an embodiment of the present invention. The computer device 700 comprises a memory 701, a processor 702 and a computer program stored on the memory 701 and operable on the processor 702, wherein the processor 702 implements the message processing method provided by the embodiment of the invention when executing the computer program.
The message processing apparatus, the storage medium, and the computer device provided in the foregoing embodiments may execute the message processing method provided in any embodiment of the present invention, and have corresponding functional modules and advantageous effects for executing the method. For technical details that are not described in detail in the above embodiments, reference may be made to a message processing method provided in any embodiment of the present invention.
Note that the above is only a preferred embodiment of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in more detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the claims.
Claims (15)
1. A message processing method, comprising:
consuming a first message from a distributed message system through a first consumption instance, and adding a state record corresponding to the first message in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message;
when the business processing failure corresponding to the first message is detected, updating the processing state in the state record corresponding to the first message into the processing failure;
and reading a state record corresponding to the first message from the preset state table through a first retry consumption instance, and consuming the first message from the distributed message system and retrying service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure.
2. The method of claim 1, further comprising:
and deleting the state record corresponding to the first message when the service processing corresponding to the first message is detected to be successful.
3. The method of claim 1, wherein the status record further includes a retry time corresponding to the message; when the consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes:
updating the retry time in the state record corresponding to the first message according to a preset retry time rule;
correspondingly, after consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes:
and if the retry service processing fails, reading the state record corresponding to the first message from the preset state table again through the first retry consumption instance, and consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined to be processing failure and the current time is greater than or equal to the retry time corresponding to the first message.
4. The method of claim 1, wherein the status record further includes a remaining number of retries corresponding to the message; when the consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes:
updating the residual retry times in the state record corresponding to the first message according to a preset retry time rule;
correspondingly, after consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message, the method further includes:
and if the retry service processing fails, reading the state record corresponding to the first message from the preset state table again through the first retry consumption instance, and consuming the first message from the distributed message system and retrying the service processing according to the identification information corresponding to the first message under the condition that the processing state corresponding to the first message is determined as the processing failure and the remaining retry number corresponding to the first message is greater than or equal to 1.
5. The method according to claim 4, further comprising, after updating the remaining retry number in the status record corresponding to the first message according to a preset retry number rule:
and if the current residual retry number is 0, performing corresponding exception handling, and deleting the state record corresponding to the first message.
6. The method of any of claims 3-5, further comprising:
when the state record corresponding to the first message is updated, if the update failure is detected, writing the corresponding update content into a memory cache;
correspondingly, reading the state record corresponding to the first message from the preset state table by the first retry consuming instance, including:
and reading a state record corresponding to the first message from the preset state table through a first retry consumption instance, reading the latest updated content corresponding to the first message from the memory cache, and correcting the state record corresponding to the first message according to the latest updated content when the latest updated content is inconsistent with the state record corresponding to the first message.
7. The method of claim 1, further comprising, prior to said consuming a first message from a distributed messaging system by a first consuming instance:
and simultaneously starting a plurality of consumption instances, determining a first consumption instance serving as a main instance through a competition mechanism corresponding to the memory storage system, wherein the instances except the first consumption instance in the plurality of consumption instances become candidate consumption instances.
8. The method of claim 1, further comprising, before the reading the state record corresponding to the first message from the preset state table by the first retry consuming instance, the following steps:
and simultaneously starting a plurality of retry consumption instances, determining a first retry consumption instance serving as a main instance through a competition mechanism corresponding to the memory storage system, wherein the instances except the first retry consumption instance in the retry consumption instances become candidate retry consumption instances.
9. The method of claim 1, wherein the status record further comprises a processing service identifier; the method further comprises the following steps:
and reading a state record corresponding to a second message from the preset state table through a first retry consuming instance, and consuming the second message from the distributed message system and restarting service processing through the first retry consuming instance according to identification information corresponding to the second message when a processing state corresponding to the second message is in processing and a processing service identification corresponding to the second message is different from a service identification corresponding to the first retry consuming instance.
10. The method according to claim 1, further comprising, after adding the status record corresponding to the first message to a preset status table in the memory storage system:
and adding the first message into an asynchronous processing queue to wait for service processing.
11. The method of claim 1, wherein the distributed messaging system is Kafka.
12. The method of claim 1, wherein the memory storage system is Redis or Codis.
13. A message processing apparatus, comprising:
the state record adding module is used for consuming a first message from the distributed message system through a first consumption instance and adding a state record corresponding to the first message in a preset state table in a memory storage system, wherein the state record comprises identification information and a processing state corresponding to the message;
a status record updating module, configured to update a processing status in a status record corresponding to the first message to a processing failure when it is detected that the service processing corresponding to the first message fails;
and the service processing retry module is configured to read a state record corresponding to the first message from the preset state table by using a first retry consumption instance, and consume the first message from the distributed message system and retry service processing according to the identification information corresponding to the first message when it is determined that the processing state corresponding to the first message is processing failure.
14. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-12.
15. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1-12 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911295625.4A CN110941502B (en) | 2019-12-16 | 2019-12-16 | Message processing method, device, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911295625.4A CN110941502B (en) | 2019-12-16 | 2019-12-16 | Message processing method, device, storage medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110941502A true CN110941502A (en) | 2020-03-31 |
| CN110941502B CN110941502B (en) | 2023-06-23 |
Family
ID=69911472
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911295625.4A Active CN110941502B (en) | 2019-12-16 | 2019-12-16 | Message processing method, device, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110941502B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111522672A (en) * | 2020-04-24 | 2020-08-11 | 北京思特奇信息技术股份有限公司 | Method and system for repeated consumption of fusing data |
| CN111884769A (en) * | 2020-08-06 | 2020-11-03 | 中国银行股份有限公司 | Message synchronization method and device |
| CN111949497A (en) * | 2020-06-23 | 2020-11-17 | 贝壳技术有限公司 | Message queue system and message processing method based on message queue system |
| CN112463407A (en) * | 2020-11-25 | 2021-03-09 | 北京神州数字科技有限公司 | Message transmission and consumption method |
| CN112559217A (en) * | 2020-12-22 | 2021-03-26 | 鲁班(北京)电子商务科技有限公司 | Message reliability consumption idempotent processing method and processing device |
| CN112732460A (en) * | 2021-01-05 | 2021-04-30 | 卓望数码技术（深圳）有限公司 | Data state management method, system and storage medium |
| CN113452602A (en) * | 2021-06-21 | 2021-09-28 | 网易（杭州）网络有限公司 | Message transmission method and device, electronic equipment and storage medium |
| CN113608896A (en) * | 2021-08-12 | 2021-11-05 | 重庆紫光华山智安科技有限公司 | Method, system, medium and terminal for dynamically switching data stream |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108647083A (en) * | 2018-04-28 | 2018-10-12 | 北京京东金融科技控股有限公司 | Task executing method, device, system, electronic equipment and computer-readable medium |
| US20180375783A1 (en) * | 2017-06-27 | 2018-12-27 | Atlassian Pty Ltd | Retry handling in messaging queues |
| CN109451032A (en) * | 2018-11-20 | 2019-03-08 | 上海联寓智能科技有限公司 | A kind of messaging system |
| CN110532066A (en) * | 2018-05-23 | 2019-12-03 | 中国移动通信集团浙江有限公司 | A kind of realization method and system of transaction consistency |
-
2019
- 2019-12-16 CN CN201911295625.4A patent/CN110941502B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180375783A1 (en) * | 2017-06-27 | 2018-12-27 | Atlassian Pty Ltd | Retry handling in messaging queues |
| CN108647083A (en) * | 2018-04-28 | 2018-10-12 | 北京京东金融科技控股有限公司 | Task executing method, device, system, electronic equipment and computer-readable medium |
| CN110532066A (en) * | 2018-05-23 | 2019-12-03 | 中国移动通信集团浙江有限公司 | A kind of realization method and system of transaction consistency |
| CN109451032A (en) * | 2018-11-20 | 2019-03-08 | 上海联寓智能科技有限公司 | A kind of messaging system |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111522672A (en) * | 2020-04-24 | 2020-08-11 | 北京思特奇信息技术股份有限公司 | Method and system for repeated consumption of fusing data |
| CN111522672B (en) * | 2020-04-24 | 2023-02-28 | 北京思特奇信息技术股份有限公司 | Method and system for repeated consumption of fusing data |
| CN111949497A (en) * | 2020-06-23 | 2020-11-17 | 贝壳技术有限公司 | Message queue system and message processing method based on message queue system |
| CN111949497B (en) * | 2020-06-23 | 2024-04-12 | 贝壳技术有限公司 | Message queue system and message processing method based on same |
| CN111884769A (en) * | 2020-08-06 | 2020-11-03 | 中国银行股份有限公司 | Message synchronization method and device |
| CN111884769B (en) * | 2020-08-06 | 2023-10-27 | 中国银行股份有限公司 | Message synchronization method and device |
| CN112463407A (en) * | 2020-11-25 | 2021-03-09 | 北京神州数字科技有限公司 | Message transmission and consumption method |
| CN112559217A (en) * | 2020-12-22 | 2021-03-26 | 鲁班(北京)电子商务科技有限公司 | Message reliability consumption idempotent processing method and processing device |
| CN112732460A (en) * | 2021-01-05 | 2021-04-30 | 卓望数码技术（深圳）有限公司 | Data state management method, system and storage medium |
| CN113452602A (en) * | 2021-06-21 | 2021-09-28 | 网易（杭州）网络有限公司 | Message transmission method and device, electronic equipment and storage medium |
| CN113608896A (en) * | 2021-08-12 | 2021-11-05 | 重庆紫光华山智安科技有限公司 | Method, system, medium and terminal for dynamically switching data stream |
| CN113608896B (en) * | 2021-08-12 | 2023-09-08 | 重庆紫光华山智安科技有限公司 | Method, system, medium and terminal for dynamically switching data streams |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110941502B (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110941502B (en) | Message processing method, device, storage medium and equipment | |
| JP6362685B2 (en) | Replication method, program, and apparatus for online hot standby database | |
| US9251021B2 (en) | Asynchronous replication in a distributed storage environment | |
| US6018746A (en) | System and method for managing recovery information in a transaction processing system | |
| EP2474919B1 (en) | System and method for data replication between heterogeneous databases | |
| US8001091B2 (en) | Apparatus, system, and method for hierarchical rollback of business operations | |
| US20090063807A1 (en) | Data redistribution in shared nothing architecture | |
| CN103092903A (en) | Database Log Parallelization | |
| KR20090055608A (en) | Data processing system and method of handling requests | |
| CN110727539A (en) | Method and system for processing exception in batch processing task and electronic equipment | |
| US20130318059A1 (en) | Transfer of data from transactional data sources to partitioned databases in restartable environment | |
| US8156084B2 (en) | Transfer of data from positional data sources to partitioned databases in restartable environments | |
| CN112181723A (en) | Financial disaster recovery method and device, storage medium and electronic equipment | |
| US20230315713A1 (en) | Operation request processing method, apparatus, device, readable storage medium, and system | |
| CN113779149A (en) | Message processing method and device, electronic equipment and readable storage medium | |
| US6944618B2 (en) | Method, computer program product, and system for unloading a hierarchical database utilizing segment specific selection criteria | |
| CN113254278B (en) | Method and system for processing internet request | |
| US7673302B1 (en) | System and method for managing messages in a queuing system | |
| CN113687935A (en) | Cloud native storage scheduling mode based on super-fusion design | |
| JPH11353284A (en) | Job re-executing method | |
| CN117112311B (en) | I/O driven data recovery method, system and device | |
| US20240118982A1 (en) | Early Database Transaction Visibility | |
| CN116775228A (en) | Data processing method and device, electronic equipment and medium | |
| CN113254528A (en) | Implementation method of high-availability database system and related equipment | |
| CN117675185A (en) | Cryptographic protocol transaction management method, medium and device based on transaction message |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231008 Address after: 31a, 15th floor, building 30, maple commercial city, bangrang Road, Brazil Patentee after: Baiguoyuan Technology (Singapore) Co.,Ltd. Address before: 511400 floor 5-13, West Tower, building C, 274 Xingtai Road, Shiqiao street, Panyu District, Guangzhou City, Guangdong Province Patentee before: GUANGZHOU BAIGUOYUAN INFORMATION TECHNOLOGY Co.,Ltd. |