CN115086414A - Message processing method and device, storage medium and electronic equipment - Google Patents
Message processing method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN115086414A CN115086414A CN202210505177.1A CN202210505177A CN115086414A CN 115086414 A CN115086414 A CN 115086414A CN 202210505177 A CN202210505177 A CN 202210505177A CN 115086414 A CN115086414 A CN 115086414A
- Authority
- CN
- China
- Prior art keywords
- message
- retry
- retransmitted
- attribute information
- retransmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/07—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail characterised by the inclusion of specific contents
- H04L51/18—Commands or executable codes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The invention discloses a message processing method and device, a storage medium and electronic equipment. Wherein, the method comprises the following steps: storing the attribute information of each message in the message set which is currently sent in failure to a retry service database according to the received retry service request; acquiring a message set to be retransmitted which accords with a preset retransmission condition based on the attribute information; paging the message set to be retransmitted based on a snapshot paging mode to obtain a plurality of message paging subsets; and retransmitting the messages in the plurality of message page subsets. The invention solves the technical problems of poor data accuracy and consistency in the message sending process in the related technology.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a message processing method and apparatus, a storage medium, and an electronic device.
Background
In the process of sending messages by the Kafka publishing and subscribing message system, the Kafka server is abnormal due to overtime of message sending caused by network fluctuation, or the sending is failed due to abnormal producer in the process of sending messages, so that the messages can not be guaranteed to be normally sent to the Kafka server. In some situations where the message is sent only once, such as callback information of WeChat, if the message is failed to be sent in the process of sending the message to the Kafka server, the problem of data loss is caused, and the accuracy and consistency of data processing are poor. In addition, in the existing message sending architecture, when multiple services fail to process the Kafka message by themselves, the message sending is repeated, and the message resending efficiency is low.
Disclosure of Invention
The embodiment of the invention provides a message processing method and device, a storage medium and electronic equipment, which are used for at least solving the technical problems of poor data accuracy and consistency in the message sending process in the related technology.
According to an aspect of an embodiment of the present invention, there is provided a message processing method, including: storing the attribute information of each message in the message set which is currently sent in failure to a retry service database according to the received retry service request; acquiring a message set to be retransmitted which accords with a preset retransmission condition based on the attribute information; paging the message set to be retransmitted based on a snapshot paging mode to obtain a plurality of message paging subsets; and retransmitting the messages in the plurality of message page subsets.
According to another aspect of the embodiments of the present invention, there is also provided a message processing apparatus, including: a storage unit, which stores the current failed message set and the attribute information of each message in the message set to a retry service database according to the received retry service request; an obtaining unit, configured to obtain a message set to be retransmitted, which meets a preset retransmission condition, based on the attribute information; a paging unit, configured to page the message set to be retransmitted based on a snapshot paging manner, so as to obtain a plurality of message paging subsets; a retransmission unit for retransmitting the messages in the plurality of page subsets of messages.
According to still another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the message processing method through the computer program.
According to still another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above message processing method when running.
The invention can at least realize the following beneficial effects:
the invention can be applied to the field of data transmission (second level of the technical field of the patent matrix in the system) of data capacity (first level of the technical field of the patent matrix in the system). In the embodiment of the invention, the attribute information of each message in the message set which is failed to be sent currently is stored in a retry service database according to the received retry service request; acquiring a message set to be retransmitted which accords with a preset retransmission condition based on the attribute information; paging the message set to be retransmitted based on a snapshot paging mode to obtain a plurality of message paging subsets; in the method, a message set to be retransmitted which meets preset retransmission conditions is obtained through attribute information, and the message set to be retransmitted is paged based on a snapshot paging mode to obtain a plurality of message paging subsets, so that not only can a message which fails to be transmitted be accurately obtained, but also the message which fails to be transmitted is retransmitted according to the preset retransmission conditions, the data accuracy and consistency in the message transmission process are ensured, the message transmission efficiency is improved, and the technical problem that the data accuracy and consistency in the message transmission process in the related technology are poor is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a schematic diagram of an application environment of an alternative message processing method according to an embodiment of the invention;
FIG. 2 is a schematic diagram of an application environment of an alternative message processing method according to an embodiment of the invention;
FIG. 3 is a flow chart illustrating an alternative message processing method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an alternative Kafka topology according to embodiments of the present invention;
FIG. 5 is a timing diagram illustrating an alternative message processing method according to an embodiment of the present invention;
FIG. 6 is a block diagram illustrating an alternative message processing method according to an embodiment of the present invention;
FIG. 7 is a flow diagram illustrating an alternative message processing method according to an embodiment of the present invention;
FIG. 8 is a block diagram of an alternative message processing apparatus according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an alternative electronic device according to an embodiment of the invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an aspect of the embodiments of the present invention, there is provided a message processing method, as an alternative implementation manner, the message processing method may be, but is not limited to be, applied to an application environment as shown in fig. 1. The application environment comprises: the terminal equipment 102, the network 104 and the server 106 are used for human-computer interaction with the user. User 108 and terminal device 102 may perform human-computer interaction, and a message processing application is running in terminal device 102. The terminal device 102 includes a human-machine interaction screen 1022, a processor 1024, and a memory 1026. The man-machine interaction screen 1022 is used for displaying a message set of failed sending; processor 1024 is configured to obtain a retry service request. The memory 1026 is used for storing the above-mentioned message set of transmission failures.
In addition, the server 106 includes a database 1062 and a processing engine 1064, and the database 1062 is used for storing the message set of the transmission failure. The processing engine 1064 is configured to store, according to the received retry service request, attribute information of each message in the message set that is currently failed to be sent to a retry service database; acquiring a message set to be retransmitted which accords with a preset retransmission condition based on the attribute information; paging the message set to be retransmitted based on a snapshot paging mode to obtain a plurality of message paging subsets; resending messages in said plurality of message page subsets; and sending the message sending result to the client of the terminal device 102.
In one or more embodiments, the message processing method described above in this application may be applied to the application environment shown in fig. 2. As shown in fig. 2, a human-computer interaction may be performed between a user 202 and a user device 204. The user equipment 204 includes a memory 206 and a processor 208. The user equipment 204 in this embodiment may refer to, but is not limited to, performing the above-described operation performed by the terminal equipment 102 to obtain the result of the message transmission.
Optionally, the terminal device 102 and the user device 204 include, but are not limited to, a mobile phone, a tablet computer, a notebook computer, a PC, a vehicle-mounted electronic device, a wearable device, and the like, and the network 104 may include, but is not limited to, a wireless network or a wired network. Wherein, this wireless network includes: WIFI and other networks that enable wireless communication. Such wired networks may include, but are not limited to: wide area networks, metropolitan area networks, and local area networks. The server 106 may include, but is not limited to, any hardware device capable of performing computations. The server may be a single server, a server cluster composed of a plurality of servers, or a cloud server. The above is merely an example, and this is not limited in this embodiment.
In the related art, when the amount of messages required to be sent by a system is large, a Kafka message queue is selected for message transmission. Meanwhile, in order to ensure the throughput, a mechanism that an ACK (acknowledgement character) is equal to 1 is adopted to send the message, namely, only Kafka server Leader (each partition has multiple copies, wherein only one copy is used as a Leader, and the Leader is the partition currently responsible for reading and writing the data) needs to receive the producer data and is confirmed, and the message cannot be ensured to be received by the Kafka server. The specific flow is shown in fig. 4:
after receiving the request, the Producer service calls a message sending interface of a Kafka client to send a message to a Kafka server, and sends the message to a Leader of a Broker (a Kafka cluster contains one or more servers, and a server node is called the Broker).
Copy replica is then synchronized to the Follower, but if the Follower of the Partition fails, the Follower has not been copied, and the data is lost. Following the Leader, all write requests are routed through the Leader, data changes are broadcast to all the leaders, and the leaders and the follers maintain data synchronization. If the Leader fails, a new Leader is selected from the Follower. When the Follower and the Leader hang up, get stuck or are too slow to synchronize, the Leader deletes the Follower from a synchronous replication (ISR) list and recreates a Follower; here, the ISR list is a collection of all replicas synchronized with Partition leader data.
And iii, monitoring corresponding Topic by a Consumer business service, and consuming the message when the message is received.
In the related art, in the process of sending the message, the message is failed to be sent due to network fluctuation, overtime sending, exception of the Kafka server, or exception of a producer in the sending process, and the like, so that the message cannot be normally sent to the Kafka server. In some important scenes that messages are only sent once, such as callback information of WeChat, if the messages are lost in the Kafka sending process, the problem of data loss is caused, and normal use of functions is influenced. In addition, in the existing architecture system, when each service module fails to process the Kafka message by itself, a large amount of repeated work is generated, which causes waste of computing resources.
In order to solve the above technical problem, as an optional implementation manner, as shown in fig. 3, an embodiment of the present invention provides a message processing method, including the following steps:
s302, according to the received retry service request, storing the attribute information of each message in the message set which fails to be sent currently to a retry service database.
Specifically, as shown in fig. 5, in the message production service module, after receiving a request for sending a message, the Producer first attempts to send the message, and then monitors a callback (i.e., message acknowledgement) of the sent message, where the callback is a callback message that calls the sending interface of Kafka and informs the caller of the condition of the sent message, such as whether the sending is successful, a failure reason, and the like.
When the Producer receives the callback message, whether the message is successfully sent is judged, and if the message is successfully sent, the sending process of the current message is ended.
When the current message sending fails, the function provided by the spring-retry third-party library is used locally for local retry, and the message sending is completed locally without extra overhead space for storing the failed message and the message sent by retry. The method comprises the following specific steps:
1) adding @ EnableRetry annotation start retry function to the start class, adding annotated class or method, and the like, wherein EnableRetry gives the system a function of starting message sending retry.
2) Adding Retryable annotation on a sending method, setting retry times and interval time, and intercepting an exception, so that when a corresponding exception is encountered, the method with the added annotation retries the corresponding times according to the setting according to the time interval, for example, the retry times is set to be 3, the interval time is 3 seconds, and the method retries when an http child error exception request is encountered.
In the embodiment of the invention, a retry service module is arranged to process a message set which fails to be sent; when the retransmission fails for three times by using spring-retry, the http request calls an interface of the retry service module to send a failed message record, after the retry service receives the retry service request, the retry service stores the message which is sent in failure to a retry service database and records a log, wherein the log mainly stores information: the id of the message in the retry service database (mainly used for solving the problem of data duplicate storage), the transmitted topoc, the service item to which the message belongs, the index value of the message, the message content, the type of the message, whether the message needs to be retransmitted, the expiration time of the message transmission, and other information.
S304, based on the attribute information, acquiring a message set to be retransmitted which meets the preset retransmission condition.
Specifically, the preset retransmission condition includes, but is not limited to, preferentially transmitting a message with a long dwell time after a transmission failure, retransmitting a record that a current retransmission is not successful and the number of retries does not exceed a maximum threshold value nor a maximum timeout time, and determining the message set meeting the preset retransmission condition. It should be noted that, in the embodiment of the present invention, the retry service module may sequentially send messages to be retransmitted according to the configured timing duration.
S306, paging the message set to be retransmitted based on the snapshot paging mode to obtain a plurality of message paging subsets.
Specifically, the list of the message set to be retransmitted is paged, a certain amount of data is queried each time, by the technical means, the retransmission task executed at the current moment is cached to the local as a snapshot, and then small-batch data is queried by paging, so that the situation that the memory occupation is too high or the memory overflows due to too many query message records can be prevented.
S308, the messages in the message paging subsets are retransmitted.
Specifically, the retry service module calls the API interface of Kafka to send the message to be retransmitted.
In the embodiment of the invention, the attribute information of each message in the message set which is failed to be sent currently is stored in a retry service database according to the received retry service request; acquiring a message set to be retransmitted which accords with a preset retransmission condition based on the attribute information; paging the message set to be retransmitted based on a snapshot paging mode to obtain a plurality of message paging subsets; in the method, a message set to be retransmitted which meets preset retransmission conditions is obtained through attribute information, and the message set to be retransmitted is paged based on a snapshot paging mode to obtain a plurality of message paging subsets, so that not only can a message which fails to be transmitted be accurately obtained, but also the message which fails to be transmitted is retransmitted according to the preset retransmission conditions, the data accuracy and consistency in the message transmission process are ensured, the message transmission efficiency is improved, and the technical problem that the data accuracy and consistency in the message transmission process are poor in the related technology is solved.
In one or more embodiments, the obtaining of the set of messages to be retransmitted that meets a preset retransmission condition based on the attribute information includes:
determining the retransmission priority of each message in the message set which fails to be sent according to the creation time;
and acquiring a message set to be retransmitted which accords with a preset retransmission condition according to the retransmission priority and based on the attribute information.
Specifically, suppose that after the message is failed to be sent, the retry service module receives a message retry service request corresponding to the message, generates creation time of the message in a retry service database according to the time of the service request, and sets the priority with longer creation time as a high sending priority; thus, the message with longer retention time after the message transmission failure can be preferentially transmitted.
In one or more embodiments, obtaining a set of messages to be retransmitted that meets a preset retransmission condition based on the attribute information includes:
and acquiring a message set to be retransmitted which accords with the preset retransmission condition in each preset time interval.
Specifically, the retry service module in the embodiment of the present invention may send the message to be retransmitted in a polling manner continuously within a preset time interval, and each time interval may select a certain number of messages for retransmission of the message.
In one or more embodiments, obtaining a set of messages to be retransmitted that meets a preset retransmission condition based on the attribute information includes:
acquiring a target message set which is not successfully sent currently, has no retry sending times exceeding a first threshold value and does not exceed a preset overtime time;
and determining the target message set as the message set to be retransmitted.
Specifically, in the embodiment of the present invention, the stability of message sending may be controlled by configuring the first threshold, and the accuracy and consistency of message transmission are ensured by configuring the preset timeout.
In one or more embodiments, the attribute information includes a retry transmission number and a transmission status, and after the resending of the messages in the plurality of message page subsets, the method further includes:
and updating the transmission state of the first message to be transmitted when the first message in the plurality of message paging subsets is successfully transmitted.
Specifically, when the message in the message paging subset is successfully transmitted, the transmission status of the successfully transmitted message in the retry service data is updated from non-transmission to transmitted.
And when the second message in the plurality of message page subsets fails to be transmitted, increasing the number of times of retry transmission of the second message and retransmitting the second message in the next retransmission time period.
Specifically, when the transmission of a message in the message page subset fails, the number of times of retry transmission of the message with failed transmission in the retry service data is increased by 1, and the message is preferentially transmitted in the next retransmission period.
In one or more embodiments, the attribute information further includes a message retransmission unique identifier, and the increasing the number of times of retry transmission of the second message and retransmitting the second message in a next retransmission time period includes:
and increasing the number of times of retry transmission of each second message according to the message retransmission unique identifier of the second message, and retransmitting the second message in the next transmission time period.
Specifically, the message retransmission unique identifier (retry id) is used to uniquely identify the current message, the producer listens for callback messages, and when the transmission is successful, the transmission status of the transmission success message is marked as transmitted in the retry service database, and if it fails, the number of retries is increased by 1 and retransmitted in the next timing task, in the process of retransmitting the Kafka message, the retry id is configured to the request header, in the case that the consumer fails to consume the message and needs to retransmit, the consumer can obtain the retry id according to the received message request header, if the retry id is not null, the retry id is transmitted to the retry service module, when the retry service database is put in storage, if the retry id is not null, the corresponding record of the database is updated, no message needs to be added, and the repeated record of the message is prevented, so that the same message is sent again.
In one or more embodiments, the attribute information includes a retransmission flag, and the method further includes:
and updating a retransmission flag corresponding to a third message in the plurality of page subsets of messages to stop retransmission when the number of times of retry transmission of the third message exceeds a first threshold.
The producer monitors the callback message, when the transmission is successful, the transmission state of the successful transmission message is marked as transmitted in the retry service database, if the transmission is failed, the retry number is added with 1, the message is retransmitted in the next timing task, and when the retry number exceeds a certain number, the retry is not carried out any more, thereby preventing the waste of system resources.
In one or more embodiments, before storing the attribute information of the currently failed-to-send message pair column in the retry service database according to the received retry service request, the method further includes:
configuring a retry service instance on a plurality of servers; a server that meets the target requirement is selected to receive the retry service request.
Specifically, as shown in fig. 6, after the current message fails to retry for three times, the http request calls an interface of the retry service module to send a failed message record, the retry service module adopts a micro-service architecture (different from a single system, the micro-service architecture divides different functional modules of the system into different service systems), a load balancing component is used when sending the message, when the retry service configures multiple instances, such as 3 instances, 3 instances may be on one server or on different servers, and when on different servers, the problem that the message retry service is unavailable due to downtime of one of the servers may be prevented; the high availability of the message retransmission service is ensured by the technical characteristics.
In the related art, products such as a Distributed Transaction Coordinator (DTC) are business expansion based on enterprise WeChat data, and provide a better and more humanized service for clients. The system in the embodiment of the invention adopts a micro-service architecture, different modules in the system are mutually independent, and for various data integration and system integration and message communication among different modules, as the RPC synchronous processing technology is adopted, the system has a plurality of defects in performance, robustness and expandability. The asynchronous processing model based on the message adopts the non-blocking calling characteristic, the sender sends the message to the message server, and the message server forwards the message to the receiver when appropriate; the sending and receiving are asynchronous, the sender does not need to wait, the life cycles of the sender and the receiver are not required to be the same, and the sender can indirectly transmit the message to a plurality of receivers, so that the performance, the expandability and the robustness of the program are greatly improved, and the asynchronous processing model has more advantages in distributed application than a synchronous processing model.
The message middleware is used as middle layer software, provides a set of reliable and universal method for creating, sending and receiving messages in the distributed system, and realizes reliable, efficient and real-time cross-platform data transmission in the distributed system. The message middleware reduces the complexity of developing software across platforms and network protocols, shields the specific details of different operating systems and network protocols,
faced with distributed systems of increasing size and complexity. Compared with the traditional object-oriented middleware, the method has the following advantages:
adopting an asynchronous communication mode: the sender can do other work after sending the message without waiting for the receiver to respond and the receiver does not have to process the sender's request immediately after receiving the message.
Loosely coupled relationship of customer and service object lifecycle: the client process and the service object process are not required to run normally, if the request of the client is not reachable due to the crash of the service object or network failure, the client does not receive the exception, and the message middleware can ensure that the message is not lost; thus, in this system, message middleware is largely used for message delivery.
In the related art, when the amount of messages required to be sent by a system is large, a Kafka message queue is selected for message transmission. Meanwhile, in order to ensure the throughput, a mechanism that an ACK (acknowledgement character) is equal to 1 is adopted to send the message, namely, only Kafka server Leader (each partition has multiple copies, wherein only one copy is used as a Leader, and the Leader is the partition currently responsible for reading and writing the data) needs to receive the producer data and is confirmed, and the message cannot be ensured to be received by the Kafka server. The specific flow is shown in fig. 4:
after receiving the request, the Producer service calls a message sending interface of the Kafka client to send a message to a Kafka server, and sends the message to a Leader of a Broker (a Kafka cluster contains one or more servers, and a server node is called the Broker).
Copy replica is then synchronized to the Follower, but if the Follower of the Partition fails, the Follower has not been copied, and the data is lost. Following the Leader, all write requests are routed through the Leader, data changes are broadcast to all the leaders, and the leaders and the follers maintain data synchronization. If the Leader fails, a new Leader is selected from the Follower. When the Follower and the Leader hang up, are stuck or are too slow to synchronize, the Leader deletes the Follower from the (In Sync Replicas, ISR) list and recreates a Follower; here, the ISR list is a collection of all replicas synchronized with Partition leader data.
And iii, monitoring corresponding Topic by a Consumer business service, and consuming the message when the message is received.
In the related art, in the process of sending the message, the message is failed to be sent due to network fluctuation, overtime sending, exception of the Kafka server, or exception of a producer in the sending process, and the like, so that the message cannot be normally sent to the Kafka server. In some important scenes that messages are only sent once, such as callback information of WeChat, if the messages are lost in the Kafka sending process, the problem of data loss is caused, and normal use of functions is influenced. In addition, in the existing architecture system, when each service module fails to process the Kafka message by itself, a large amount of repeated work is generated, which causes waste of computing resources.
Based on the foregoing embodiment, in an application embodiment, as shown in fig. 5, the foregoing message processing method further includes:
1. in the message production service module, after receiving a request for sending a message, a Producer first attempts to send the message, and then monitors a callback (i.e. message acknowledgement) of the sent message, where the message callback is a callback message for calling a sending interface of Kafka and informs a caller of the condition of sending the message, such as whether the message is sent successfully, a failure reason, and the like.
When the Producer receives the callback message, whether the message is successfully sent is judged, and if the message is successfully sent, the current message sending process is ended.
2. When the current message sending fails, the function provided by the spring-retry third-party library is used locally for local retry, and the message sending is completed locally without extra overhead space for storing the failed message and the message sent by retry. The method comprises the following specific steps:
1) adding @ EnableRetry annotation start retry function to the start class, adding annotated class or method, and the like, the EnableRetry can endow the system with the function of start retry.
2) Adding Retryable annotation on a sending method, setting retry times and interval time, and intercepting an exception, so that when a corresponding exception is encountered, the method with the added annotation retries the corresponding times according to the setting according to the time interval, for example, the retry times are set to be 3, the interval time is 3 seconds, and the method retries when an http child error exception request is encountered.
3. When all three retries fail, the http request calls an interface of the retry service to send a failed message record, as shown in fig. 6, because a micro-service architecture (different from a single system, the micro-service architecture divides different functional modules of the system into different service systems) is adopted, a load balancing component is used during sending, and when the retry service configures a plurality of instances, for example, 3 instances, the message resending service is not unavailable due to the abnormality of some instances of the retry service during resending the message, thereby ensuring the high availability of the retry service.
4. After receiving the request, the retry service puts the record in a database and records the log, wherein the main storage information includes: retry id of the message in the retry service database, the sent topic, the service to which the message belongs, the index of the message, the message content, the type of the message, whether the message needs to be retransmitted or not, and the message sending expiration time.
5. As shown in fig. 7, the retry service module is configured with a timing task, and the timing task execution flow is as follows:
1) and setting a timing task, and inquiring retry ids of all sending failure records meeting the conditions in the current retry service database according to the creation date sequence at intervals. And acquiring the list to be sent as the message which is not successfully sent currently, does not exceed the maximum threshold value in retry number and does not exceed the maximum timeout time, and preferentially sending the longest message according to the creation time.
2) And paging the list, and inquiring a certain amount of data each time, so that the main purpose is to cache the timing task executed at the current moment as a snapshot to the local, and then paging to inquire a small amount of data, thereby preventing the memory occupation from being too high or the memory from overflowing due to too many records.
3) And the retry service module calls an API (application programming interface) of Kafka to send the message list to be retransmitted.
6. When the message to be retransmitted is successfully transmitted, the transmission state of the message to be retransmitted is marked as transmitted in a retry service database, if the message to be retransmitted is failed, the retry number is increased by 1, the message is retransmitted in the next timing task, in the process of retransmitting the Kafka message, a retry id is configured to a request head, when the message is failed to be consumed by a consumer and needs to be retransmitted, the consumer can obtain the retry id according to the received message request head, if the retry id is not null, the retry id is transmitted to the retry service, and when the message is put in storage in the retry service, if the retry id is not null, the corresponding record of the database is updated, no message needs to be added, the message is prevented from being repeatedly recorded, and the same message is retransmitted and transmitted for multiple times.
In one or more embodiments, the above message processing method includes:
using a transaction mode, carrying out message retransmission by logging and then scanning a log table at regular time, wherein the specific flow is as follows:
1. and carrying out message data persistence locally.
2. And identifying the message according to the serial number, and locally configuring a rollback method.
3. The information recorded by the message to be retransmitted is configured into an xml file, and then a log table is generated by using the xml file as a template through a third-party library (such as log4j or logback). The log state is updated locally for transactions.
4. And scanning abnormal or overtime records in the log table by the timed task, then calling an external query interface to retransmit messages, and changing the log state of the successfully requested and retransmitted messages.
5. If the external interface returns that the retransmission fails, a rollback method is called to carry out message rollback operation.
6. When the external interface realizes idempotency (idempotency is that the result of one request or multiple requests initiated by the user for the same operation is consistent, and other effects cannot be generated due to multiple clicks), message sending retry can be carried out according to the message identification.
The embodiment of the invention also has the following beneficial effects:
1. for the message which fails to be sent for the first time, the message can be automatically retransmitted in the service, and the record can be carried out in the retry service data after the continuous sending fails, so that the probability of message sending failure is reduced, and the frequency of the record of the sending failure is reduced.
2. And for the failure record, inquiring and retransmitting at regular time, recording the number of times of transmission failure, and not retrying when reaching a set threshold value, thereby preventing the system resource waste caused by the unlimited retransmission of the message.
3. For the message which fails to be sent, the id of the message is added to the request header in the retry service, so that the message is prevented from being sent for many times, multiple pieces of repeated data are recorded, and the consistency is improved.
4. The rules of the retry service are unified, all the services on the whole message system can complete the access of the retry service only by introducing the corresponding jar packet and calling an interface.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art will appreciate that the embodiments described in this specification are presently preferred and that no acts or modules are required by the invention.
According to another aspect of the embodiment of the present invention, there is also provided a message processing apparatus for implementing the message processing method. As shown in fig. 8, the apparatus includes:
a storage unit 802, which stores a message set that fails to be sent currently and attribute information of each message in the message set to a retry service database according to a received retry service request;
an obtaining unit 804, configured to obtain a message set to be retransmitted, which meets a preset retransmission condition, based on the attribute information;
a paging unit 806, configured to page the message set to be retransmitted based on a snapshot paging manner, so as to obtain a plurality of message paging subsets;
a retransmission unit 808, configured to retransmit the message in the message page subset.
In the embodiment of the invention, the attribute information of each message in the message set which is failed to be sent currently is stored in a retry service database according to the received retry service request; acquiring a message set to be retransmitted which accords with a preset retransmission condition based on the attribute information; paging the message set to be retransmitted based on a snapshot paging mode to obtain a plurality of message paging subsets; in the method, a message set to be retransmitted which meets preset retransmission conditions is obtained through attribute information, and the message set to be retransmitted is paged based on a snapshot paging mode to obtain a plurality of message paging subsets, so that not only can a message which fails to be transmitted be accurately obtained, but also the message which fails to be transmitted is retransmitted according to the preset retransmission conditions, the data accuracy and consistency in the message transmission process are ensured, the message transmission efficiency is improved, and the technical problem that the data accuracy and consistency in the message transmission process in the related technology are poor is solved.
In one or more embodiments, the attribute information includes a creation time of a message that fails to be sent in the retry service database, and the obtaining unit 804 specifically includes:
a first determining module, configured to determine, according to the creation time, a retransmission priority of each message in a message set that fails to be sent;
and the first acquisition module is used for acquiring a message set to be retransmitted which accords with a preset retransmission condition according to the retransmission priority and based on the attribute information.
In one or more embodiments, the obtaining unit 804 further includes:
and the second acquisition module is used for acquiring the message set to be retransmitted which accords with the preset retransmission condition in each preset time interval.
In one or more embodiments, the obtaining unit 804 further includes:
the third acquisition module is used for acquiring a target message set which is not successfully sent currently, does not exceed the first threshold in the retry sending times and does not exceed the preset timeout time;
and a second determining module, configured to determine the target message set as the message set to be retransmitted.
In one or more embodiments, the attribute information includes a retry transmission number and a transmission status, and the message processing apparatus further includes:
a first updating unit, configured to update a transmission status of a first message in the message page subsets to a transmitted status when the first message is successfully transmitted;
an adjusting unit, configured to increase a retry transmission number of the second message and retransmit the second message in a next retransmission time period when transmission of the second message in the plurality of message page subsets fails.
In one or more embodiments, the attribute information further includes a unique identifier for message retransmission, and the adjusting unit specifically includes:
and the adjusting module is used for increasing the retry sending times of each second message according to the message retransmission unique identifier of the second message and retransmitting the second message in the next sending time period.
In one or more embodiments, the attribute information includes a retransmission flag, and the message processing apparatus further includes:
and a second updating unit configured to update the retransmission flag corresponding to the third message to stop retransmission when the number of times of retry transmission of the third message in the plurality of message page subsets exceeds a first threshold.
In one or more embodiments, the message processing apparatus further includes:
a configuration unit configured to configure a retry service instance on a plurality of servers;
and the selection unit is used for selecting the server meeting the target requirement to receive the retry service request.
According to another aspect of the embodiment of the present invention, there is also provided an electronic device for implementing the message processing method, where the electronic device may be the terminal device or the server shown in fig. 9. The present embodiment takes the electronic device as an example for explanation. As shown in fig. 9, the electronic device comprises a memory 902 and a processor 904, the memory 902 having stored therein a computer program, the processor 904 being arranged to perform the steps of any of the above-described method embodiments by means of the computer program.
Optionally, in this embodiment, the electronic device may be located in at least one network device of a plurality of network devices of a computer network.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, according to the received retry service request, storing the attribute information of each message in the current failed message set to the retry service database;
s2, acquiring a message set to be retransmitted which accords with the preset retransmission condition based on the attribute information;
s3, paging the message set to be retransmitted based on the snap paging mode to obtain a plurality of message paging subsets;
and S4, retransmitting the messages in the plurality of message page subsets.
Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 9 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 9 is a diagram illustrating a structure of the electronic device. For example, the electronics may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 9, or have a different configuration than shown in FIG. 9.
The memory 902 may be used to store software programs and modules, such as program instructions/modules corresponding to the message processing method and apparatus in the embodiments of the present invention, and the processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, implements the message processing method described above. The memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 902 can further include memory located remotely from the processor 904, which can be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 902 may be, but is not limited to, specifically configured to store attribute information of the transmission failure message. As an example, as shown in fig. 9, the memory 902 may include, but is not limited to, a storage unit 802, an obtaining unit 804, a paging unit 806, and a retransmission unit 808 in the message processing apparatus. In addition, other module units in the message processing apparatus may also be included, but are not limited to, and are not described in detail in this example.
Optionally, the transmitting device 906 is used for receiving or sending data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 906 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 906 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
In addition, the electronic device further includes: a display 908 for displaying the processing result of the message of the transmission failure; and a connection bus 910 for connecting the respective module components in the above-described electronic apparatus.
In other embodiments, the terminal device or the server may be a node in a distributed system, where the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting a plurality of nodes through a network communication. Nodes can form a Peer-To-Peer (P2P, Peer To Peer) network, and any type of computing device, such as a server, a terminal, and other electronic devices, can become a node in the blockchain system by joining the Peer-To-Peer network.
According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. A processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the message processing method, wherein the computer program is configured to execute the steps in any of the method embodiments described above.
Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, according to the received retry service request, storing the attribute information of each message in the current failed message set to the retry service database;
s2, acquiring a message set to be retransmitted which accords with the preset retransmission condition based on the attribute information;
s3, paging the message set to be retransmitted based on the snap paging mode to obtain a plurality of message paging subsets;
and S4, retransmitting the messages in the plurality of message page subsets.
Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages and disadvantages of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be substantially or partially implemented in the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, and including instructions for causing one or more computer devices (which may be personal computers, servers, or network devices) to execute all or part of the steps of the method according to the embodiments of the present invention.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, a division of a unit is merely a division of a logic function, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that it is obvious to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and these modifications and improvements should also be considered as the protection scope of the present invention.
Claims (11)
1. A method of message processing, the method comprising:
storing the attribute information of each message in the message set which is currently sent in failure to a retry service database according to the received retry service request;
acquiring a message set to be retransmitted which accords with a preset retransmission condition based on the attribute information;
paging the message set to be retransmitted based on a snapshot paging mode to obtain a plurality of message paging subsets;
resending the messages in the plurality of paging subsets of messages.
2. The method according to claim 1, wherein the attribute information includes a creation time of a transmission failure message in the retry service database, and the obtaining a set of messages to be retransmitted that meet a preset retransmission condition based on the attribute information includes:
determining the retransmission priority of each message in the message set which fails to be sent according to the creation time;
and acquiring a message set to be retransmitted which accords with a preset retransmission condition according to the retransmission priority and based on the attribute information.
3. The method according to claim 1 or 2, wherein obtaining a set of messages to be retransmitted that meet a preset retransmission condition based on the attribute information comprises:
and acquiring a message set to be retransmitted which accords with the preset retransmission condition in each preset time interval.
4. The method according to claim 1 or 2, wherein obtaining a set of messages to be retransmitted that meet a preset retransmission condition based on the attribute information comprises:
acquiring a target message set which is not successfully sent currently, has no retry sending times exceeding a first threshold value and does not exceed a preset overtime time;
and determining the target message set as the message set to be retransmitted.
5. The method of claim 1, wherein the attribute information includes a number of retries to send and a status of sending, and wherein after the resending of the messages in the subset of the plurality of message pages, the method further comprises:
when a first message in the message paging subsets is successfully sent, updating the sending state of the first message to be sent;
when transmission of a second message in the plurality of page subsets of messages fails, increasing a number of retried transmissions of the second message and retransmitting the second message within a next retransmission time period.
6. The method of claim 5, wherein the attribute information further includes a message retransmission unique identifier, and wherein increasing the number of retried transmissions of the second message and retransmitting the second message within a next retransmission time period comprises:
and according to the message retransmission unique identification of the second message, increasing the retry transmission times of each second message and retransmitting the second message in the next transmission time period.
7. The method according to claim 5 or 6, wherein the attribute information comprises a retransmission flag, the method further comprising:
when the retry sending times of a third message in the plurality of message page subsets exceed a first threshold, updating a retransmission identifier corresponding to the third message to stop retransmission.
8. The method of claim 1, wherein before storing the attribute information of the currently failed-to-send message pair in the retry service database according to the received retry service request, the method further comprises:
configuring a retry service instance on a plurality of servers;
a server that meets the target requirement is selected to receive the retry service request.
9. A message processing apparatus, comprising:
the storage unit is used for storing a message set which fails to be sent currently and attribute information of each message in the message set to a retry service database according to a received retry service request;
an obtaining unit, configured to obtain a message set to be retransmitted, which meets a preset retransmission condition, based on the attribute information;
a paging unit, configured to page the message set to be retransmitted based on a snapshot paging manner, so as to obtain a plurality of message paging subsets;
a retransmission unit for retransmitting messages in the plurality of page subsets of messages.
10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 8 by means of the computer program.
11. A computer-readable storage medium, comprising a stored program, wherein the program when executed performs the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210505177.1A CN115086414A (en) | 2022-05-10 | 2022-05-10 | Message processing method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210505177.1A CN115086414A (en) | 2022-05-10 | 2022-05-10 | Message processing method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115086414A true CN115086414A (en) | 2022-09-20 |
Family
ID=83247048
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210505177.1A Pending CN115086414A (en) | 2022-05-10 | 2022-05-10 | Message processing method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115086414A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102804817A (en) * | 2009-06-18 | 2012-11-28 | 阿尔卡特朗讯 | Individualized Retry Configurations For Messages Having Failed Delivery |
| CN108965164A (en) * | 2017-05-17 | 2018-12-07 | 北京京东尚科信息技术有限公司 | Service request repeating method, device and readable storage medium storing program for executing based on message queue |
| CN112565418A (en) * | 2020-12-04 | 2021-03-26 | 平安科技（深圳）有限公司 | Message retransmission method and device for service cluster, computer equipment and storage medium |
| CN114168390A (en) * | 2021-12-13 | 2022-03-11 | 深圳前海环融联易信息科技服务有限公司 | Distributed consistent transaction execution method based on retry mechanism |
-
2022
- 2022-05-10 CN CN202210505177.1A patent/CN115086414A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102804817A (en) * | 2009-06-18 | 2012-11-28 | 阿尔卡特朗讯 | Individualized Retry Configurations For Messages Having Failed Delivery |
| CN108965164A (en) * | 2017-05-17 | 2018-12-07 | 北京京东尚科信息技术有限公司 | Service request repeating method, device and readable storage medium storing program for executing based on message queue |
| CN112565418A (en) * | 2020-12-04 | 2021-03-26 | 平安科技（深圳）有限公司 | Message retransmission method and device for service cluster, computer equipment and storage medium |
| CN114168390A (en) * | 2021-12-13 | 2022-03-11 | 深圳前海环融联易信息科技服务有限公司 | Distributed consistent transaction execution method based on retry mechanism |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11222043B2 (en) | System and method for determining consensus within a distributed database | |
| CN107729366B (en) | Universal multi-source heterogeneous large-scale data synchronization system | |
| US10614098B2 (en) | System and method for determining consensus within a distributed database | |
| CN110633320A (en) | Processing method, system, equipment and storage medium of distributed data service | |
| CN108712457B (en) | Method and device for adjusting dynamic load of back-end server based on Nginx reverse proxy | |
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| CN108833479B (en) | Data synchronization method and device | |
| CN110032478B (en) | Method, device and system for real-time synchronization of data of main and standby centers and storage medium | |
| CN112118315A (en) | Data processing system, method, device, electronic equipment and storage medium | |
| CN107368369B (en) | Distributed container management method and system | |
| CN112507029A (en) | Data processing system and data real-time processing method | |
| CN110099084B (en) | Method, system and computer readable medium for ensuring storage service availability | |
| CN109144748B (en) | Server, distributed server cluster and state driving method thereof | |
| CN105306585A (en) | Data synchronization method for plurality of data centers | |
| CN111464612B (en) | Method for providing stable computing service in severe environment | |
| CN101594256A (en) | Disaster recovery method, device and system | |
| US20130139178A1 (en) | Cluster management system and method | |
| CN111787058A (en) | Lightweight information subscription and push method in cross-domain virtual data space | |
| CN112417042A (en) | Method and device for processing service request | |
| CN108140035B (en) | Database replication method and device for distributed system | |
| CN111416823A (en) | Data transmission method and device | |
| CN110232053B (en) | Log processing method, related equipment and system | |
| EP3026860B1 (en) | Method and system for transmission management of full configuration synchronization between eml-nml | |
| CN110798358B (en) | Distributed service identification method and device, computer readable medium and electronic equipment | |
| CN112328560A (en) | File scheduling method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |