CN111367628A

CN111367628A - Distributed transaction processing method and device, message producer and consumer system

Info

Publication number: CN111367628A
Application number: CN202010146490.1A
Authority: CN
Inventors: 钱峰
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2020-03-05
Filing date: 2020-03-05
Publication date: 2020-07-03
Anticipated expiration: 2040-03-05
Also published as: CN111367628B

Abstract

The invention provides a distributed transaction processing method and device, a message producer and a consumer system, wherein the method comprises the following steps: when the message producer system receives the distributed transaction triggering instruction, processing the transaction in the message producer system; while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table; after the transaction is successful, sending a first processing result message to a message consumer system; the message consumer system processes the affairs in the message consumer system according to the first processing result message and sends the processing result feedback information to the message producer system; the message producer system updates the local event table according to the feedback information; when the whole distributed transaction processing process is abnormal, executing retry processing; and executing the batch compensation processing when the execution of the step of the retry processing fails. The technical scheme realizes the final strictly consistent distributed transaction processing.

Description

Distributed transaction processing method and device, message producer and consumer system

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a distributed transaction processing method and apparatus, and a message producer and consumer system.

Background

With the development of micro-service architecture, more and more calls between systems are required. The micro-service advocates splitting the complex single application into a plurality of services with simple functions and loose coupling, so that the development difficulty can be reduced, the expansibility can be enhanced, and the quick development is facilitated. At present, more and more developers advocate that discussion and practice of micro services are started in huge internet industries, open source communities and the like, but after system micro services are realized, a function which looks simple is provided, a plurality of services may need to be called inside and a plurality of databases are operated to realize the function, the problem of distributed transactions called by the services becomes very outstanding, at present, a general scheme is not provided for well solving the transaction problem generated by the micro services, the distributed transactions become the biggest obstacle of the micro services and are the most challenging technical problem, and consistency models of distributed transaction processing data can be divided into the following three types:

1. strong consistency: after the data is updated successfully, the data in all the copies are consistent at any time, and the data updating is generally realized in a synchronous mode.

2. Weak consistency: after a successful data update, the system does not commit to read the most recently written value immediately, nor for how long.

3. Final consistency: in one form of weak consistency, after a successful data update, the system does not commit to returning the most recently written value immediately, but ensures that the value of the last update operation will eventually be returned.

The distributed transactions that are currently the mainstream in the industry focus mainly on the implementation of strong consistency and ultimate consistency.

The distributed transaction processing typical scheme mainly comprises the following steps:

1. two-stage submission;

TCC compensation mode;

3. asynchronous assurance based on event tables;

4. best effort notification.

Wherein a two-phase commit and TCC compensation mode is used to achieve strong consistency, and event table based asynchronous assurance and best effort notification are used to achieve final consistency. However, distributed processing schemes based on asynchronous assurance of event tables and best effort notification exist: the business side (message generating side) has heavy processing burden, and the transactions can not be finally consistent due to the situations of message loss and the like.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The embodiment of the invention provides a distributed transaction processing method, which is used for realizing the final strictly consistent distributed transaction processing and comprises the following steps:

when the message producer system receives the distributed transaction triggering instruction, processing the transaction in the message producer system; while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table; after the transaction processing in the message producer system is successful, sending a first processing result message to the message consumer system;

the message consumer system executes the following transaction processing and feedback steps: processing the affairs in the message consumer system according to the first processing result message, and sending processing result feedback information to the message producer system;

the message producer system updates the local event list according to the processing result feedback information;

when the whole distributed transaction processing process is abnormal, executing the following retry processing steps:

the message producer system polls the local event table at regular time, and resends the data which is failed to be processed in the local event table or is not successfully processed in a preset time period through the preset maximum retry times and retry intervals;

when the message consumer system receives the retransmitted data, the transaction processing and feedback steps are executed to obtain feedback information of the retransmission result;

the message producer system updates the local event list according to the feedback information of the reprocessing result;

wherein, when the step of retrying processing fails to be executed, the following steps of batch compensation processing are executed:

after the message producer system is in a preset time period, reading a local event table, taking out messages which cannot be successfully sent when the upper limit of the retry times is reached, forming a batch of texts to be checked, and sending the texts to a message consumer system;

the message consumer system processes the affairs in the message consumer system according to the batch texts to be checked, and the processing result is formed into batch feedback texts to be returned to the message producer system;

and the message producer system updates the local event table according to the batch feedback text.

The embodiment of the invention also provides a distributed transaction processing method, which is applied to a message producer system and used for realizing the final strictly consistent distributed transaction processing, and the method comprises the following steps:

processing the affairs in the message producer system when receiving the distributed affair processing triggering instruction;

while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table;

after the transaction processing in the message producer system is successful, sending a first processing result message to the message consumer system;

updating the local event table according to the processing result feedback information;

polling the local event table at regular time, and retransmitting data which is failed to be processed or is not processed successfully within a preset time period in the local event table through the preset maximum retry times and retry intervals;

updating the local event table according to the feedback information of the reprocessing result;

after a preset time period, reading a local event table, taking out messages which can not be successfully sent after reaching the upper limit of the retry times, forming a batch of texts to be checked, and sending the texts to a message consumer system;

and updating the local event table according to the batch feedback text.

The embodiment of the invention also provides a processing method of the distributed transaction, which is applied to a message consumer system and used for realizing the final strictly consistent distributed transaction processing, and the method comprises the following steps:

receiving a first processing result message;

the steps of performing the following transactions: processing the transaction in the message consumer system according to the first processing result message;

the following feedback steps are executed: sending the processing result feedback information to a message producer system;

when the whole distributed transaction processing process is abnormal, executing the following retry processing steps: when receiving the retransmitted data, executing the transaction processing and feedback steps to obtain feedback information of the retransmission result;

wherein, when the step of retrying processing fails to be executed, the following steps of batch compensation processing are executed: and processing the affairs in the message consumer system according to the batch texts to be checked, and forming batch feedback texts from the processing results to return to the message producer system.

The embodiment of the present invention further provides a distributed transaction processing apparatus, configured to implement final strictly consistent distributed transaction processing, where the apparatus includes:

the message producer system is used for processing the affairs in the message producer system when receiving the distributed affair processing triggering instruction; while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table; after the transaction processing in the message producer system is successful, sending a first processing result message to the message consumer system; when the whole distributed transaction processing process is abnormal, executing the following retry processing steps: polling the local event table at regular time, and retransmitting data which is failed to be processed or is not processed successfully within a preset time period in the local event table through the preset maximum retry times and retry intervals; updating the local event table according to the feedback information of the reprocessing result; when the step of retrying the processing fails to be executed, the following steps of batch compensation processing are executed: after a preset time period, reading a local event table, taking out messages which can not be successfully sent after reaching the upper limit of the retry times, forming a batch of texts to be checked, and sending the texts to a message consumer system; updating a local event table according to the batch feedback texts;

the message consumer system is used for executing the following transaction processing and feedback steps: processing the affairs in the message consumer system according to the first processing result message, and sending processing result feedback information to the message producer system; when the whole distributed transaction processing process is abnormal, executing the following retry processing steps: when receiving the retransmitted data, executing the transaction processing and feedback steps to obtain feedback information of the retransmission result; when the step of retrying the processing fails to be executed, the following steps of batch compensation processing are executed: and processing the affairs in the message consumer system according to the batch texts to be checked, and forming batch feedback texts from the processing results to return to the message producer system.

The embodiment of the invention also provides a distributed transaction processing message producer system, which is used for realizing the final strictly consistent distributed transaction processing and comprises the following components:

the transaction processing unit is used for processing the transaction in the message producer system when receiving the distributed transaction processing triggering instruction;

the write unit is used for writing all data related to the distributed transaction processing into the local event table while processing the distributed transaction;

the first sending unit is used for sending a first processing result message to the message consumer system after the transaction processing in the message producer system is successful;

the updating unit is used for updating the local event table according to the processing result feedback information;

the system also comprises a first retry processing unit, which is used for executing the following retry processing steps when the whole distributed transaction processing process is abnormal:

wherein, also include the first batch compensation processing unit, is used in when the step execution of the retry treatment fails, carry out the following steps of batch compensation treatment:

and updating the local event table according to the batch feedback text.

The embodiment of the invention also provides a message consumption system for processing the distributed transaction, which is used for realizing the final strictly consistent distributed transaction processing and comprises the following components:

a second receiving unit, configured to receive the first processing result message;

a processing unit for executing the following transaction processing steps: processing the transaction in the message consumer system according to the first processing result message;

a second sending unit, configured to perform the following feedback steps: sending the processing result feedback information to a message producer system;

the system also comprises a second retry processing unit, which is used for executing the following retry processing steps when the whole distributed transaction processing process is abnormal: when receiving the retransmitted data, executing the transaction processing and feedback steps to obtain feedback information of the retransmission result;

wherein, also include the second batch compensation processing unit, is used in when the step execution of the retry treatment fails, carry out the following steps of batch compensation treatment: and processing the affairs in the message consumer system according to the batch texts to be checked, and forming batch feedback texts from the processing results to return to the message producer system.

The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the processor implements the processing method of the distributed transaction.

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program for executing the processing method of the distributed transaction is stored in the computer-readable storage medium.

The technical scheme provided by the embodiment of the invention comprises the following steps: when the message producer system receives the distributed transaction triggering instruction, processing the transaction in the message producer system; while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table; after the transaction is successful, sending a first processing result message to a message consumer system; the message consumer system processes the affairs in the message consumer system according to the first processing result message and sends the processing result feedback information to the message producer system; the message producer system updates the local event table according to the feedback information; when the whole distributed transaction processing process is abnormal, executing retry processing; and executing the batch compensation processing when the execution of the step of the retry processing fails. The technical scheme realizes the final strictly consistent distributed transaction processing.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of a prior art scheme for asynchronous assurance based on an event table;

FIG. 2 is a schematic diagram of the processing scheme of a distributed transaction in an embodiment of the invention;

FIG. 3 is a flow chart of a method for processing a distributed transaction according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a distributed transaction processing scheme in accordance with yet another embodiment of the invention;

FIG. 5 is a flow chart illustrating a distributed transaction processing method applied to a message producer system in an embodiment of the present invention;

FIG. 6 is a flow chart illustrating a distributed transaction processing method applied to a message consumer system according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a distributed transaction processing apparatus according to an embodiment of the present invention;

FIG. 8 is a block diagram of a message producer system for processing distributed transactions in an embodiment of the invention;

fig. 9 is a schematic structural diagram of a message consumer system for processing distributed transactions in the embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Before describing the embodiments of the present invention, first, technical names related to the embodiments of the present invention will be described.

1. Transaction: a transaction is a transaction that provides a mechanism to incorporate a series of operations into an indivisible execution unit, and all operations that make up the transaction can only commit if all operations can execute normally, and if any one of the operations fails to execute, the transaction will roll back. Briefly, a transaction provides a "do nothing, do All nothing" mechanism.

2. The transaction compensation mechanism: in any forward transaction operation in the transaction chain, there must be a reversible transaction that fully complies with the rollback rules.

3. Idempotent: briefly, the business operations support retries without adverse effects.

Next, the inventor finds technical problems of the existing distributed transaction processing scheme based on asynchronous assurance and best effort notification of the event table, and proposes a concept of the embodiment scheme of the present invention.

1. First, an asynchronous assurance scheme based on an event table is introduced.

The inventor finds that: the core idea of the asynchronous guarantee scheme based on the event table is to split a distributed transaction into local transactions for processing, and the implementation details are shown in fig. 1. As shown in fig. 1, the basic idea of this scheme is:

as can be seen from the flow chart shown in fig. 1, in the embodiment of the present invention,

firstly, the message producer needs to build an additional message table and record the message sending status. Message lists and business data are submitted in a transaction, i.e. they are located in a database. The message may then be sent via the MQ to the consumer of the message. If the message transmission fails, a retry transmission is performed.

The message consumer, in turn, needs to process the message and complete its business logic. If the local transaction is successful, indicating that the transaction has been successful, then the execution will be retried if the transaction fails. If the service fails, a service compensation message can be sent to the production party to inform the production party of rollback and other operations.

Then, the production side regularly scans the local message table and sends the messages which are not processed or failed again.

According to the scheme, the DB message table needs to be designed, and meanwhile, a background task is needed to continuously scan the local messages. Resulting in additional burden on the business party to the processing of messages and the business logic coupling.

2. Next, a best effort notification scheme is introduced.

The best effort notification scheme is the least demanding of the distributed transactions, also implemented through message middleware, and differs from the previous asynchronous guaranteed type operation in that there is no guarantee link between the message sender and the message bus, and in addition, after the message is delivered to the consumer, the transaction is allowed to end normally after the maximum number of retries is reached. I.e. the final transaction cannot succeed and is not fully guaranteed.

Through the above comparative description, it can be seen that the schemes of asynchronous assurance and best effort notification based on the event table both have the problem that the transaction cannot be finally agreed due to the loss of the message, which is the key problem to be solved by the present invention.

It can be seen from the above description that, currently, the mainstream distributed transaction in the industry mainly focuses on implementation of strong consistency and final consistency, and a scheme for implementing a distributed transaction of final strict consistency is lacking, and the embodiment of the present invention provides a solution for implementing a distributed transaction of final strict consistency aiming at the problem, that is, the present invention provides a novel solution by combining two schemes of asynchronous assurance and best effort notification based on an event table with a goal of implementing final strict consistency.

Fig. 2 is a schematic diagram of a processing scheme of a distributed transaction in an embodiment of the present invention, which combines two schemes of asynchronous assurance and best effort notification based on an event table, and guarantees final strict consistency of the distributed transaction by using a local event table, message notification and batch mode, where the specific implementation scheme shown in fig. 2 is as follows (assuming that the system one and the system two implement the distributed transaction):

(1) the system completes local transaction processing by a service module (a message producer system);

(2) writing an event table locally in the same transaction as the transaction (1) and recording all data required by the distributed transaction;

(3) when the system finishes the transaction, the system sends a message to a message bus;

(4) a message receiving module of a system II (a consumer system) receives the message;

(5) performing message analysis, idempotent and time sequence processing;

(6) completing the transaction processing in the second system;

(7) the second system sends the processing result to the first system through a message bus;

(8) the system receives a processing result feedback message of the system II;

(9) and the system updates the local event table and finishes the normal process.

If the event table in the system is not updated successfully due to an exception occurring in the whole transaction processing process, a message retransmission mechanism is needed to ensure, specifically:

(1) the message retransmission module of the system polls the event table at regular time, and attempts to send the data which fails to be processed or is not processed successfully for a long time in the event table to the message bus.

(2) The retry is carried out for multiple times by configuring the maximum retry number and the retry interval algorithm to ensure the correct sending of the message to the maximum.

(3) And the second system still performs processing from the (4) th step in the normal flow to complete message processing and feedback, and the first system receives a feedback result to update the event table.

In an extreme case, if the consistency requirement of the transaction cannot be completed through message retransmission, the synchronization process is completed in a batch mode, that is:

(1) and the batch compensation module of the system reads the local event table, extracts the messages which can not be successfully processed when reaching the upper limit of the retry times, forms batch texts and sends the batch texts to the batch check module of the system II.

(2) And a batch checking module of the second system receives the texts to be checked of the second system, performs power and time sequence judgment one by one, forms processing results into feedback texts after transaction processing in the second system is completed, and returns the feedback texts to the second system.

(3) And after receiving the batch feedback text, a batch compensation module of the system updates a local event table to complete the whole processing flow.

(4) The scheduling interval of the batch module can be adjusted according to the service requirement.

In summary, currently, distributed transactions with the goal of achieving final consistency basically adopt asynchronous assurance or best effort notification based on an event table, and since the inventor finds technical problems existing in the two schemes, the embodiment of the present invention provides a new scheme of distributed transaction processing with the goal of achieving final strict consistency. The processing scheme of the distributed transaction is described in detail below.

Fig. 3 is a flowchart illustrating a processing method of a distributed transaction according to an embodiment of the present invention, and as shown in fig. 3, the method includes the following steps:

step 101: when the message producer system receives the distributed transaction triggering instruction, processing the transaction in the message producer system; while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table; after the transaction processing in the message producer system is successful, sending a first processing result message to the message consumer system;

step 102: the message consumer system executes the following transaction processing and feedback steps: processing the affairs in the message consumer system according to the first processing result message, and sending processing result feedback information to the message producer system;

step 103: the message producer system updates the local event list according to the processing result feedback information;

step 104: when the whole distributed transaction processing process is abnormal, executing the following retry processing steps:

step 105: wherein, when the step of retrying processing fails to be executed, the following steps of batch compensation processing are executed:

In one embodiment, processing a transaction within the message consumer system based on the first process result message, and sending second process result feedback information to the message producer system may include:

and according to the event identification and the event occurrence time in the first processing result message, after analysis, idempotent, time sequence and concurrency control are carried out, processing of local transactions in the message consumer system is completed, and processing result feedback information is sent to the message producer system.

In specific implementation, the embodiment of the invention performs analysis, idempotent (idempotent), timing and concurrency control processing in a mode of serial number + timestamp in a message, and has low system performance cost.

In one embodiment, the local event table may include: event identification, event occurrence time, processing state, next retry time and all service data information;

the first processing result message may include: event identification, event occurrence time and transaction state;

the processing result feedback information may include: event identification, event occurrence time, and transaction status.

In one embodiment, the first processing result message may further include: event message retransmission times, event message next retransmission time and each service data information;

the processing result feedback information may further include: the retransmission times of the event message, the next retransmission time of the event message and the processing result.

To facilitate an understanding of how the invention may be practiced, reference is now made to FIG. 4 for illustration.

The embodiment of the invention realizes the final strict consistency of the multi-system transactions through the message bus, wherein a message retransmission (retry processing step) and a batch compensation mechanism (batch compensation processing step) are respectively introduced to the abnormal condition, and the detailed description is given below on the specific scheme.

First, a processing mechanism of a transaction initiator (a processing method of a distributed transaction of a message producer system) is introduced.

In specific implementation, in order to ensure the consistency of transactions among multiple systems, a transaction initiating terminal (a message producer system, a service party and a service terminal) needs to put actions of writing self service data and writing a local event table into one transaction for processing, and after the transaction is successfully submitted, a corresponding message (a first processing result message) is sent to a message bus; and if the transaction fails, executing a rollback operation and feeding back the failure of the transaction.

The table structure of the local event table and related information may be as shown in table 1 below.

TABLE 1

After the transaction is successful, the transaction initiator needs to send a message to the message bus (the first processing result message) (in the embodiment of the present invention, the message transmission between the message producer system and the message consumer system can be sent in a bus manner), and the specific message format can be as shown in table 2 below:

serial number	Data item	Data type	Optional (M)/optional (O)
				1	eventId	Event ID	M
2	eventTime	Time of occurrence of event	M
				3	status	Status of transaction	M
4	retryTimes	Number of event message retransmissions	O
				5	nextRetryTime	Time of next retransmission of event message	O
6	busiData1	Service data 1	O
				7	…	…	O
8	busiDataN	Service data N	O

TABLE 2

Secondly, a processing method of distributed transaction of the message consumer system, namely an implementation mechanism of the message receiving module in fig. 2 and fig. 4.

In specific implementation, the message (the first processing result message) is acquired and then analyzed, whether the message exists or not is judged according to the EVENT _ ID (a form of the EVENT identifier), and if the message does not exist, relevant transaction processing in the system (the message consumer system) is carried out; if the message exists, the time sequence (the event serial number + the event occurrence time) needs to be judged, and if the occurrence time of the message is more advanced than the occurrence time of the message, the message can be ignored; if the occurrence time of the message is later, the operation of adding, deleting and modifying the data in the system is carried out, and the section of description is an example of the processing of completing the processing of the local transaction in the message consumer system after the analysis, the idempotent, the time sequence and the concurrency control are carried out according to the event identifier and the event occurrence time in the first processing result message.

Taking the distributed transaction of the transaction system and the to-be-handled system in the enterprise online banking as an example (see fig. 4), after the customer modifies the transaction data in the transaction system (i.e., the system i, the message producer system, the service provider, and the service provider), the number of to-be-processed transactions according to the state of the corresponding operator needs to be represented in the to-be-handled system (i.e., the system ii and the message consumer system) finally, and for such a case, the specific data adding, deleting and modifying operations are as follows:

1) if the received transaction state is in-transit (draft, to-be-modified, to-be-rechecked, to-be-authorized), subtracting 1 from the number of relevant operators in the to-be-handled summary table according to the relationship maintained in the current [ to-be-handled transaction and operator relationship table ]; deleting the related data of the transaction in the current (to-do transaction and operator relation table); changing the state and the timestamp of the transaction in the (to-do transaction list) and adding the state and the timestamp according to a new operator relationship; in the summary table to be dealt with, the number of strokes of the corresponding state of the relevant operator is added by 1 according to the latest state and the operator relationship.

2) If the received transaction state is a bank end state (in bank processing, transaction success, transaction failure and the like), the state and the timestamp of the transaction in the to-do transaction table are changed, the number of strokes of the corresponding state of the relevant operator in the to-do summary table is reduced by 1 according to the relationship maintained in the current to-do transaction and operator relationship table, and finally the relevant data of the transaction in the current to-do transaction and operator relationship table are deleted.

Here, when the message concurrent processing is performed, serial synchronization processing is performed for each EVENT stream number (EVENT _ ID). After the transaction processing in the system is completed, the processing result is fed back to the message bus in a message form.

In a specific implementation, the message format may be as shown in table 3 below:

serial number	Data item	Data type	Optional (M)/optional (O)
				1	eventId	Event ID	M
2	eventTime	Time of occurrence of event	M
				3	status	Status of transaction	M
4	retryTimes	Number of event message retransmissions	O
				5	nextRetryTime	Time of next retransmission of event message	O
6	dealStatus	Processing the results	M

TABLE 3

Next, the processing logic of the message result receiving module in the message producer system of FIGS. 2 and 4 is introduced.

In specific implementation, after receiving the message processing result, the event table processing state is updated after analysis.

Fourthly, next, a processing mechanism of the message retransmission module, namely a retry processing step is introduced.

In specific implementation, a polling service end (an event table) tries to write the message bus again for an event which is not successful in time-out, and if the event is still not successful after a plurality of attempts, the state of the polling service end (the event table) is updated to be a state to be checked.

The configuration of the retransmission times and the transmission intervals is involved, the transmission intervals need to be sequentially increased under the general condition, and the interval algorithm can be adjusted according to the needs.

And fifthly, next, a processing mechanism of the message batch compensation module, namely a step of batch compensation processing, and an execution method of a first batch compensation processing unit (namely the batch compensation module in fig. 2 and 4) are introduced.

The method comprises two parts of batch check text sending and feedback result text receiving.

Batch verification text sending function:

and (4) polling a service end (an event table), and generating a batch text sending system II for the event data to be checked.

Batch verification result receiving function:

and receiving batch texts of the check results, and updating the state of the affair terminal (an event table) to be successful.

The format of the batch text file and the content can be defined as follows:

the text to be checked (batch text to be checked) sent by the service end:

file name: XXX _ yyyymmddhh24miss.dat;

wherein XXX is a system identifier;

the file content definition may be:

in specific implementation, the check result feedback text (batch feedback text) may be:

XXX_feedback_YYYYMMDDHH24MISS.dat；

file name: yyyymmddhh24miss.dat;

wherein XXX is a system identifier;

the file content definition may be:

that is, in one embodiment, the text to be checked may include: file identification (which may be a file name), system (message producer system) identification, file content; the feedback text may include: file identification (which may be a file name), system (message consumer system) identification, file content.

Next, a processing mechanism of the message batch verification module, namely a step of batch compensation processing, and an execution method of the second batch compensation processing unit (i.e. the batch verification module in fig. 2 and 4) are introduced.

In specific implementation, the service modules send batch texts to the system II through the message compensation module for the EVENTs which are not fed back after being retransmitted overtime, the system II needs to check the texts to be checked one by one after receiving the texts to be checked, whether the messages exist or not is judged according to the EVENT serial number EVENT _ ID, and relevant transaction processing in the system is carried out if the messages do not exist; if the message exists, the time sequence (the event serial number + the event occurrence time) needs to be judged, and if the occurrence time of the message is more advanced than the occurrence time of the message, the message can be ignored; if the message occurs later, the operation of adding, deleting and modifying data in the system is carried out. The specific processing logic is substantially identical to the message receiving module. After the relevant data is checked, a result feedback text is finally organized and sent to the message batch compensation module.

In specific implementation, in the embodiment of the present invention, the first system may be a message producer system, and the second system may be a message consumer system. Each system may have the functionality of a message producer system and the functionality of a message consumer system.

As shown in fig. 4, the above embodiment is described by taking the first system as a transaction system and the second system as a to-do system as an example, but the first system and the second system may also be other systems. Taking a to-do center in an implemented transaction bank as an example, a transfer system and the to-do center are two independent systems and have respective databases, and after a client submits entrustment, transfer authorization, authorization return, authorization rejection and other operations in the transfer system, the to-do center needs to synchronously update the to-do data of an operator, namely, final data consistency is realized. In the specific implementation, according to the description of the invention, firstly, in the same transaction of submitting operation such as delegation, transfer authorization, authorization return, authorization rejection and the like, the transfer system additionally writes a to-do message into a to-do event table in the local database, sends the message to a message bus after the completion of the transaction, finally receives the message by a to-do center for data updating, sends an updating result to the transfer system after the updating is completed, and updates the local event table by the transfer system to complete normal logic processing; if the whole normal logic processing process is abnormal, the message compensation module retransmits the data for N times, and the data is not successfully retransmitted, and finally the batch check module finally ensures the data consistency in a text mode. In the prior art, the final consistency cannot be guaranteed only by sending the message, for example, on the premise that a message bus has a long-time fault, the embodiment of the invention adds message retransmission (a retry processing step) and batch check (a batch compensation processing step) on the basis, and ensures the final consistency requirement under the abnormal condition of distributed transaction processing.

In summary, from the perspective of those skilled in the art, it is necessary to clean the boundaries of each transaction, and a transaction initiator or a transaction coordinator performs transaction concatenation to ensure consistency of transactions between systems through online message notification, message retransmission mechanism, and bulk compensation mechanism.

The essence of the technical scheme provided by the embodiment of the invention is that: the technical problems of the conventional scheme based on asynchronous assurance and best effort notification of the event table in the industry are found, the problem that the transaction cannot be finally consistent due to the loss of the message is solved, and online message notification, a message retransmission mechanism and a batch compensation mechanism in the whole process are most important for the invention.

Based on the same inventive concept, the embodiment of the present invention further provides a processing method for distributed transactions applied to a message producer system, as described in the following embodiments. As the principle of the solution to the problem of the distributed transaction processing method applied to the message producer system is similar to that of the distributed transaction processing method described above, the implementation of the distributed transaction processing method applied to the message producer system may refer to the implementation of the distributed transaction processing method described above, and repeated details are not described again. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 5 is a flowchart illustrating a processing method of a distributed transaction applied to a message producer system in an embodiment of the present invention, as shown in fig. 5, the method includes the following steps:

step 201: processing the affairs in the message producer system when receiving the distributed affair processing triggering instruction;

step 202: while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table;

step 203: after the transaction processing in the message producer system is successful, sending a first processing result message to the message consumer system;

step 204: updating the local event table according to the processing result feedback information;

step 205: when the whole distributed transaction processing process is abnormal, executing the following retry processing steps:

step 206: wherein, when the step of retrying processing fails to be executed, the following steps of batch compensation processing are executed:

and updating the local event table according to the batch feedback text.

In a specific implementation, before step 201, a step of receiving a distributed transaction triggering instruction may be further included.

Based on the same inventive concept, the embodiment of the present invention further provides a processing method of a distributed transaction applied to a message consumer system, as described in the following embodiments. As the principle of the solution to the problem of the distributed transaction processing method applied to the message consumer system is similar to the above-described first distributed transaction processing method, the implementation of the distributed transaction processing method applied to the message consumer system may refer to the implementation of the above-described first distributed transaction processing method, and repeated details are not described again. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 6 is a flowchart illustrating a processing method of a distributed transaction applied to a message consumer system in an embodiment of the present invention, as shown in fig. 6, the method includes the following steps:

step 301: receiving a first processing result message;

step 302: the steps of performing the following transactions: processing the transaction in the message consumer system according to the first processing result message;

step 303: the following feedback steps are executed: sending the processing result feedback information to a message producer system;

step 304: when the whole distributed transaction processing process is abnormal, executing the following retry processing steps: when receiving the retransmitted data, executing the transaction processing and feedback steps to obtain feedback information of the retransmission result;

step 305: wherein, when the step of retrying processing fails to be executed, the following steps of batch compensation processing are executed: and processing the affairs in the message consumer system according to the batch texts to be checked, and forming batch feedback texts from the processing results to return to the message producer system.

In one embodiment, processing a transaction within the message consumer system based on the first process result message, and sending process result feedback information to the message producer system may include: and according to the event identification and the event occurrence time in the first processing result message, after analysis, idempotent, time sequence and concurrency control are carried out, processing of local transactions in the message consumer system is completed, and processing result feedback information is sent to the message producer system.

Based on the same inventive concept, the embodiment of the present invention further provides a processing apparatus for distributed transactions applied to a message consumer system, as described in the following embodiments. As the principle of the device for processing distributed transactions applied to the message consumer system to solve the problem is similar to the method for processing distributed transactions described above, the implementation of the device for processing distributed transactions applied to the message consumer system may refer to the implementation of the method for processing distributed transactions described above, and repeated details are not described herein. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 7 is a schematic structural diagram of a distributed transaction processing apparatus in an embodiment of the present invention, and as shown in fig. 7, the apparatus includes:

the message producer system 01 is used for processing the affairs in the message producer system when receiving the distributed affair processing triggering instruction; while processing the distributed transaction, writing all data related to processing the distributed transaction into a local event table; after the transaction processing in the message producer system is successful, sending a first processing result message to the message consumer system; when the whole distributed transaction processing process is abnormal, executing the following retry processing steps: polling the local event table at regular time, and retransmitting data which is failed to be processed or is not processed successfully within a preset time period in the local event table through the preset maximum retry times and retry intervals; updating the local event table according to the feedback information of the reprocessing result; when the step of retrying the processing fails to be executed, the following steps of batch compensation processing are executed: after a preset time period, reading a local event table, taking out messages which can not be successfully sent after reaching the upper limit of the retry times, forming a batch of texts to be checked, and sending the texts to a message consumer system; updating a local event table according to the batch feedback texts;

a message consumer system 02 for executing the following transaction processing and feedback steps: processing the affairs in the message consumer system according to the first processing result message, and sending processing result feedback information to the message producer system; when the whole distributed transaction processing process is abnormal, executing the following retry processing steps: when receiving the retransmitted data, executing the transaction processing and feedback steps to obtain feedback information of the retransmission result; when the step of retrying the processing fails to be executed, the following steps of batch compensation processing are executed: and processing the affairs in the message consumer system according to the batch texts to be checked, and forming batch feedback texts from the processing results to return to the message producer system.

In one embodiment, the message consumer system is specifically configured to: executing the following transaction processing and feedback steps: and according to the event identification and the event occurrence time in the first processing result message, after analysis, idempotent, time sequence and concurrency control are carried out, processing of local transactions in the message consumer system is completed, and processing result feedback information is sent to the message producer system.

Based on the same inventive concept, the embodiment of the present invention further provides a system for processing a message producer of a distributed transaction, as described in the following embodiments. Because the principle of the distributed transaction message processing producer system for solving the problem is similar to the above first introduced distributed transaction processing method, the implementation of the distributed transaction message processing producer system can refer to the implementation of the above first introduced distributed transaction processing method, and repeated details are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 8 is a schematic structural diagram of a message producer system for processing distributed transactions in an embodiment of the present invention, and as shown in fig. 8, the message producer system includes:

the transaction processing unit 011 is used for processing the transaction in the message producer system when receiving the distributed transaction processing triggering instruction;

a writing unit 012 configured to write each data related to processing the distributed transaction into the local event table while processing the distributed transaction;

a first sending unit 013, configured to send a first processing result message to the message consumer system after the transaction processing in the message producer system is successful;

an updating unit 014, configured to update the local event table according to the processing result feedback information;

the system further includes a first retry processing unit 015, configured to, when an exception occurs in the entire distributed transaction processing process, perform the following retry processing steps:

wherein, it also includes a first batch compensation processing unit 016, for executing the following steps when the retry step fails to execute:

and updating the local event table according to the batch feedback text.

In a specific implementation, the distributed transaction processing system may further include a first receiving unit, configured to receive the distributed transaction processing triggering instruction.

Based on the same inventive concept, the embodiment of the present invention further provides a message consumer system for processing distributed transactions, as described in the following embodiments. Because the principle of the distributed transaction message consumer system for solving the problem is similar to the above-described first distributed transaction processing method, the implementation of the distributed transaction message consumer system for processing the message may refer to the implementation of the above-described first distributed transaction processing method, and repeated details are not described again. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 9 is a schematic structural diagram of a message consumer system for processing distributed transactions in an embodiment of the present invention, and as shown in fig. 9, the message consumer system includes:

a second receiving unit 021, configured to receive the first processing result message;

a processing unit 022 configured to perform the following steps of the transaction: processing the transaction in the message consumer system according to the first processing result message;

a second sending unit 023, configured to perform the following feedback steps: sending the processing result feedback information to a message producer system;

wherein, it also includes a second retry processing unit 024, which is used for executing the following retry processing steps when the whole distributed transaction processing process is abnormal: when receiving the retransmitted data, executing the transaction processing and feedback steps to obtain feedback information of the retransmission result;

wherein, the system further comprises a second batch compensation processing unit 025, configured to execute the following steps of batch compensation processing when the execution of the step of retrying processing fails: and processing the affairs in the message consumer system according to the batch texts to be checked, and forming batch feedback texts from the processing results to return to the message producer system.

In one embodiment, the processing unit is specifically configured to: and according to the event identification and the event occurrence time in the first processing result message, performing analysis, idempotent, time sequence and concurrency control processing, and finishing the processing of local transactions in the message consumer system.

In specific implementation, fig. 2 is also an architecture diagram of a product (a distributed transaction processing apparatus), and as shown in fig. 2, the product may include a service module, a message receiving module, a message result receiving module, a message retransmitting module, a batch compensation module, and a batch verification module.

The connection between each module is as shown in fig. 2, firstly, the service module initiates the distributed transaction processing, while completing the transaction processing in the system, it needs to write the event table at the same time, after the local transaction is successful, it sends the message (the first processing result message) to the message receiving module, after the message receiving module completes the analysis, power, time sequence and concurrency control processing of the message, it completes the processing of the local transaction in the system two, after the transaction is completed, it sends the processing result (the processing result feedback message) to the message result receiving module through the message bus, and after the analysis, it updates the local event table to complete the normal flow of the distributed transaction. If the normal flow is not successfully completed, the message retransmission module is required to perform message retransmission, the specified retry times are still unsuccessful, finally, message compensation can be initiated in a batch text mode through a batch compensation module (a message compensation module), after the batch verification module receives a batch compensation text (a batch text to be verified), analysis, idempotent and time sequence processing are performed, then the local transaction processing is completed, a result feedback text (a batch feedback text) is generated and sent to the batch compensation module, and finally, the batch compensation module updates a local event table to complete the final transaction consistency.

In a specific implementation, the service module may include the transaction processing unit, the write unit, and the first sending unit, for example, "1. write service data", "2. write local schedule", and "3. send message" in fig. 2. The message receiving module may include the second receiving unit, the processing unit, and the second sending unit, such as "4. receive message", "5. message parsing, idempotent, timing processing", "6. local transaction", "7. send result notification message" in fig. 2. The message result receiving module may include the above-described updating unit, such as "8. received message" and "9. update event status" shown in fig. 2. The message retransmission module in fig. 2 may include the first retry processing unit and the second retry processing unit described above. The batch compensation module of fig. 2 may be the first batch compensation processing unit described above, and the batch verification module of fig. 2 may be the second batch compensation processing unit described above. The function of the various parts in fig. 4 is described with reference to fig. 2.

To facilitate an understanding of how embodiments of the present invention may be practiced, reference is now made to the description in the preceding paragraph.

The method combines online and batch modes to realize the final consistency requirement of the distributed transaction, and is specifically completed by matching a service module (service initiating module), a message receiving module, a message result receiving module, a message retransmitting module, a message compensating module and a message checking module.

1. Writing a local event table while the service module completes the local transaction, and sending a message bus in a message mode after the local event table is completed;

2. after the message receiving module receives the message, the message receiving module completes local transaction processing after analysis, idempotent and time sequence processing, and a processing result is sent to a message bus through the message;

3. the message result receiving module receives the processing result message, and updates the local event table after analysis;

4. the message in the event table which is not successful in the specified time period is tried to be retransmitted to the message bus by the message retransmitting module;

5. finally, a batch compensation module generates a text to be checked and sends a message batch checking module for the message in the event table which still cannot be successfully retransmitted by the message;

6. after receiving the text, the message batch checking module feeds back the processing result to the message compensation module (batch checking module) in a batch text mode after performing power sequence and local transaction processing;

7. and after receiving the check result feedback text, the message compensation module analyzes and updates the event table to complete the compensation processing of the distributed transaction.

In summary, to meet the final consistency requirement of the distributed transaction, the recording is performed by means of a local data table (local event table), the message sending is performed by means of a message queue, and the message sending and the batch compensation are performed to ensure, so that the final strict consistency distributed transaction processing is realized.

Therefore, the embodiment of the invention realizes a novel distributed transaction through a series of mechanisms and module series connection. The advantages are as follows:

1. the development intrusiveness is relatively low, and the development only needs to record an event table and send messages;

2. the method is improved on the basis of the traditional asynchronous assurance and best effort notification scheme based on the event table, achieves the final strict consistency of distributed transactions, and can be applied to more business scenarios.

The technical scheme provided by the embodiment of the invention has the beneficial technical effects that: a final strict consistent distributed transaction is achieved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes may be made to the embodiment of the present invention by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing a distributed transaction, comprising:

2. The method for processing distributed transaction as claimed in claim 1, wherein processing the transaction in the message consumer system according to the first processing result message, and sending the second processing result feedback information to the message producer system, comprises:

3. The method of processing a distributed transaction of claim 1, wherein the local event table comprises: event identification, event occurrence time, processing state, next retry time and all service data information;

the first processing result message includes: event identification, event occurrence time and transaction state;

the processing result feedback information includes: event identification, event occurrence time, and transaction status.

4. The method of processing a distributed transaction of claim 3, wherein the first processing result message further comprises: event message retransmission times, event message next retransmission time and each service data information;

the processing result feedback information further includes: the retransmission times of the event message, the next retransmission time of the event message and the processing result.

5. A distributed transaction processing method applied to a message producer system is characterized by comprising the following steps:

and updating the local event table according to the batch feedback text.

6. A distributed transaction processing method applied to a message consumer system is characterized by comprising the following steps:

receiving a first processing result message;

7. The method for processing distributed transaction as claimed in claim 6, wherein processing the transaction in the message consumer system according to the first processing result message, and sending the processing result feedback information to the message producer system, comprises: and according to the event identification and the event occurrence time in the first processing result message, after analysis, idempotent, time sequence and concurrency control are carried out, processing of local transactions in the message consumer system is completed, and processing result feedback information is sent to the message producer system.

8. An apparatus for processing a distributed transaction, comprising:

9. The distributed transaction processing apparatus of claim 8, wherein the message consumer system is specifically configured to: executing the following transaction processing and feedback steps: and according to the event identification and the event occurrence time in the first processing result message, after analysis, idempotent, time sequence and concurrency control are carried out, processing of local transactions in the message consumer system is completed, and processing result feedback information is sent to the message producer system.

10. A distributed transaction message producer system, comprising:

and updating the local event table according to the batch feedback text.

11. A message consumer system for processing distributed transactions, comprising:

12. The message consumer system of claim 11, wherein the processing unit is specifically configured to: and according to the event identification and the event occurrence time in the first processing result message, performing analysis, idempotent, time sequence and concurrency control processing, and finishing the processing of local transactions in the message consumer system.

13. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when executing the computer program.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 7.