CN112346827A - Data consistency method and device for multi-transaction system - Google Patents
Data consistency method and device for multi-transaction system Download PDFInfo
- Publication number
- CN112346827A CN112346827A CN202011264382.0A CN202011264382A CN112346827A CN 112346827 A CN112346827 A CN 112346827A CN 202011264382 A CN202011264382 A CN 202011264382A CN 112346827 A CN112346827 A CN 112346827A
- Authority
- CN
- China
- Prior art keywords
- transaction
- module
- message
- local
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000001360 synchronised effect Effects 0.000 claims abstract description 130
- 238000012790 confirmation Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 53
- 230000000875 corresponding effect Effects 0.000 claims description 51
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 8
- 230000002547 anomalous effect Effects 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 4
- 230000002596 correlated effect Effects 0.000 claims description 3
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 description 22
- 230000009471 action Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 3
- 230000001143 conditioned effect Effects 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 2
- 239000003999 initiator Substances 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012384 transportation and delivery Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention provides a data consistency method of a multi-transaction system, for a synchronous mode, all transaction modules execute local transactions in sequence, a next transaction module is called, data after the local transactions are executed are reported to a consistency platform, the consistency platform updates data of corresponding global transactions, and if any transaction module fails to execute the local transactions, the consistency platform is informed to cancel the corresponding global transactions and inform all transaction modules in the preamble of the global transactions to synchronously cancel the corresponding local transactions. For the asynchronous mode, the transaction module sends a confirmation message to the consistency platform and executes local transaction, the consistency platform sends a message to the message queue, the consistency platform monitors the state of the message queue, and if the message queue fails, the message queue is adjusted or the sending of the message is retried. The invention also provides a data consistency device of the multi-transaction system, which is suitable for a synchronous mode or an asynchronous mode.
Description
Technical Field
The invention relates to the technical field of software, in particular to the technical field of SaaS deployed in a micro-service mode.
Background
Software as a Service (SaaS) is becoming a new development direction in the Software industry, and SaaS is widely used particularly in mobile applications. In online local life services, SaaS is becoming the mainstream configuration. For example, in a take-out service scenario, SaaS deployed in a microservice manner is often used. For local life scenarios such as take-away business, a business process may involve multiple transactions such as placing orders, locking inventory, locking coupons, generating orders, calling deliveries, initiating payments, printing documents, and the like. The transactions are respectively completed by independent transaction modules, each transaction module works simultaneously or sequentially according to certain business logic, executes corresponding transactions, exchanges data among different transaction modules and completes the whole process together in a matching way.
In this process, data consistency is important, but after a business process is initiated, data consistency must be maintained for each transaction, such as ordering, inventory, coupons, orders, deliveries, payments, and printed documents. If one link fails to execute the transaction and data synchronization is not carried out timely, data inconsistency can be caused, data confusion can be caused, the problems of missing orders, wrong inventory, wrong orders, incapability of payment, abnormal deduction, wrong documents and the like can be caused, and the user is dissatisfied.
From the perspective of transaction type analysis, the problems of processing for data consistency in the prior art include:
consistency issues of synchronization patterns: for example, the order process for taking a take-away order needs inventory service to lock inventory, the marketing service locks coupons, the order service generates orders, the payment service initiates payment, etc. A series of processes need to ensure consistency, if one ring in the middle fails, data inconsistency can be caused, so that a closed loop is difficult to form, and the method belongs to the consistency problem of a synchronous mode.
Consistency problems for asynchronous mode: after the order service completes an order, the order needs to be pushed to the printer service in a message queue mode. If the intermediate fails, the order payment is successful, but the closed loop is difficult to form in the case of being unable to be pushed to the printer. Belonging to the consistency problem of the asynchronous mode.
In summary, for the problem of transaction data consistency of SaaS in the micro service deployment environment in the prior art, the following points generally exist:
1) synchronous mode and asynchronous mode consistent transactions exist, but are scattered in each component, and a unified management platform is not used for management.
2) The asynchronous mode relies on the message queue, but cannot continue to execute when the whole message queue is down. A disaster recovery scheme for message queue downtime is lacking.
3) The execution of the affairs belongs to a black box, and related technicians are difficult to perceive the execution condition of the affairs and do not count and monitor global affairs, so that the alarm is quickly given.
Disclosure of Invention
The invention provides a scheme for solving the problem of data consistency in SaaS under micro-service deployment.
According to an embodiment of the present invention, a data consistency method for a multi-transaction system is provided, which is used for implementing data consistency between a plurality of transaction modules in a synchronous mode, and includes:
a first transaction module initiates a flow, creates a corresponding global transaction on a consistency platform, and executes a local transaction;
the first transaction module executes the local transaction, calls the next transaction module, reports the data after the local transaction is executed to the consistency platform, and the consistency platform updates the data of the corresponding global transaction;
after each transaction module executes the local transaction in sequence, calling the next transaction module, reporting data after the local transaction is executed to a consistency platform, updating the data of the corresponding global transaction by the consistency platform until all the transaction modules execute the local transaction, sequentially confirming the successful execution of the local transaction by each transaction module and informing the consistency platform of confirming the result, wherein the data of the corresponding global transaction on the consistency platform is consistent with the data after all the transaction modules execute the local transaction;
if the local transaction still fails to be executed after the retry condition is met, the consistency platform is informed, and the consistency platform cancels the corresponding global transaction and informs all transaction modules in the preamble of the global transaction to synchronously cancel the corresponding local transaction.
In one embodiment, the retry condition comprises one of: retry execution of the local transaction for a predetermined number of times, retry execution of the local transaction for a predetermined length of time, discover anomalous data, or a retry condition is enforced to be set to not retry direct retraction.
According to an embodiment of the present invention, a data consistency method for a multi-transaction system is provided, which is used for implementing data consistency between a plurality of transaction modules in an asynchronous mode, and includes:
a transaction module sends a creation message to the consistency platform and executes a local transaction;
if the transaction module successfully executes the local transaction, sending a confirmation message to the consistency platform, sending a message to the message queue by the consistency platform, and if the transaction module fails to execute the local transaction, sending a cancellation message to the consistency platform;
the coherency platform monitors the status of the message queue and, if the message queue fails, adjusts the message queue or retries sending the message.
In one embodiment, the message queue comprises a plurality of sets of message queues, the plurality of sets of message queues are endowed with different priorities, the consistency platform sends messages to the message queue with high priority, and if the message queue with high priority fails, the consistency platform switches to send messages to the message queue with low priority; if all message queues fail, the coherency platform continues to retry sending messages to each message queue until the message queue recovers.
In one embodiment, the plurality of sets of message queues includes a Kafka message queue and a Redis message queue, the Kafka message queue having a higher priority than the Redis message queue.
According to an embodiment of the present invention, a data consistency apparatus for a multi-transaction system is provided, which is configured to implement data consistency between multiple transaction modules in a synchronous mode or an asynchronous mode, and includes:
the system comprises a plurality of synchronous mode transaction modules, a plurality of synchronous mode transaction modules and a plurality of synchronous mode transaction modules, wherein the synchronous mode transaction modules execute local transactions in sequence, and the local transactions executed by the synchronous mode transaction modules are correlated;
the system comprises a plurality of asynchronous mode transaction modules, a plurality of cache modules and a plurality of cache modules, wherein the asynchronous mode transaction modules respectively execute local transactions, and the local transactions executed by the asynchronous mode transaction modules are independent;
the system comprises a consistency platform, a data processing module and a data processing module, wherein the consistency platform comprises a synchronous transaction processing module, an asynchronous message transaction module and a message trigger module, the synchronous transaction processing module is communicated with each synchronous transaction module, the asynchronous message transaction module is communicated with each asynchronous mode transaction module, and the asynchronous message transaction module is also communicated with the message trigger module;
wherein, the first synchronous transaction module informs the synchronous transaction processing module of the consistency platform to create a corresponding global transaction and execute the local transaction according to the initiation flow, the first synchronous transaction module finishes executing the local transaction, calls the next synchronous transaction module and reports the data after executing the local transaction to the synchronous transaction processing module, the synchronous transaction processing module updates the data of the corresponding global transaction, each synchronous transaction module finishes executing the local transaction in sequence, calls the next synchronous transaction module and reports the data after executing the local transaction to the synchronous transaction processing module, the synchronous transaction processing module updates the data of the corresponding global transaction until all synchronous transaction modules finish executing the local transaction, each synchronous transaction module confirms the successful execution of the local transaction in sequence and informs the consistency platform to confirm the result, the data of the corresponding global transaction of the synchronous transaction processing module is consistent with the data of all the synchronous transaction modules after the local transaction is executed;
if any one synchronous transaction module fails to execute the local transaction, the local transaction is selected to be executed again until the retry condition is met, if the local transaction still fails to be executed after the retry condition is met, the synchronous transaction processing module is informed, the synchronous transaction processing module cancels the corresponding global transaction and informs each synchronous transaction module in the preamble of the global transaction to synchronously cancel the corresponding local transaction;
each asynchronous transaction module sends a created message to an asynchronous message transaction module of the consistency platform and executes a local transaction, if the asynchronous transaction module successfully executes the local transaction, a confirmation message is sent to the asynchronous message transaction module, the asynchronous message transaction module sends a message to a message queue in the message reach module, and if the asynchronous transaction module fails to execute the local transaction, a revocation message is sent to the asynchronous message transaction module; the asynchronous message transaction module monitors the status of the message queue in the message reach module and adjusts the message queue or retries sending the message if the message queue fails.
In one embodiment, the consistency platform further comprises a global transaction statistics module, and the global transaction statistics module calculates the transaction execution success rate and the transaction execution speed in a preset period.
In one embodiment, the global transaction statistics module includes a calculation module and a monitoring module,
the calculation module calculates the transaction execution success rate and the transaction execution speed in a preset period;
the monitoring module displays the transaction execution success rate and the transaction execution speed calculated by the calculation module, and sends an alarm when the transaction execution success rate and the transaction execution speed reach the set threshold.
In one embodiment, the retry condition comprises one of: retry execution of the local transaction for a predetermined number of times, retry execution of the local transaction for a predetermined length of time, discover anomalous data, or a retry condition is enforced to be set to not retry direct retraction.
In one embodiment, the consistency platform further comprises a message configuration module, the message queue comprises a plurality of sets of message queues, the message configuration module assigns different priorities to the plurality of sets of message queues, the message reach module sends a message to the message queue with a high priority, and if the message queue with a high priority fails, the message reach module switches to send a message to the message queue with a low priority.
In one embodiment, the plurality of sets of message queues includes a Kafka message queue and a Redis message queue, and the message configuration module gives priority to the Kafka message queue over the Redis message queue.
In one embodiment, when the asynchronous message transaction module detects that all message queues in the message reach module have failed, the asynchronous message transaction module continuously retries to send messages to each message queue until the message queues recover.
The data consistency method and the device of the multi-transaction system can ensure the data consistency under the long-flow multi-transaction environment deployed in a micro-service mode.
Drawings
FIG. 1 discloses a flow diagram of a data consistency method for a multi-transaction system according to an embodiment of the invention, which is used in a synchronization mode.
FIG. 2 discloses a flow diagram of a data coherency method for a multi-transaction system according to an embodiment of the invention, which is used in asynchronous mode.
FIG. 3 is a block diagram of a data coherency mechanism for a multi-transaction system according to an embodiment of the present invention.
Detailed Description
FIG. 1 discloses a flow diagram of a data consistency method for a multi-transaction system according to an embodiment of the invention, which is used in a synchronization mode. Referring to fig. 1, the data consistency method of the multi-transaction system is used for realizing data consistency in a synchronous mode among a plurality of transaction modules, and comprises the following steps:
the first transaction module initiates a flow, creates a corresponding global transaction on the consistency platform, and executes a local transaction.
And the first transaction module executes the local transaction, calls the next transaction module, reports the data after the local transaction is executed to the consistency platform, and the consistency platform updates the data of the corresponding global transaction.
And each transaction module executes the local transaction in sequence, calls the next transaction module, reports the data after the local transaction is executed to the consistency platform, the consistency platform updates the data of the corresponding global transaction until all the transaction modules execute the local transaction, each transaction module confirms the successful execution of the local transaction in sequence and informs the consistency platform of confirming the result, and the data of the corresponding global transaction on the consistency platform is consistent with the data after all the transaction modules execute the local transaction.
In the embodiment shown in FIG. 1, three transaction modules are shown: a transaction module A, a transaction module B and a transaction module C, wherein the execution sequence of the three transaction modules is A, B, C. When each transaction module normally executes the local transaction, and the whole process is normally executed, the transaction module A is used as an initiator of the whole process and the global transaction to create a corresponding global transaction on the consistency platform P, then the transaction module A executes the local transaction, and the action of creating the global transaction is realized in a way that the transaction module A submits a transaction notice to the consistency platform P. And the transaction module A successfully executes the local transaction, calls the next transaction module B, reports the data after the local transaction is executed to the consistency platform P, and the consistency platform P updates the data of the corresponding global transaction. And the transaction module A reports the data after the local transaction execution to the action of the consistency platform P in a mode of submitting a confirmation notice. The transaction module B, C repeats the above process until all the transaction modules have executed the local transaction, and each transaction module sequentially confirms that the local transaction has been executed successfully and notifies the consistency platform of the confirmation result, and when all the transaction modules have finished confirming the local transaction, the global transaction is completed. The data of the corresponding global transaction on the coherency platform P is consistent with the data of all the transaction modules A, B, C after the local transaction has been executed. When each transaction module executes local transactions normally and the whole process is executed normally, the data on the consistency platform P can ensure consistency.
When a certain transaction module fails to execute the local transaction, according to the scheme in the prior art, because the prior art does not have the consistency platform P, the execution of the local transaction can only be selected to be continuously retried until the local transaction is successfully executed, which may cause the execution time of the whole process to be uncontrollable, and may take a long time. If the execution process is interrupted in the middle of the process due to other reasons, data inconsistency among the transaction modules can be caused.
According to the method of the present invention, when any one of the transaction modules A, B, C fails to execute the local transaction, it first tries to choose to retry to execute the local transaction until the retry condition is satisfied, and if the local transaction still fails to execute after the retry condition is satisfied, the coherency platform P is notified, and the coherency platform P cancels the corresponding global transaction and notifies the preamble transaction modules of the global transaction to synchronously cancel the corresponding local transaction. The transaction module informs the consistency platform P that the execution of the action failed is realized in a mode of submitting failure notice, and the action of informing each transaction module in the preamble of synchronously cancelling the corresponding local transaction is realized in a mode of issuing global rollback notice by the consistency platform P. Such as failure of the transaction execution module C to execute the local transaction, the transaction execution module C first attempts to retry execution of the local transaction. This retry is not conditioned on success of the retry as in the conventional technique, but if the retry is not successful even after repetition, the retry is terminated when the retry condition described below is met. The retry condition includes one of: retry execution of the local transaction for a predetermined number of times, retry execution of the local transaction for a predetermined duration, finding anomalous data (such as data errors or receipt of an artificial termination instruction), or in some aging-pursuing procedures, retry conditions may be enforced to initiate revocation directly without retry. When the retry condition is met, the transaction execution module C still does not successfully execute the local transaction, and the transaction execution module C notifies the consistency platform P, and the consistency platform P cancels the corresponding global transaction and notifies the transaction modules in the preamble of the global transaction, that is, the transaction module a and the transaction module B synchronously cancel the corresponding local transaction.
The flow chart of the data consistency method of the multi-transaction system in the synchronous mode disclosed in fig. 1 is added with serial numbers to facilitate understanding of the execution sequence of the steps in the method. If all the transaction modules can normally execute the local transaction and confirm, the execution sequence is as follows:
1. the transaction module A sends a transaction notification to the consistency platform P, and then executes the local transaction A;
2. the transaction module A calls the transaction module B;
3. the transaction module B sends a transaction notification to the consistency platform P, and then executes the local transaction B;
4. the transaction module B calls the transaction module C;
5. the transaction module C sends a transaction notification to the consistency platform P, and then executes the local transaction C;
6. the transaction module A confirms that the local transaction is successfully executed and informs the consistency platform P;
7. the transaction module B confirms that the local transaction is successfully executed and informs the consistency platform P;
8. the transaction module C confirms that the local transaction execution was successful and informs the consistency platform P.
This is the completion of the global transaction.
If an execution failure occurs in the transaction module A, B, C and retries also fail, then:
in step 6, the transaction module A confirms that the local transaction execution fails and informs the consistency platform P;
and step 9 is added, and the consistency platform P issues a global rollback notification (sends the notification to the transaction module A).
In step 7, the transaction module B confirms that the local transaction execution fails and informs the consistency platform P;
and step 10 is added, and the consistency platform P issues a global rollback notification (sends the notification to the transaction modules A and B).
In step 8, the transaction module C confirms that the local transaction execution fails and informs the consistency platform P;
step 11 is added, the coherency platform P issues a global rollback notification (issued to transaction modules A, B and C).
Therefore, in the invention, in the synchronous mode, if any transaction module fails to execute the local transaction to cause the process revocation, all other transaction modules participating in the process under the control of the consistency platform can synchronously revoke the related data, thereby ensuring that the data consistency can be still ensured when the execution fails.
FIG. 2 discloses a flow diagram of a data coherency method for a multi-transaction system according to an embodiment of the invention, which is used in asynchronous mode. Referring to fig. 2, the data consistency method of the multi-transaction system is used for realizing data consistency between a plurality of transaction modules in an asynchronous mode, and comprises the following steps:
a transaction module sends a create message to the coherency platform and performs a local transaction.
If the transaction module successfully executes the local transaction, a confirmation message is sent to the consistency platform, the consistency platform sends a message to the message queue, and if the transaction module fails to execute the local transaction, a cancellation message is sent to the consistency platform.
The coherency platform monitors the status of the message queue and, if the message queue fails, adjusts the message queue or retries sending the message.
In one embodiment, the message queue comprises multiple sets of message queues, the multiple sets of message queues are endowed with different priorities, the consistency platform sends messages to the message queue with high priority, and if the message queue with high priority fails, the consistency platform switches to sending messages to the message queue with low priority. In one embodiment, the plurality of sets of message queues includes a Kafka message queue and a Redis message queue, the Kafka message queue having a higher priority than the Redis message queue. When the Kafka message queue is normal, the Kafka message queue is used as a main message queue, and when the Kafka message queue is in failure, such as downtime, the Redis message queue is switched to be used as an alternative.
If all message queues fail, for example, when all message queues are down, the consistency platform continuously retries to send messages to each message queue until the message queues recover, so as to ensure that data is not lost. Message queues are typically operated under continuous monitoring, so that an overall downtime of all message queues typically triggers an alarm that is restarted or repaired within a short period of time, so that it is feasible for a message queue to continue resending messages to the message queue.
In the embodiment shown in FIG. 2, a transaction module D is shown. The transaction module D sends a create message to the coherency platform P and executes the local transaction. And if the transaction module D fails to execute the local transaction and does not retry successfully when the retry condition is met, a cancel message is sent to the consistency platform P, and the consistency platform P cancels the related transaction. After the transaction module D successfully executes the local transaction and sends an acknowledgement message to the consistency platform P, the consistency platform P sends a message to the message queue S. The consistency platform P monitors the status of the message queue S and, if the message queue fails, adjusts the message queue or retries sending the message. In the embodiment shown in fig. 2, two kinds of message queues are encapsulated in the message queue S: a high priority Kafka message queue and a low priority Redis message queue, the Kafka queue being the primary message queue. The Kafka queue is used when the Kafka queue is operating normally. If the Kafka queue is down, the Kafka queue is switched to the Redis queue. If both the Kafka queue and the Redis queue are down, the consistency platform P continuously retries to send the messages to the message queue S until the message queue S is recovered. After the message queue S is processed, a processing message is sent to the subsequent transaction module E. In one embodiment, transaction module D is a print transaction, while transaction module E consumes a notification transaction. Transaction module D and transaction module E are transaction modules that operate in an asynchronous mode.
In the prior art, because there is no consistency platform P, the transaction module D needs to register in the message queue S by itself and send a message to the message queue S after the local transaction is successfully executed. For the case of configuring multiple sets of message queues, such as Kafka message queue and Redis message queue, it is necessary to configure the sending function of multiple sets of messages in the transaction module D. Meanwhile, the transaction module D needs to monitor the operation condition of the message queue S, and switch the message queue in time or retry to send messages when the message queue goes down to prevent data loss. In this mode of the prior art, the transaction module D takes on too much work, both to execute local transactions and to take care of the status of the message queue and to send messages. When local transaction execution fails or a message queue goes down, data errors or loss caused by overload of a load easily occur, and global data inconsistency is caused. The invention strips the work of message sending from the transaction module D, and then places the work on the consistency platform P for uniform processing, so that the transaction module D is concentrated on executing local transactions, and the consistency platform P is concentrated on sending messages, thereby ensuring the data consistency among the transaction modules in an asynchronous mode.
FIG. 3 is a block diagram of a data coherency mechanism for a multi-transaction system according to an embodiment of the present invention. Referring to fig. 3, a data consistency apparatus of the multi-transaction system for implementing data consistency in a synchronous mode or an asynchronous mode between a plurality of transaction modules includes:
the system comprises a plurality of synchronous mode transaction modules, a plurality of synchronous mode transaction modules and a plurality of synchronous mode transaction modules, wherein the synchronous mode transaction modules execute local transactions in sequence, and the local transactions executed by the synchronous mode transaction modules are correlated.
The plurality of asynchronous mode transaction modules respectively execute local transactions, and the local transactions executed by the asynchronous mode transaction modules are independent.
The consistency platform P comprises a synchronous transaction processing module 101, an asynchronous message transaction module 102 and a message reach module 103, wherein the synchronous transaction processing module 101 is communicated with each synchronous transaction module, the asynchronous message transaction module 102 is communicated with each asynchronous mode transaction module, the asynchronous message transaction module 102 is also communicated with the message reach module 103, and a message queue is packaged in the message reach module 103.
The working process of the data consistency device is as follows: the first synchronous transaction module initiates a flow to inform the synchronous transaction processing modules of the consistency platform of creating a corresponding global transaction and executing the local transaction, the first synchronous transaction module finishes executing the local transaction, calls the next synchronous transaction module and reports the data after executing the local transaction to the synchronous transaction processing modules, the synchronous transaction processing modules update the data of the corresponding global transaction, each synchronous transaction module finishes executing the local transaction in sequence, calls the next synchronous transaction module and reports the data after executing the local transaction to the synchronous transaction processing modules, the synchronous transaction processing modules update the data of the corresponding global transaction until all synchronous transaction modules finish executing the local transaction, each synchronous transaction module confirms successful execution of the local transaction in sequence and informs the consistency platform of confirming the result, and the data of the corresponding global transaction of the synchronous transaction processing modules and all synchronous transaction modules finish executing the local transaction The later data are consistent;
if any one synchronous transaction module fails to execute the local transaction, the local transaction is selected to be executed again until the retry condition is met, if the local transaction still fails to be executed after the retry condition is met, the synchronous transaction processing module is informed, the synchronous transaction processing module cancels the corresponding global transaction and informs each synchronous transaction module in the preamble of the global transaction to synchronously cancel the corresponding local transaction;
each asynchronous transaction module sends a created message to an asynchronous message transaction module of the consistency platform and executes a local transaction, if the asynchronous transaction module successfully executes the local transaction, a confirmation message is sent to the asynchronous message transaction module, the asynchronous message transaction module sends a message to a message queue in the message reach module, and if the asynchronous transaction module fails to execute the local transaction, a revocation message is sent to the asynchronous message transaction module; the asynchronous message transaction module monitors the status of the message queue in the message reach module and adjusts the message queue or retries sending the message if the message queue fails.
In the embodiment shown in FIG. 3, three synchronized transaction modules are shown: the synchronous transaction module A, the synchronous transaction module B and the synchronous transaction module C, and the execution sequence of the three synchronous transaction modules is A, B, C. When each synchronous transaction module normally executes a local transaction and the whole process is normally executed, the synchronous transaction module a, as an initiator of the whole process and the global transaction, creates a corresponding global transaction on the synchronous transaction processing module 101 of the consistency platform P, then the synchronous transaction module a executes the local transaction, and the action of creating the global transaction is implemented in a manner that the synchronous transaction module a submits a transaction notification to the synchronous transaction processing module 101. The synchronous transaction module a successfully executes the local transaction, the synchronous transaction module a calls the next synchronous transaction module B, and reports the data after the local transaction is executed to the synchronous transaction processing module 101, and the synchronous transaction processing module 101 updates the data of the corresponding global transaction. The action of the synchronous transaction module a reporting the data after the local transaction execution to the synchronous transaction processing module 101 is implemented in a manner of submitting a confirmation notification. The synchronized transaction module B, C repeats the above process until all transaction modules have executed the local transaction, and each synchronized transaction module sequentially confirms that the local transaction has been executed successfully and notifies the consistency platform of the confirmation result, and when all the local transactions of the transaction modules have been confirmed, the global transaction is completed. The data of the corresponding global transaction at the synchronized transaction processing module 101 is consistent with the data of all the transaction modules A, B, C after the local transaction is executed. When each transaction module executes the local transaction normally and the whole process executes normally, the data on the synchronous transaction processing module 101 of the consistency platform P can ensure consistency.
When any one of the synchronized transaction modules A, B, C fails to execute the local transaction, it first tries to retry to execute the local transaction until the retry condition is satisfied, and if the local transaction still fails to execute after the retry condition is satisfied, the synchronized transaction module 101 is notified, and the synchronized transaction module 101 cancels the corresponding global transaction and notifies each synchronized transaction module in the front of the global transaction to synchronously cancel the corresponding local transaction. The synchronous transaction module notifies the synchronous transaction processing module 101 of executing the failed action in a manner of submitting the failure notification, and the synchronous transaction processing module 101 notifies each synchronous transaction module in the preamble of synchronously revoking the corresponding local transaction in a manner of issuing a global rollback notification by the synchronous transaction processing module 101. Such as the synchronized transaction execution module C failing to execute the local transaction, the synchronized transaction execution module C first attempts to retry executing the local transaction. This retry is not conditioned on success of the retry as in the conventional technique, but if the retry is not successful even after repetition, the retry is terminated when the retry condition described below is met. The retry condition includes one of: retry execution of the local transaction for a predetermined number of times, retry execution of the local transaction for a predetermined duration, finding anomalous data (such as data errors or receipt of an artificial termination instruction), or in some aging-pursuing procedures, retry conditions may be enforced to initiate revocation directly without retry. When the retry condition is met, the synchronous transaction execution module C still does not successfully execute the local transaction, and then the synchronous transaction execution module C notifies the synchronous transaction processing module 101, and the synchronous transaction processing module 101 cancels the corresponding global transaction and notifies each synchronous transaction module in the preamble of the global transaction, that is, the synchronous transaction module a and the synchronous transaction module B synchronously cancels the corresponding local transaction. Thus, in the invention, in the synchronization mode, if any one synchronization transaction module fails to execute the local transaction and causes the process revocation, all other synchronization transaction modules participating in the process under the control of the synchronization transaction processing module of the consistency platform synchronously revoke the related data, thereby ensuring that the data consistency can be still ensured when the execution fails.
In the embodiment shown in FIG. 3, two asynchronous transaction modules D and E are shown. In one embodiment, asynchronous transaction module D is a print transaction, while asynchronous transaction module E consumes a notification transaction. Asynchronous transaction module D and asynchronous transaction module E are transaction modules that operate in asynchronous mode. The asynchronous transaction module D sends a create message to the asynchronous message transaction module 102 of the coherency platform P and then executes the local transaction. If the asynchronous transaction module D fails to execute the local transaction and does not retry successfully when the retry condition is satisfied, a cancel message is sent to the asynchronous message transaction module 102, and the asynchronous message transaction module 102 cancels the relevant transaction. After the asynchronous transaction module D succeeds in performing the local transaction and sends an acknowledgement message to the asynchronous message transaction module 102, the asynchronous message transaction module 102 sends a message to the message reach module 103. The message queue is packaged in the message reach module 103, and the asynchronous message transaction module 102 monitors the status of the message queue in the message reach module 103, and adjusts the message queue or retries to send the message if the message queue fails. In the embodiment shown in fig. 3, the conformance platform P further comprises a message configuration module 104, the message configuration module 104 being connected to the message reach module 103. The message queue packaged in the message reach module 103 includes multiple sets of message queues, the message configuration module 104 assigns different priorities to the multiple sets of message queues, the message reach module 103 sends a message to the message queue with a high priority, and if the message queue with a high priority fails, the message reach module switches to send a message to the message queue with a low priority. In the embodiment shown in fig. 3, two message queues are encapsulated in the message reach module 103: kafka message queues and Redis message queues, the message configuration module 104 prioritizes Kafka message queues over Redis message queues. The Kafka queue is thus the primary message queue. The Kafka queue is used when the Kafka queue is operating normally. If the Kafka queue is down, the Kafka queue is switched to the Redis queue. When the asynchronous message transaction module 102 detects that all message queues in the message reach module 103 have failed, the asynchronous message transaction module 102 continuously retries to send messages to each message queue until the message queues recover. For example, if the asynchronous message transaction module 102 detects that both the Kafka queue and the Redis queue are down, the asynchronous message transaction module 102 continuously retries sending messages to the message reach module 103 until the message reach module 103 recovers. After the message reach module 103 finishes processing, a processing message, such as a consumption notification, is sent to the subsequent asynchronous transaction module E. The invention strips the message sending work from the asynchronous transaction module D, and then places the work on the asynchronous message transaction module 102, the message trigger module 103 and the message configuration module 104 of the consistency platform P for uniform processing, so that the asynchronous transaction module D is focused on executing local transactions, and the consistency platform P is focused on sending messages, thereby ensuring the data consistency among the transaction modules in the asynchronous mode.
In the embodiment shown in fig. 3, the consistency platform P further includes a global transaction statistics module M, and the global transaction statistics module M calculates the transaction execution success rate and the transaction execution speed in a predetermined period. In the illustrated embodiment, the global transaction statistics module M includes a calculation module 105 and a monitoring module 106. The calculation module 105 calculates a transaction execution success rate and a transaction execution speed in a predetermined period. The monitoring module 106 displays the transaction execution success rate and the transaction execution speed calculated by the calculation module 105, and sends an alarm when the transaction execution success rate and the transaction execution speed reach the set threshold. In the prior art, data consistency services are mostly scattered providing functions, and various modes are not aggregated on a platform, and the scattered functions are adopted, so that the execution condition of the whole global transaction is a black box for a user, and technicians are difficult to perceive how the execution condition of the current transaction is, how much failure rate is, and how much efficiency is. The invention can count the global affairs by using the global affair counting module M on the consistency platform P, and display the affairs through the monitoring module 106, thereby providing the technical personnel with perception of the affair execution situation at any time. The calculation module 105 in the global transaction statistics module M aggregates the execution success rate, execution rate, etc. of the transactions in the specified period through the periodic tasks, and sends an alarm through the threshold configured by the user when the failure rate of the transactions is too high or the transactions are executed for too long.
The data consistency method and the device of the multi-transaction system can ensure the data consistency under the long-flow multi-transaction environment deployed in a micro-service mode.
It should also be noted that the above-mentioned embodiments are only specific embodiments of the present invention. It is apparent that the present invention is not limited to the above embodiments and similar changes or modifications can be easily made by those skilled in the art from the disclosure of the present invention and shall fall within the scope of the present invention. The embodiments described above are provided to enable persons skilled in the art to make or use the invention and that modifications or variations can be made to the embodiments described above by persons skilled in the art without departing from the inventive concept of the present invention, so that the scope of protection of the present invention is not limited by the embodiments described above but should be accorded the widest scope consistent with the innovative features set forth in the claims.
Claims (12)
1. A data consistency method for a multi-transaction system for achieving data consistency in a synchronous mode among a plurality of transaction modules, comprising:
a first transaction module initiates a flow, creates a corresponding global transaction on a consistency platform, and executes a local transaction;
the first transaction module executes the local transaction, calls the next transaction module, reports the data after the local transaction is executed to the consistency platform, and the consistency platform updates the data of the corresponding global transaction;
after each transaction module executes the local transaction in sequence, calling the next transaction module, reporting data after the local transaction is executed to a consistency platform, updating the data of the corresponding global transaction by the consistency platform until all the transaction modules execute the local transaction, sequentially confirming the successful execution of the local transaction by each transaction module and informing the consistency platform of confirming the result, wherein the data of the corresponding global transaction on the consistency platform is consistent with the data after all the transaction modules execute the local transaction;
if the local transaction still fails to be executed after the retry condition is met, the consistency platform is informed, and the consistency platform cancels the corresponding global transaction and informs all transaction modules in the preamble of the global transaction to synchronously cancel the corresponding local transaction.
2. The data coherency method for a multi-transaction system according to claim 1, wherein the retry condition comprises one of: retry execution of the local transaction for a predetermined number of times, retry execution of the local transaction for a predetermined length of time, discover anomalous data, or a retry condition is enforced to be set to not retry direct retraction.
3. A data consistency method for a multi-transaction system for implementing data consistency in an asynchronous mode among a plurality of transaction modules, comprising:
a transaction module sends a creation message to the consistency platform and executes a local transaction;
if the transaction module successfully executes the local transaction, sending a confirmation message to the consistency platform, sending a message to the message queue by the consistency platform, and if the transaction module fails to execute the local transaction, sending a cancellation message to the consistency platform;
the coherency platform monitors the status of the message queue and, if the message queue fails, adjusts the message queue or retries sending the message.
4. The data consistency method of a multi-transaction system as recited in claim 3,
the message queues comprise a plurality of sets of message queues, the plurality of sets of message queues are endowed with different priorities, the consistency platform sends messages to the message queue with high priority, and if the message queue with high priority fails, the consistency platform is switched to send messages to the message queue with low priority;
if all message queues fail, the coherency platform continues to retry sending messages to each message queue until the message queue recovers.
5. The data consistency method of a multi-transaction system as recited in claim 4,
the multiple sets of message queues comprise Kafka message queues and Redis message queues, and the Kafka message queues have higher priority than the Redis message queues.
6. A data coherency apparatus for a multi-transaction system, for implementing data coherency between a plurality of transaction modules in a synchronous mode or an asynchronous mode, comprising:
the system comprises a plurality of synchronous mode transaction modules, a plurality of synchronous mode transaction modules and a plurality of synchronous mode transaction modules, wherein the synchronous mode transaction modules execute local transactions in sequence, and the local transactions executed by the synchronous mode transaction modules are correlated;
the system comprises a plurality of asynchronous mode transaction modules, a plurality of cache modules and a plurality of cache modules, wherein the asynchronous mode transaction modules respectively execute local transactions, and the local transactions executed by the asynchronous mode transaction modules are independent;
the system comprises a consistency platform, a data processing module and a data processing module, wherein the consistency platform comprises a synchronous transaction processing module, an asynchronous message transaction module and a message trigger module, the synchronous transaction processing module is communicated with each synchronous transaction module, the asynchronous message transaction module is communicated with each asynchronous mode transaction module, and the asynchronous message transaction module is also communicated with the message trigger module;
wherein,
the first synchronous transaction module informs the synchronous transaction processing modules of the consistency platform to create a corresponding global transaction and execute the local transaction according to the initiation flow, the first synchronous transaction module finishes executing the local transaction, calls the next synchronous transaction module and reports the data after executing the local transaction to the synchronous transaction processing modules, the synchronous transaction processing modules update the data of the corresponding global transaction, each synchronous transaction module finishes executing the local transaction in sequence, calls the next synchronous transaction module and reports the data after executing the local transaction to the synchronous transaction processing modules, the synchronous transaction processing modules update the data of the corresponding global transaction until all synchronous transaction modules finish executing the local transaction, each synchronous transaction module confirms successful execution of the local transaction in sequence and informs the consistency platform of a confirmation result, and the data of the corresponding global transaction of the synchronous transaction processing modules and all synchronous transaction modules finish executing the local transaction The data after the transaction are consistent;
if any one synchronous transaction module fails to execute the local transaction, the local transaction is selected to be executed again until the retry condition is met, if the local transaction still fails to be executed after the retry condition is met, the synchronous transaction processing module is informed, the synchronous transaction processing module cancels the corresponding global transaction and informs each synchronous transaction module in the preamble of the global transaction to synchronously cancel the corresponding local transaction;
each asynchronous transaction module sends a created message to an asynchronous message transaction module of the consistency platform and executes a local transaction, if the asynchronous transaction module successfully executes the local transaction, a confirmation message is sent to the asynchronous message transaction module, the asynchronous message transaction module sends a message to a message queue in the message reach module, and if the asynchronous transaction module fails to execute the local transaction, a revocation message is sent to the asynchronous message transaction module; the asynchronous message transaction module monitors the status of the message queue in the message reach module and adjusts the message queue or retries sending the message if the message queue fails.
7. The data consistency device of the multi-transaction system according to claim 6, wherein the consistency platform further comprises a global transaction statistics module, and the global transaction statistics module calculates a transaction execution success rate and a transaction execution speed within a predetermined period.
8. The data consistency device of a multi-transaction system of claim 7, wherein the global transaction statistics module comprises a calculation module and a monitoring module,
the calculation module calculates the transaction execution success rate and the transaction execution speed in a preset period;
the monitoring module displays the transaction execution success rate and the transaction execution speed calculated by the calculation module, and sends an alarm when the transaction execution success rate and the transaction execution speed reach the set threshold.
9. The data coherency mechanism for a multi-transaction system of claim 6 wherein the retry condition comprises one of: retry execution of the local transaction for a predetermined number of times, retry execution of the local transaction for a predetermined length of time, discover anomalous data, or a retry condition is enforced to be set to not retry direct retraction.
10. The data coherency device for a multi-transaction system according to claim 6, wherein the coherency platform further comprises a message configuration module, the message queue comprises a plurality of sets of message queues, the message configuration module assigns different priorities to the plurality of sets of message queues, the message reach module sends a message to the message queue with a high priority, and if the message queue with a high priority fails, the message reach module switches to send a message to the message queue with a low priority.
11. The data coherency mechanism for a multiservice system according to claim 10, wherein the plurality of sets of message queues include Kafka message queues and Redis message queues, and wherein the message configuration module gives the Kafka message queues a higher priority than the Redis message queues.
12. The data coherency device for a multi-transaction system according to claim 10, wherein the asynchronous message transaction module monitors that all message queues in the message reach module fail, and the asynchronous message transaction module continues to retry sending messages to each message queue until the message queues recover.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011264382.0A CN112346827B (en) | 2020-11-12 | 2020-11-12 | Data consistency method and device for multi-transaction system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011264382.0A CN112346827B (en) | 2020-11-12 | 2020-11-12 | Data consistency method and device for multi-transaction system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112346827A true CN112346827A (en) | 2021-02-09 |
| CN112346827B CN112346827B (en) | 2023-05-02 |
Family
ID=74362721
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011264382.0A Active CN112346827B (en) | 2020-11-12 | 2020-11-12 | Data consistency method and device for multi-transaction system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112346827B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113467898A (en) * | 2021-09-02 | 2021-10-01 | 北京开科唯识技术股份有限公司 | Multi-party cooperative service processing method and system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101488149A (en) * | 2009-02-19 | 2009-07-22 | 上海交通大学 | Context self-adapting movable transacter in general fitting surroundings |
| CN106330688A (en) * | 2016-10-21 | 2017-01-11 | 上海览海在线健康管理有限公司 | Method and system for sending medical message based on random message queue |
| US20190089575A1 (en) * | 2017-09-15 | 2019-03-21 | Walmart Apollo, Llc | Event Ordering Framework in Distributed Asynchronous Systems |
| CN109976919A (en) * | 2017-12-28 | 2019-07-05 | 北京京东尚科信息技术有限公司 | A kind of transmission method and device of message request |
| CN111225012A (en) * | 2018-11-27 | 2020-06-02 | 阿里巴巴集团控股有限公司 | Transaction processing method, device and equipment |
| CN111367628A (en) * | 2020-03-05 | 2020-07-03 | 中国银行股份有限公司 | Distributed transaction processing method and device, message producer and consumer system |
| CN111651244A (en) * | 2020-07-01 | 2020-09-11 | 中国银行股份有限公司 | Processing system for distributed transactions |
| CN111897825A (en) * | 2020-06-01 | 2020-11-06 | 中国人民财产保险股份有限公司 | Distributed transaction processing method and device |
-
2020
- 2020-11-12 CN CN202011264382.0A patent/CN112346827B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101488149A (en) * | 2009-02-19 | 2009-07-22 | 上海交通大学 | Context self-adapting movable transacter in general fitting surroundings |
| CN106330688A (en) * | 2016-10-21 | 2017-01-11 | 上海览海在线健康管理有限公司 | Method and system for sending medical message based on random message queue |
| US20190089575A1 (en) * | 2017-09-15 | 2019-03-21 | Walmart Apollo, Llc | Event Ordering Framework in Distributed Asynchronous Systems |
| CN109976919A (en) * | 2017-12-28 | 2019-07-05 | 北京京东尚科信息技术有限公司 | A kind of transmission method and device of message request |
| CN111225012A (en) * | 2018-11-27 | 2020-06-02 | 阿里巴巴集团控股有限公司 | Transaction processing method, device and equipment |
| CN111367628A (en) * | 2020-03-05 | 2020-07-03 | 中国银行股份有限公司 | Distributed transaction processing method and device, message producer and consumer system |
| CN111897825A (en) * | 2020-06-01 | 2020-11-06 | 中国人民财产保险股份有限公司 | Distributed transaction processing method and device |
| CN111651244A (en) * | 2020-07-01 | 2020-09-11 | 中国银行股份有限公司 | Processing system for distributed transactions |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113467898A (en) * | 2021-09-02 | 2021-10-01 | 北京开科唯识技术股份有限公司 | Multi-party cooperative service processing method and system |
| CN113467898B (en) * | 2021-09-02 | 2022-01-18 | 北京开科唯识技术股份有限公司 | Multi-party cooperative service processing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112346827B (en) | 2023-05-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100479575C (en) | Method and apparatus for realizing scheduled operation in equipment management | |
| US20040001449A1 (en) | System and method for supporting automatic protection switching between multiple node pairs using common agent architecture | |
| ES2929144T3 (en) | Manage and synchronize independent computing resources | |
| CN113742107B (en) | Processing method for avoiding message loss in message queue and related equipment | |
| CN102263674A (en) | Alarm reporting method, system and equipment | |
| CN112346827A (en) | Data consistency method and device for multi-transaction system | |
| US20050081110A1 (en) | Hybrid agent-oriented object model to provide software fault tolerance between distributed processor nodes | |
| EP0175170B1 (en) | Off-line notification in communication networks | |
| CN114971506A (en) | System and method for separating process engine and business event processing | |
| CN100373976C (en) | Method and apparatus for processing fault task in apparatus management | |
| CN110109772B (en) | CPU restarting method, communication equipment and readable storage medium | |
| CN112667383A (en) | Task execution and scheduling method, system, device, computing equipment and medium | |
| CN111010313B (en) | Batch processing state monitoring method, server and storage medium | |
| CN101350735A (en) | Method for synchronization of alarm | |
| CN101409633B (en) | Method for implementing business plate report and alarm in communication equipment | |
| CN118369245A (en) | Scalable method for handling faults in an electric vehicle charging station network | |
| CN115630107A (en) | Abnormal data processing method, electronic device and computer readable storage medium | |
| CN115544034A (en) | Data consistency method and service system | |
| CN114567536A (en) | Abnormal data processing method and device, electronic equipment and storage medium | |
| CN114880137A (en) | Message processing method and device, electronic equipment and storage medium | |
| JP4716492B2 (en) | Redundant configuration system and method for second computer system to immediately recover when first computer system fails | |
| JPH05143377A (en) | Alarm informing system | |
| CN100377526C (en) | Method of treating calling example maintenance management abnormal | |
| CN116932274B (en) | Heterogeneous computing system and server system | |
| CN113706113A (en) | Customs declaration message declaration processing method based on local signature adding and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |