CN112395102A

CN112395102A - Distributed database middleware solution method

Info

Publication number: CN112395102A
Application number: CN202011102929.7A
Authority: CN
Inventors: 甄慧斌
Original assignee: Beijing Simulation Center
Current assignee: Beijing Simulation Center
Priority date: 2020-10-15
Filing date: 2020-10-15
Publication date: 2021-02-23

Abstract

One embodiment of the invention discloses a distributed database middleware solution, which provides a uniform distributed transaction interface for local transactions, two-stage transactions and flexible transactions by integrating the existing mature transactions, wherein the integrated transactions comprise: XA-based two-phase transactions and sata-based flexible transactions; wherein the XA-based two-phase transaction integrates an XA transaction and a two-phase transaction; the SEATA-based flexible transaction integrates the SEAta AT transaction in the SEATA framework. The method integrates the existing mature affairs, provides a uniform distributed affair interface for local affairs, two-stage affairs and flexible affairs, and makes up the defects of the current method.

Description

Distributed database middleware solution method

Technical Field

The invention relates to the field of computer databases, in particular to a distributed database middleware solution.

Background

Under the distributed application popular environment based on micro-service, more and more application scenes require that access to a plurality of services and a plurality of corresponding database resources can be incorporated into the same transaction, and distributed transactions occur. In a distributed application scenario, it becomes a system performance constraint. How to make the database meet the characteristics of ACID in a distributed scene or find a corresponding alternative method is a key solution direction of distributed transactions. The current solutions for transactions are mainly local transactions, two-phase commit transactions, and flexible transactions.

Local transaction: and each data node is allowed to manage own transaction on the premise of not starting any distributed transaction manager. There is no coordination and communication capability between them and there is no mutual knowledge of the success or failure of other data node transactions. Local transactions do not have any loss in performance, but are not distracting in strong and final coherency.

Two-phase commit transaction: transactional execution requires locking all of the required resources in the process, which is more suitable for short transactions where execution time is fixed. For long transactions, the monopolizing of data during the whole transaction process can cause the concurrent performance of the business system depending on hot spot data to be obviously degraded.

Flexible affairs: the idea of flexible transaction is to move the exclusive lock operation from the resource layer to the service layer through the service logic. The throughput of the system is improved by relaxing the requirement on strong consistency.

In the daily work of developers, due to different application scenes, the developers are required to be capable of reasonably balancing various distributed transactions between performance and functions.

The API and functions of the strong consistent transaction and the flexible transaction are not identical, and free transparent switching cannot be achieved between the strong consistent transaction and the flexible transaction. In the development decision phase, a choice has to be made between strong and flexible transactions, so that the design and development costs are greatly increased.

XA-based strongly consistent transactions are relatively simple to use, but do not cope well with the high concurrency of the Internet or the long transaction scenarios of complex systems; flexible business requires the developer to modify the application, the access cost is very high, and the developer needs to realize resource locking and reverse compensation by himself.

Disclosure of Invention

Therefore, aiming at the problems, the invention integrates the existing mature affairs, provides a uniform distributed affair interface for local affairs, two-phase affairs and flexible affairs, makes up the defects of the current method, and provides a distributed affair solution.

In order to achieve the purpose, the invention adopts the following technical scheme:

the invention provides a distributed database middleware solution method on one hand, the method provides a uniform distributed transaction interface for local transaction, two-stage transaction and flexible transaction by integrating the existing mature transaction, and the integrated transaction comprises the following steps:

XA-based two-phase transactions and sata-based flexible transactions;

wherein the content of the first and second substances,

the XA-based two-phase transaction integrates an XA transaction and a two-phase transaction;

the SEATA-based flexible transaction integrates the SEAta AT transaction in the SEATA framework.

In one embodiment, the method includes integrating an embedded transaction manager to provide services in the form of jar packages.

In one embodiment, the XA-based two-phase transaction submission uses application engineering abstracted by a DTP model defined by an X/OPEN organization, and the transaction manager and the resource manager communicate in both directions using an XA protocol.

In one embodiment, the XA-based two-phase transaction includes a prepare phase, the database, in addition to passively accepting a commit instruction, can also inform the caller backwards whether the transaction can be committed; the transaction manager collects the ready results of all branch transactions and makes an atomic commit at the end.

In one embodiment, the method employs separate XA transaction management and connection pool management when integrating XA transactions, and the XA-based two-phase transaction can use connection pools of XA and non-XA simultaneously.

In one embodiment, the SEATA-based flexible transaction model includes a transaction manager, a resource manager, and a transaction coordinator;

wherein the content of the first and second substances,

the transaction coordinator is an independently deployed service, the transaction manager and the resource manager are deployed together with the service application in a jar package mode, and long connection is established between the transaction manager and the transaction coordinator, so that remote communication is kept in the whole transaction life cycle;

the transaction manager is an initiator of the global transaction and is responsible for starting, submitting and rolling back the global transaction;

the resource manager is a participant of the global transaction, is responsible for reporting the execution result of the branch transaction, and submits and rolls back the branch transaction through the coordination of the transaction coordinator.

In a specific embodiment, the data source interface based on the SEAATA flexible transaction is used for aggregating the data sources configured by the user.

In one embodiment, the method encapsulates the DataSource interface as a SEAATA-based DataSource interface when integrating the Seata AT transaction.

In a specific embodiment, the method is implemented in a multithreading mode when the partitioned physical SQL is executed and the Seata AT transaction is integrated, and context transfer of the global transaction ID is carried out between a main thread and a sub-thread.

In one embodiment, the method for implementing an XA-based two-phase transaction includes the steps of:

s101: starting a global transaction;

when receiving access end set autoCommit 0, XADCSTRATActionmanager calls specific XA transaction manager to start XA global transaction, and marks in XID form;

s102: executing the real slicing SQL;

after XADCSTRATransactionManager registers XAResource corresponding to database connection in the current XA transaction, the transaction manager sends XAResource. All SQL operations by the database before the xaresource. end command is received are marked as XA transactions;

s103: commit or rollback transactions;

after receiving the commit command of the access end, the XADCSTRATransactionManager entrusts the actual XA transaction manager to carry out a commit action, and the transaction manager collects all XAResources in the current thread and sends an XAResource. Then, sending a prefix instruction in sequence, and collecting all votes participating in XAResource;

if the feedback results of all XAResources are correct, calling a commit instruction to carry out final submission;

if the feedback result of any XAResource is incorrect, calling a rollback instruction to roll back;

after the transaction manager issues the commit instruction, any exception generated by the XAResource will be retried through the recovery log;

wherein the content of the first and second substances,

XADCSTRATRANSACTIONMANAGER is an XA implementation class of distributed transactions of the distributed database middleware solution; multiple data sources are managed and adapted and the opening, commit and rollback operations of the corresponding transactions are delegated to specific XA transaction managers.

In a specific embodiment, the implementation method based on the SEATA flexible transaction includes the following steps:

s201: initializing an engine;

when an application based on the SEATA flexible transaction is started, a data source configured by a user is adapted to a DataSourceProxy required by the SEATA transaction according to the configuration of the SEAta.conf, and is registered in a resource manager;

s202: starting a global transaction;

the transaction manager controls the boundary of the global transaction, the transaction manager acquires a global transaction ID by sending a Begin instruction to the transaction coordinator, and all branch transactions participate in the global transaction through the global transaction ID; the context of the global transaction ID is stored in a current thread variable;

s203: executing the real slicing SQL;

the fragment SQL in the Seata global transaction generates an undo snapshot through a resource manager, and sends a partidate instruction to a transaction coordinator to be added into the global transaction;

s204: commit or rollback transactions;

when the flexible transaction based on the SEATA is submitted, the transaction manager sends a submission or rollback instruction of the global transaction to the transaction coordinator, and the transaction coordinator coordinates all branch transactions to be submitted or rolled back according to the global transaction ID.

The invention has the following beneficial effects:

the invention can be completely compatible with all distributed transaction scenes, and achieves the optimal performance, but the distributed transactions are necessarily chosen under the guidance of CAP theorem. The invention hopes to give the user the option of distributed transaction, and the most suitable distributed transaction solution is used in different scenes. The method integrates the existing mature affairs, provides a uniform distributed affair interface for local affairs, two-stage affairs and flexible affairs, and makes up the defects of the current method.

In the method provided by the invention, the cross-library transaction after the data fragmentation is supported based on the XA two-phase transaction support item; two-stage submission ensures atomicity of operation and strong consistency of data;

after the service downtime is restarted, the transaction in the submission/rollback can be automatically recovered;

connection pools that support the use of both XA and non-XA.

In the method provided by the invention, the support item based on the SEAATA flexible transaction supports the cross-library transaction after the data fragmentation;

support for RC isolation levels;

performing transaction rollback through the undo snapshot;

and automatically recovering the transaction in submission after the service is down.

Drawings

In order to more clearly illustrate the embodiments of the present application or the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are one embodiment of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 illustrates a transaction manager and resource manager transaction exchange information architecture diagram according to one embodiment of the invention.

FIG. 2 illustrates an XA-based two-phase transaction architecture diagram according to one embodiment of the invention.

Fig. 3 shows a SEATA architecture diagram according to one embodiment of the present invention.

Figure 4 illustrates a SEATA-based flexible transaction architecture diagram according to one embodiment of the invention.

Detailed Description

In order to make the technical solution of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. The present invention will be described in detail with reference to specific examples, but the present invention is not limited to these examples. Variations and modifications may be made by those skilled in the art without departing from the principles of the invention and should be considered within the scope of the invention.

The invention provides a distributed database middleware solution (DCS for short), which integrates the existing mature affairs and provides a uniform distributed affair interface for local affairs, two-stage affairs and flexible affairs, wherein the integrated affairs comprise:

XA-based two-phase transactions (XA is the XA protocol, a specification of distributed transaction processing proposed by the X/Open organization) and sata-based flexible transactions (sata is a distributed transaction framework created jointly by the ali group and the ant golden service);

first is an XA-based two-phase transaction:

two-phase transaction commit uses the application engineering, TM (transaction manager) and RM (resource manager) concepts abstracted by the DTP model defined by the X/OPEN organization to ensure strong consistency of distributed transactions, wherein the transaction manager TM and the resource manager RM communicate in both directions using the XA protocol. Referring to fig. 1, fig. 1 illustrates a transaction manager and resource manager transaction exchange information architecture diagram in which an application engineering AP uses a set of resources from resource manager RMs and defines transaction boundaries through a TX interface, according to one embodiment of the invention.

Compared with traditional local transactions, XA transactions add a preparation phase, and the database can inform the caller backwards whether the transaction can be committed, besides passively accepting a commit instruction. The TM may collect the ready results of all branch transactions and make an atomic commit at the end to ensure strong coherency of the transactions.

The XA-based two-phase transaction integrates the XA transaction and the two-phase transaction, and thus the characteristics of the XA transaction and the two-phase transaction are also based on the characteristics of the XA two-phase transaction.

Java implements the XA model by defining JTA interfaces, a resource manager in the JTA interfaces needs to be provided by a database vendor for XA-driven implementation, a transaction manager needs to be implemented by the vendor of the transaction manager, and the traditional transaction manager needs to be bound with an application server, so that the use cost is high. The embedded transaction pipe device can provide services in the form of jar packets, and after the embedded transaction pipe device is integrated with a DCS, strong consistency of cross-library transactions after fragmentation can be guaranteed.

Typically, transactions of XA are supported only if the XA transaction connection pool provided by the transaction manager vendor is used. When an XA transaction is integrated, the DCS adopts a mode of separating XA transaction management and connection pool management, and can simultaneously use connection pools of XA and non-XA to achieve zero intrusion on an application program.

Referring to fig. 2, fig. 2 shows an XA-based two-phase transaction architecture diagram according to one embodiment of the invention.

The implementation method of the XA-based two-phase transaction comprises the following steps:

s101: starting a global transaction;

s102: executing the real slicing SQL;

for example:

XAResource1.start # # Enlist phase execution

Execution ("sql 1"); the # simulation executes a sliced SQL1

Execution ("sql 2"); the # simulation executes a sliced SQL2

XAResource1.end # # commit phase execution

The sql1 and sql2 in the above example will be marked as XA transactions.

S103: commit or rollback transactions;

after the transaction manager issues the commit instruction, any exception generated by the XAResource will be retried through the recovery log; so as to ensure the operation atomicity in the submission stage and strong data consistency.

For example:

XAResource1.prepare##ack:yes

XAResource2.prepare##ack:yes

XAResource1.commit

XAResource2.commit

XAResource1.prepare##ack:yes

XAResource2.prepare##ack:no

XAResource1.rollback

XAResource2.rollback

wherein the content of the first and second substances,

XADCSTRATransactionmanager is an XA implementation class of distributed transactions of the DCS method; multiple data sources are managed and adapted and the opening, commit and rollback operations of the corresponding transactions are delegated to specific XA transaction managers.

Based on the SEAATA flexible transaction:

SEATA is a distributed transaction framework created by the Ali group and the ant golden suit. The AT transaction aims AT providing incremental transaction ACID semantics under a micro-service architecture, so that a developer uses distributed transactions as local transactions, and the core concept is the same as that of DCS.

The Seata AT Transaction model contains TM, RM and TC (shorthand for Transaction coordinator, i.e., Transaction coordinator).

The TC is an independently deployed service, the TM and the RM are deployed together with the business application in a jar package mode, long connection is established between the TM and the RM, and long-distance communication is kept in the whole business life cycle.

The TM is the initiator of the global transaction and is responsible for the opening, committing and rollback of the global transaction.

The RM is a participant of the global transaction and is responsible for reporting the execution result of the branch transaction and submitting and rolling back the branch transaction through the coordination of the TC.

A typical lifecycle of a SEATA managed distributed transaction is as follows:

the TM requires the TC to start a completely new global transaction, and the TC generates an XID representing the global transaction;

XID runs through the entire call chain of the microservice;

as part of the global transaction under the TC corresponding to this XID, the RM registers the local transaction;

the TM requires the TC to commit or rollback the global transaction corresponding to the XID;

the TC drives all branch transactions under the global transaction to which the XID corresponds to complete commit or rollback.

Referring to fig. 3, fig. 3 shows a SEATA architecture diagram according to one embodiment of the present invention.

When integrating the Seata AT transaction, the method needs to integrate the TM, RM and TC models into the distributed transaction method of the DCS. On the database resource, Seata enables JDBC operation to remotely communicate with TC through interfacing a DataSource interface. And the DCS also faces the DataSource interface and aggregates the data sources configured by the user. Therefore, after the DataSource interface is packaged into the DataSource interface based on the Seata, the Seata AT transaction can be merged into the DCS method.

Because the SEATA-based flexible transaction integrates the SEATA AT transaction in the SEATA framework. The above-mentioned characteristics of the sea AT transaction are also based on the sea flexible transaction.

Referring to FIG. 4, FIG. 4 illustrates a SEATA-based flexible transaction architecture diagram in which the TM C/R commits or rolls back for the transaction manager, according to one embodiment of the present invention.

The implementation method of the present embodiment based on the SEAATA flexible transaction is as follows:

s201: initializing an engine;

s202: starting a global transaction;

s203: executing the real slicing SQL;

s204: commit or rollback transactions;

Since the partitioned physical SQL of the DCS is executed in a multi-threaded manner, context transfer of a global transaction ID needs to be performed between a main thread and a sub-thread when the data AT transaction is integrated.

It should be understood that the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention, and it will be obvious to those skilled in the art that other variations or modifications may be made on the basis of the above description, and all embodiments may not be exhaustive, and all obvious variations or modifications may be included within the scope of the present invention.

Claims

1. A distributed database middleware solution is characterized in that the method provides a uniform distributed transaction interface for local transactions, two-phase transactions and flexible transactions by integrating existing mature transactions, and the integrated transactions comprise:

XA-based two-phase transactions and sata-based flexible transactions;

wherein the content of the first and second substances,

2. The method of claim 1, comprising integrating an embedded transaction manager to provide services in the form of jar packages.

3. The method of claim 1, wherein the XA-based two-phase transaction commit uses application engineering abstracted by a DTP model defined by an X/OPEN organization, and wherein a transaction manager and a resource manager communicate bi-directionally using a protocol of XA.

4. The method of claim 1, wherein the XA-based two-phase transaction comprises a prepare phase, wherein the database, in addition to passively accepting a commit instruction, is capable of notifying a caller backwards whether the transaction can be committed; the transaction manager collects the ready results of all branch transactions and makes an atomic commit at the end.

5. The method of claim 1, wherein the method employs separate XA transaction management and connection pool management when integrating XA transactions, and wherein the XA based two phase transactions can use connection pools of XA and non-XA simultaneously.

6. The method of claim 1, wherein the SEATA-based flexible transaction model includes a transaction manager, a resource manager, and a transaction coordinator;

wherein the content of the first and second substances,

7. The method of claim 1, wherein the data source configured by the user is aggregated based on the SEAATA flexible transaction oriented to DataSource interface.

8. The method of claim 1, wherein the method encapsulates the DataSource interface as a SEATA-based DataSource interface when integrating the Seata AT transaction.

9. The method according to claim 1, wherein context transfer of global transaction ID is performed between the main thread and the sub-thread when the fragmented physical SQL is executed in a multi-threaded manner and the data AT transaction is integrated.

10. The method of claim 1, wherein the method for XA-based two-phase transaction implementation comprises the steps of:

s101: starting a global transaction;

s102: executing the real slicing SQL;

s103: commit or rollback transactions;

wherein the content of the first and second substances,

11. The method according to claim 1, wherein the implementation method based on the SEATA flexible transaction comprises the following steps:

s201: initializing an engine;

s202: starting a global transaction;

s203: executing the real slicing SQL;

s204: commit or rollback transactions;