CN115658805B

CN115658805B - Transaction consistency management engine and method

Info

Publication number: CN115658805B
Application number: CN202211121313.3A
Authority: CN
Inventors: 李昕辉
Original assignee: Transwarp Technology Shanghai Co Ltd
Current assignee: Transwarp Technology Shanghai Co Ltd
Priority date: 2022-09-15
Filing date: 2022-09-15
Publication date: 2023-10-17
Anticipated expiration: 2042-09-15
Also published as: CN115658805A

Abstract

The embodiment of the invention discloses a transaction consistency management engine and a method, wherein the system comprises the following steps: transaction processing system, analysis processing system and transaction consistency management system; the transaction processing system comprises at least one sub-database, and each sub-database corresponds to one log buffer area; the sub-database is used for generating a transaction change log containing a log type according to a received transaction submitting instruction and caching the transaction change log into a corresponding log buffer area; and the transaction consistency management system is used for determining target increment data corresponding to the transaction processing system according to the log type of the transaction change log in each log buffer area and synchronizing the target increment data to the analysis processing system. By utilizing the system, the increment data of the transaction change log related to the same transaction is ensured to be used as the target increment data and is synchronized to the analysis processing system, so that the consistency of the transaction and the correctness of the transaction analysis are ensured.

Description

Transaction consistency management engine and method

Technical Field

The invention relates to the technical field of databases, in particular to a transaction consistency management engine and a transaction consistency management method.

Background

The online transaction processing (On-Line Transaction Processing, OLTP) system and the online analytical processing (Online Analytical Processing, OLAP) system are heterogeneous two database systems that are adapted for different usage scenarios, OLTP databases are adapted for transactional operating scenarios, and OLAP systems are adapted for analytical operating scenarios. Systems that are optimized separately for both functions, both to meet OLTP and OLAP requirements, are referred to as hybrid transaction analysis (Hybrid Transaction and Analysis Processing, HTAP) systems. OLTP data is periodically and synchronously imported into the OLAP system.

In the prior art, data synchronization from an OLTP system to an OLAP system is performed based on a change log, and the principle is as follows: subscribing to a change log of the OLTP library, and applying the data change to the OLAP system according to the change log. This approach does not fit well in a scenario where the OLTP library is a distributed database consisting of multiple sub-databases, each of which generates an independent change log. Due to the characteristics of the distributed system, there may be time for the change logs generated by each sub-database related to the same transaction to be applied to the OLAP system, and in a stage where only one change log is applied and the other change log is not applied, reading data from the OLAP system may generate a result against the consistency of the transaction.

Disclosure of Invention

The invention provides a transaction consistency management engine and a transaction consistency management method, which are used for realizing transaction consistency and ensuring the correctness of transaction analysis.

In a first aspect, the present embodiment provides a transaction consistency management engine, including: transaction processing system, analysis processing system and transaction consistency management system; wherein,,

the transaction processing system comprises at least one sub-database, and each sub-database corresponds to one log buffer area;

the sub-database is used for generating a transaction change log containing a log type according to a received transaction submitting instruction and caching the transaction change log into a corresponding log buffer area;

and the transaction consistency management system is used for determining target increment data corresponding to the transaction processing system according to the log type of the transaction change log in each log buffer area and synchronizing the target increment data to the analysis processing system.

In a second aspect, the present embodiment provides a method for managing transaction consistency, which is executed by the transaction consistency management engine according to the embodiment of the first aspect, where the transaction consistency management engine includes a transaction processing system, an analysis processing system, and a transaction consistency management system, and the transaction processing system includes at least one sub-database, where each sub-database corresponds to one log buffer, and the method includes:

Generating a transaction change log containing a log type according to a received transaction submitting instruction through the sub database, and caching the transaction change log into a corresponding log buffer area;

and determining target incremental data corresponding to the transaction processing system by the transaction consistency management system according to the log types of the transaction change logs in the log buffer areas, and synchronizing the target incremental data to the analysis processing system.

The embodiment of the invention discloses a transaction consistency management engine and a method, wherein the system comprises the following steps: transaction processing system, analysis processing system and transaction consistency management system; the transaction processing system comprises at least one sub-database, and each sub-database corresponds to one log buffer area; the sub-database is used for generating a transaction change log containing a log type according to a received transaction submitting instruction and caching the transaction change log into a corresponding log buffer area; and the transaction consistency management system is used for determining target increment data corresponding to the transaction processing system according to the log type of the transaction change log in each log buffer area and synchronizing the target increment data to the analysis processing system. By using the system, the content of the transaction commit timestamp is added to the transaction change log, and for the case that the transaction processing system comprises a plurality of sub databases, the increment data of the transaction change log related to the same transaction are used as the target increment data and are synchronously sent to the analysis processing system, so that the transaction consistency is realized, and the correctness of the transaction analysis is ensured.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a transaction consistency management engine according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of another transaction consistency management engine according to a first embodiment of the present invention;

FIG. 3 is a diagram illustrating an exemplary interaction process for generating a one-phase commit log according to one embodiment of the present invention;

FIG. 4 is a diagram illustrating an exemplary process for a transaction system to send two-phase instructions according to one embodiment of the present invention;

FIG. 5 is an exemplary diagram of an interaction process for generating a two-stage pre-commit log according to one embodiment of the present invention;

FIG. 6 is an exemplary diagram of an interaction process for generating a two-phase commit log according to one embodiment of the present invention;

FIG. 7 is a diagram illustrating an exemplary interaction process for generating a two-stage rollback log according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of another transaction consistency management engine according to a first embodiment of the present invention;

FIG. 9 is an exemplary diagram of a secure timestamp update process provided in accordance with a first embodiment of the present invention;

FIG. 10 is an exemplary diagram of an incremental data determination process provided in accordance with a first embodiment of the present invention;

fig. 11 is a flow chart of a transaction consistency management method according to a second embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It is to be appreciated that OLTP and OLAP are heterogeneous two database systems that are adapted for different usage scenarios, OLTP databases are adapted for transactional operations, and OLAP systems (also known as data warehouses) are adapted for scenarios of analytical operations. Running both transaction and analysis functions on the same OLTP database or OLAP system can result in poor performance. The two functions are optimized respectively, so that the requirement of the OLTP can be met, and the system of the OLAP can be met, which is called an HTAP system. In order to adapt to the analysis operation scene, the OLTP data is periodically and synchronously imported into the OLAP system, and the analysis operation can be performed by accessing the OLAP system.

Considering the prior art, the principle of performing data synchronization from the OLTP system to the OLAP system based on the change log is as follows: subscribing to a change log of the OLTP library, and applying the data change to the OLAP system according to the change log. This approach does not fit well in a scenario where the OLTP library is a distributed database consisting of multiple sub-databases, each of which generates an independent change log, where reading data from the OLAP system may produce results that violate transactional consistency. For example, in one business, account 1 and account 2 are stored in score database a and score database B, respectively, a transaction is transferred 100 yuan from account 1 to account 2, at this time, database a generates a log of "100 yuan decrease", and database B generates a log of "100 yuan increase", but due to the characteristics of the distributed system, the two logs are applied to the OLAP system in a possible time sequence, and in a stage where only one of the logs is applied and the other is not applied, a user can read an erroneous result by querying the total amount of the plurality of accounts from the OLAP system. Causing the HTAP system to have transaction inconsistency problems.

Example 1

Fig. 1 is a schematic structural diagram of a transaction consistency management engine according to an embodiment of the present invention, where the embodiment is applicable to a case of implementing transaction consistency, and the engine may be implemented by hardware and/or software. The engine comprises: transaction processing system 10, analysis processing system 30, and transaction consistency management system 20; wherein, the transaction processing system 10 comprises at least one sub-database 11, and each sub-database 11 corresponds to a log buffer; the sub-database 11 is configured to generate a transaction change log including a log type according to the received transaction commit instruction, and cache the transaction change log into a corresponding log buffer; the transaction consistency management system 20 is configured to determine target incremental data corresponding to the transaction processing system according to the log types of the transaction change logs in each log buffer, and synchronize the target incremental data to the analysis processing system.

The transaction consistency management engine provided in this embodiment may be understood as a core part of the HTAP system, and is used to support the HTAP system to implement transaction consistency. The engine includes a transaction processing system 10, an analysis processing system 30, and a transaction consistency management system 20. Where transaction system 10 is referred to as an OLTP system and analysis processing system 30 is referred to as an OLAP system. Transaction system 10 includes at least one sub-database 11, each sub-database 11 corresponding to a log buffer.

It is known that the process of executing a transaction by a user includes: requests to open transactions, execute data manipulation language (Data Manipulation Language, DML) to perform data operations, request to commit transactions, and when a user performs these operations, these operations are issued to the transaction processing system 10. Transaction system 10 receives these operations and executes corresponding content in accordance with these operations.

First, when a user requests to open a transaction, the transaction processing system 10 generates a transaction identification number (Identity document, ID) of the current transaction. The "current transaction's set of partial databases" is now empty. Then, as the user performs the DML for each data operation, the transaction processing system 10 first determines which sub-database the data operation relates to, and if this sub-database is already in the "current transaction's sub-database set", performs the data operation directly on that sub-database; if this sub-database is not already in the "sub-database set for the current transaction," then the open sub-transaction request is performed on the sub-database and then the data operation is performed on the sub-database. Finally, when a user initiates a request to commit a transaction, transaction processing system 10 issues different transaction commit instructions to the score database based on the distribution of the change data, which may generate different types of change logs.

It will be appreciated that if a transaction performed by a user involves only one database, then the transaction processing system 10 need only send a transaction commit instruction to one of the partial databases to which the transaction relates. If a transaction performed by a user involves at least two sub-databases, then transaction processing system 10 may need to send a transaction commit instruction to each sub-database to which the transaction relates. In addition, for a database involving only one of the sub-databases, or at least two of the sub-databases, it may send different types of transaction commit instructions to the sub-databases.

The sub-database 11 is configured to generate a transaction change log including a log type according to the received transaction commit instruction, and cache the transaction change log in a corresponding log buffer.

In this embodiment, when the sub-database 11 receives a transaction commit request sent by the transaction processing system 10, the sub-database 11 generates a transaction change log of a corresponding type according to a transaction commit instruction. Wherein, the format of the transaction change log can be expressed as: log header|log payload, the log header may contain the transaction ID, log type, length of the log payload, the log payload may contain the specifics of the transaction execution, etc. Exemplary log types for the transaction change log include: one-phase commit, two-phase pre-commit, two-phase commit, and two-phase rollback. For transactions involving only one sub-database, the sub-database receives a transaction commit instruction as a one-phase commit, and then the sub-database generates a one-phase commit log. For transactions involving multiple sub-databases, the transaction commit instruction received by the sub-databases is a two-phase commit.

In this embodiment, each sub-database 11 corresponds to one log buffer, and the sub-databases 11 may buffer the transaction change log into the corresponding log buffer. The specific caching process can be expressed as follows: each sub-database corresponds to different log extraction threads respectively, the log extraction threads can be responsible for reading transaction change logs in different sub-databases, the log extraction threads continuously read new transaction change logs written in the responsible sub-databases, and the read logs are placed at the tail parts of the corresponding log buffers.

The transaction consistency management system 20 is configured to determine target incremental data corresponding to the transaction processing system 10 according to the log types of the transaction change logs in each log buffer, and synchronize the target incremental data to the analysis processing system.

In this embodiment, the transaction consistency management system 20, as a subsystem of the transaction consistency management engine, may process the transaction change log in the log buffer, and determine the incremental data corresponding to the transaction processing system 10 according to the transaction change log, which is referred to as the target incremental data in this embodiment. The transaction consistency management system 20 synchronizes the determined target delta data into the analysis processing system 30.

Considering that in the prior art, because the application time sequence of transaction change logs generated by different sub-databases of the same transaction may be inconsistent, when one part of the logs are applied first and the other part of the logs are not applied yet, incremental data can only be determined according to the logs applied first, and further the determined incremental data is not complete incremental data. Illustratively, in one business, account 1 and account 2 are stored in score database a and score database B, respectively, a transaction is transferred 100 yuan from account 1 to account 2, where database a generates a "100 yuan minus" log, and database B generates a "100 yuan plus" log, but due to the nature of the distributed system, the two logs are applied to the analysis processing system in order of time when there is a possibility that the user reads the wrong result by querying the total amount of the accounts from the analysis processing system 30 in a stage where only one of the logs is applied and the other is not applied.

Therefore, in order to ensure that the incremental data synchronized into the analysis processing system 30 is complete, it is necessary to ensure that the final incremental data is determined after all the logs generated by the sub-databases involved in the same transaction are applied to the analysis processing system 30, and the incremental data is synchronized into the analysis processing system 30, so as to avoid that a user queries the data from the analysis processing system 30 for erroneous results. Continuing with the above example, it is necessary to apply the log of "minus 100" generated by database a and the log of "plus 100" generated by database B to the analysis processing system 30, determine the final incremental data, and synchronize the incremental data to the analysis processing system 30.

In this embodiment, the transaction change log may be a one-phase commit log, a two-phase pre-commit log, a two-phase commit log, and a two-phase rollback log. It will be appreciated that for an executing transaction involving only one sub-database, since only one sub-database is involved in generating a transaction change log, the transaction consistency management system 20 need only obtain the contents of the transaction change log to determine the target delta data. In this embodiment, for a transaction involving a single sub-database, a single-phase commit is used, so that the sub-database generates a single-phase commit log when receiving a single-phase commit instruction.

And for transactions involving two or more sub-databases, it relates to the time order in which the transaction change logs in each sub-database are applied to the analysis processing system. To solve this problem, in this embodiment, for transactions involving multiple databases, two-phase commit is used, so that each database generates a different type of transaction change log when receiving a different instruction for the two-phase commit. The transaction change log corresponding to the two-stage commit comprises a two-stage pre-commit log, a two-stage commit log and a two-stage rollback log.

In this embodiment, a transaction commit timestamp is added to the two-phase commit log, where the transaction commit timestamp may be understood as the time at which the transaction was eventually committed. Different log buffers contain transaction change logs generated by different partial databases. The transaction consistency management system 20 repeatedly batch merges the data referenced in the log buffer and then generates the target delta data.

In this embodiment, the transaction consistency management system 20 may determine incremental data for each log buffer, and then combine the incremental data related to each buffer to determine the target incremental data. To ensure that the determined target incremental data is complete, it is necessary to ensure that the transaction change log of each of the partial databases involved in the execution of the parsed transaction is complete. Therefore, in this embodiment, a time node needs to be determined, and before the time node, the integrity of the transaction can be guaranteed, that is, the transaction change log corresponding to the transaction in each sub-database is already applied.

In order to determine the time node, for each log buffer, the transaction change log in the log buffer can be sequentially obtained, and the time node corresponding to the log buffer is determined according to the type of the transaction change log. It will be appreciated that for a one-phase commit log, the transaction commit timestamp is not involved, so the one-phase commit log has no effect on the time node and is negligible. And the two-stage commit log characterizes the final commit of the transaction in all types of logs generated by the two-stage commit, and the two-stage commit log comprises a transaction commit time stamp, so that the time node of each log buffer area can be determined according to the acquired two-stage commit log.

Further, after determining the time nodes in each log buffer, a time node capable of representing that the transaction change log related to each sub-database is a complete transaction before the time point can be selected from the time nodes. Based on the time node, it can be determined that the incremental data determined by the transaction change log before the time node contains the incremental data related to each sub-database, so that the transaction consistency can be ensured. It can be understood that, as the transaction change log is continuously generated in the sub-database, the transaction change log is continuously cached in the log buffer area, and the determined time node is changed, so that the transaction change log in the log buffer area is continuously analyzed, and the target incremental data is determined.

After the target delta data is determined, the target delta data may be synchronized to the analytics processing system 30. As transactions are executed, new target delta data is continually determined and synchronized into the analysis processing system 30. A common storage manner is adopted in the analysis processing system 30: and (5) incremental storage. The original version of the data is stored in the base file, and each time the transaction consistency management system 20 performs batch merge, a new version is generated and stored in an incremental file. The analysis processing system 30, when processing the user's analysis class request, first reads all the contents of each base file and delta file. And then filtering the result according to the filtering condition generated by the query request of the user, and filtering the data which does not meet the filtering condition to generate the result required by the user.

The embodiment of the invention discloses a transaction consistency management engine, which comprises the following components: transaction processing system, analysis processing system and transaction consistency management system; the transaction processing system comprises at least one sub-database, and each sub-database corresponds to one log buffer area; the sub-database is used for generating a transaction change log containing a log type according to a received transaction submitting instruction and caching the transaction change log into a corresponding log buffer area; and the transaction consistency management system is used for determining target increment data corresponding to the transaction processing system according to the log type of the transaction change log in each log buffer area and synchronizing the target increment data to the analysis processing system. By using the system, the content of the transaction commit timestamp is added to the transaction change log, and for the case that the transaction processing system comprises a plurality of sub databases, the increment data of the transaction change log related to the same transaction are used as the target increment data and are synchronously sent to the analysis processing system, so that the transaction consistency is realized, and the correctness of the transaction analysis is ensured.

Fig. 2 is a schematic structural diagram of another transaction consistency management engine according to a first embodiment of the present invention, which is optimized based on the first embodiment, in this embodiment, the transaction processing system 10 further includes a processing service process 12, and the processing service process 12 includes: a first sub-process 121, configured to generate a transaction commit instruction according to the received transaction commit request; a second sub-process 122, configured to generate a transaction commit instruction according to the received response success message sent by each of the sub-databases; a send sub-process 123 for sending transaction commit instructions to the respective score databases.

In this alternative embodiment, transaction system 10 also includes a processing service process 12, and processing service process 12 may be configured to process various types of received requests or generate instructions and send the instructions to the involved database of scores. Wherein transaction system 10 includes a first sub-process 121, a second sub-process 122, and a send sub-process 123.

A first sub-process 121 is configured to generate a transaction commit instruction according to the received transaction commit request. The process of executing the transaction by the user comprises the steps of requesting to start the transaction, executing the DML to perform data operation and requesting to submit the transaction, wherein the operations are issued to the transaction processing system first. When a user requests to commit a transaction by, for example, clicking on commit, a transaction commit request is generated and sent to a processing service process 12 in the transaction processing system 10. After receiving the transaction commit request, the service processing 12 may parse the transaction commit request, analyze what sub-databases are respectively involved in the transaction included in the transaction commit request, and generate a corresponding transaction commit instruction according to the transaction commit request. The transaction commit instruction is used for sending the transaction commit instruction to each related sub-database so as to enable each sub-database to generate a corresponding transaction change log.

And the second sub-process 122 is configured to generate a transaction commit instruction according to the received response success message sent by each of the sub-databases. When the partial database receives the transaction commit instruction, a response success message is generated and sent to the second sub-process 122. The second sub-process 122 generates a transaction commit instruction after receiving the response success message sent by each score database.

A send sub-process 123 for sending transaction commit instructions to the respective score databases. Specifically, the send sub-process 123 sends the transaction commit instruction to each of the score databases, so that each of the score databases generates a transaction change log corresponding to the transaction commit instruction.

Further, the first sub-process 121 is specifically configured to:

a1, determining a fractional database associated with the transaction commit request according to the received transaction commit request.

Specifically, when the first sub-process 121 receives the transaction commit request, the sub-databases associated with the transaction commit request are determined by parsing the transaction commit request, that is, determining which sub-databases are involved in executing the transaction included in the transaction commit request. It will be appreciated that the set of fractional databases associated with a transaction may be determined from the transaction identification number in the transaction commit request. The transaction commit request is associated with one or at least two of the number of the partial databases.

b1, if the sub-database is one, generating a one-stage commit instruction as a transaction commit instruction.

Specifically, if the sub-database is one, a one-phase commit instruction is generated for one-phase commit, and the one-phase commit instruction is used as a transaction commit instruction.

And c1, if the number of the sub-databases is at least two, generating a two-stage pre-commit instruction as a transaction commit instruction.

Specifically, if the number of the sub-databases is at least two, performing two-stage commit, generating a two-stage pre-commit instruction, and taking the two-stage pre-commit instruction as a transaction commit instruction.

d1, sending the transaction commit instruction to the fractional database.

Specifically, the commit instruction is sent to a database of the transaction in the transaction commit request.

Further, the second sub-process 122 is specifically configured to:

a2, receiving a response success message sent by each sub database to the two-stage pre-commit instruction.

Wherein the second sub-process 122 works when two or more sub-databases are involved, and two or more phase commit is performed. When the two-stage pre-commit instruction is sent to each of the sub-databases involved, the sub-databases will generate a response success message upon successful receipt of the two-stage pre-commit instruction and send the response success message to the second sub-process 122. The second sub-process receives a response success message sent by each sub-database to the two-stage pre-commit instruction.

And b2, judging whether response success messages of the sub databases are received within set time.

Specifically, it is determined whether a response success message of each sub database is received within a set time. Wherein the set time may be determined from historical empirical values.

And c2, if yes, acquiring the current timestamp as a transaction commit timestamp, and generating a two-stage commit instruction as a transaction commit instruction according to the transaction commit timestamp.

In this step, if the response success messages of all the associated databases are received within the set time, the current timestamp may be obtained from the time service center as a transaction commit timestamp, the transaction commit timestamp may be used as a part of the content to generate a two-stage commit instruction, and the two-stage commit instruction may be used as a transaction commit instruction.

It will be appreciated that when the response success messages of all the associated databases are received within a set time, the transaction commit timestamp is obtained and a two-phase commit instruction is generated. It is also understood that the transaction commit timestamp indicates that each of the sub-databases has responded successfully before that.

d2, otherwise, generating a two-stage rollback instruction as a transaction commit instruction.

In the step, if the response success messages of all the associated sub-databases are not received within the set time, a two-stage rollback instruction is generated as a transaction commit instruction.

As an alternative embodiment of the present invention, the present alternative embodiment is optimized based on the first embodiment, and the score database 11 is specifically configured to:

a3, receiving a transaction commit instruction sent by the transaction processing system.

Specifically, a transaction commit instruction sent by transaction processing system 10 is received. Wherein the transaction commit instruction may be a one-phase commit instruction, a two-phase pre-commit instruction, a two-phase commit instruction, or a two-phase rollback instruction.

And b3, if the transaction commit instruction is a one-stage commit instruction, generating a one-stage commit log corresponding to the transaction commit instruction as a transaction change log.

Specifically, if the transaction commit instruction is a one-stage commit instruction, which indicates that only one sub-database is involved in the execution of the transaction, the transaction commit instruction is parsed to obtain information such as a transaction ID, change data, and the like. From the information obtained, a one-phase commit log may be generated. Note that, in this embodiment, the format of the transaction change log may be expressed as: log header|log payload, the log header may contain the transaction ID, log type, length of the log payload, the log payload may contain the specifics of the transaction execution, etc. The log load content is different for different types of transaction change logs. The log load of the one-stage commit log comprises data changes with contents being transactions. And takes the one-stage commit log as a transaction change log.

Exemplary, FIG. 3 is a diagram illustrating an exemplary interaction process for generating a one-phase commit log according to one embodiment of the present invention. After the transaction processing system sends a phase commit instruction, the involved score databases generate a phase commit log.

And c3, if the transaction commit instruction is a two-stage pre-commit instruction, generating a two-stage pre-commit log corresponding to the transaction commit instruction as a transaction change log.

Specifically, if the transaction commit instruction is a two-stage pre-commit instruction, which indicates that the transaction only involves a plurality of sub-databases when executing, the transaction commit instruction is analyzed to obtain information such as a transaction ID, change data, and the like. From the information obtained, a two-phase pre-commit log may be generated. The format of the two-stage pre-commit instruction is not described in detail herein. The log load of the two-stage pre-commit log comprises data changes with contents of the transaction in the fractional database.

Illustratively, to more clearly express the interaction relationship between the transaction processing system 10 and the sub-database 11, fig. 4 is an exemplary diagram of a process for sending two-phase instructions by the transaction processing system according to the first embodiment of the present invention. The process can be expressed as:

s11, the transaction processing system sends a two-stage pre-commit instruction to the fractional databases with data changes in all current transactions.

S12, the transaction processing system waits for all the related sub-databases to return a response success message until reaching a preset timeout.

And S13, judging whether response success messages of all related sub-databases are received within preset timeout, if so, executing the steps S14-S15, and if not, executing the step S16.

S14, the transaction processing system acquires the current time stamp from the time service center and takes the current time stamp as a transaction commit time stamp.

S15, the transaction processing system sends a two-stage commit instruction to all the fractional databases with data changes in the current transaction.

S16, the transaction processing system sends a two-stage rollback instruction to all the databases with data change in the current transaction.

Exemplary, FIG. 5 is a diagram illustrating an exemplary interaction process for generating a two-stage pre-commit log according to one embodiment of the present invention. The principal processing system 10 sends a two-phase pre-commit instruction to the involved sub-databases 11, and the involved sub-databases 11 generate two-phase pre-commit logs.

d3, if the transaction commit instruction is a two-stage commit instruction, generating a two-stage commit log corresponding to the transaction commit instruction as a transaction change log.

It should be noted that, the two-stage commit instruction is sent by the transaction processing system 10 to each of the involved sub-databases 11 after each of the involved sub-databases sends a response success message to the transaction processing system 10.

Specifically, if the transaction commit instruction is a two-stage commit instruction, which indicates that multiple sub-databases are involved in the execution of the transaction, the transaction commit instruction may be parsed to obtain information such as a transaction ID, change data, and the like. From the information obtained, a two-phase commit log may be generated. The format of the two-stage commit log is not described in detail herein. Wherein the log load of the two-phase commit log contains a content-as-transaction commit timestamp.

Exemplary, FIG. 6 is a diagram illustrating an exemplary interaction process for generating a two-phase commit log according to one embodiment of the present invention. When transaction processing system 10 obtains the current timestamp from timing center 40 as the transaction commit timestamp, then sends a two-phase commit instruction to the involved sub-databases, which generate a two-phase commit log.

And e3, if the transaction commit instruction is a two-stage rollback instruction, generating a two-stage rollback log corresponding to the transaction commit instruction as a transaction change log.

Specifically, if the transaction commit instruction is a two-stage rollback instruction, which indicates that multiple sub-databases are involved in the execution of the transaction, the transaction commit instruction may be parsed to obtain information such as a transaction ID, change data, and the like. From the information obtained, a two-stage rollback log may be generated.

Fig. 7 is an exemplary diagram of an interaction procedure for generating a two-stage rollback log according to an embodiment of the present invention. After the transaction processing system 10 sends a two-phase rollback instruction to the involved sub-databases, the involved sub-databases generate two-phase rollback logs.

And f3, caching the transaction change log to a corresponding log cache area.

The sub-database 11 may cache the transaction change log into a corresponding log buffer. The specific caching process can be expressed as follows: each sub-database corresponds to different log extraction threads respectively, the log extraction threads can be responsible for reading transaction change logs in different sub-databases, and the log extraction threads continuously read new transaction change logs written in the responsible sub-databases and put the new transaction change logs at the tail parts of the corresponding log buffer areas.

FIG. 8 is a schematic diagram of another transaction consistency management engine according to a first embodiment of the present invention, wherein the transaction consistency management system 20 is optimized based on the first embodiment, and includes: a first determining module 21, configured to determine a target security timestamp according to a log type of the transaction change log in each log buffer; the second determining module 22 is configured to determine, according to the log type of the transaction change log in each log buffer, target incremental data corresponding to the transaction processing system in combination with the target security timestamp, and synchronize the target incremental data to the analysis processing system.

In this alternative embodiment, transaction consistency management system 20 includes a first determination module 21 and a second determination module 22. The first determination module 21 and the second determination module 22 may be regarded as key modules in the engine provided in this embodiment, and the key modules are used for processing the transaction change log, and merging data, so that the incremental data is the incremental data of the complete transaction.

The target security timestamp may be understood as that the transaction change log before the time contains one or more logs of complete transactions, that is, incremental data of one or more complete transactions may be determined according to the transaction change log in each log buffer before the target security timestamp. The target incremental data refers to the incremental data of the complete transaction determined from the respective sub-databases 11 involved.

In this optional embodiment, the transaction change log is sequentially acquired from each log buffer, and according to the log type of the acquired transaction change log, whether the transaction change log can update the secure timestamp can be determined. It will be appreciated that only the two-phase commit log in the transaction change log contains a commit timestamp, which can be used to update the target secure timestamp.

In this alternative embodiment, after the first determining module 21 determines the target security timestamp, the second determining module 22 sequentially obtains the transaction change logs from each log buffer, determines the target incremental data corresponding to the transaction processing system according to the log type of the obtained transaction change log and the target security timestamp, and synchronizes the target incremental data to the analysis processing system.

It will be appreciated that the one-phase commit log, which involves only one of the fractional databases, may be used to determine the target delta data directly from the one-phase commit log. For two-stage various types of logs related to a plurality of sub-databases, the log before the target security time stamp needs to be determined to be a complete transaction log, and the logs related to the sub-databases are analyzed to obtain target incremental data.

Further, with continued reference to fig. 8, the first determining module 21 includes: a first determining unit 211, configured to determine, for each log buffer, a security timestamp corresponding to the log buffer according to a log type of the transaction change log in the log buffer; the second determination unit 212 determines the security timestamp with the smallest value among the security timestamps as the target security timestamp.

The first determining module 21 includes a first determining unit 211 and a second determining unit 212. In this optional embodiment, for each log buffer, a transaction change log in the log buffer needs to be acquired, and a secure timestamp corresponding to the log buffer is determined according to a transaction commit timestamp included in a two-stage commit log in the transaction change log. After determining the security time stamp corresponding to each log buffer, the second determining unit 212 determines the security time stamp having the smallest value among the security time stamps as the target security time stamp. It will be appreciated that the transaction change log in each log buffer preceding the target secure timestamp is a log of complete transactions.

Further, the first determining unit 211 is specifically configured to:

a4, aiming at each log buffer area, acquiring the log type of the current change log in the log buffer area.

Specifically, the first determining unit 211 needs to determine the current secure timestamp corresponding to each log buffer, that is, needs to perform steps a 4) to g4 for each log buffer. And pointing the reading pointer to the head of each log buffer area, reading the current change log pointed by the pointer, and obtaining the log type of the current change log. It is understood that the log type of the current change log may be one-phase commit, two-phase pre-commit, two-phase commit, and two-phase rollback.

And b4, if the current change log is a one-stage commit log or a two-stage rollback log, the security time stamp is not updated.

Specifically, if the current change log is a one-stage commit log, it indicates that the transaction associated with the current change log involves only one partial database, and the secure timestamp is not updated if the secure timestamp is not required to be determined. If the current change log is a two-stage rollback log, it indicates that no response success message fed back by the sub-database related to the transaction is received within a set time, it may also be understood that some sub-databases have not responded to the two-stage pre-commit instruction, and in addition, the one-stage commit log and the two-stage rollback log do not contain time content, so that the secure timestamp is not updated.

And c4, if the current change log is a two-stage commit log, taking the larger value of the transaction commit timestamp and the security timestamp contained in the current change log as a new security timestamp.

Specifically, if the current change log is a two-stage commit log, the two-stage commit log includes a transaction commit timestamp. The two-phase commit log indicates that the transaction commit is complete, and thus, the transaction commit timestamp contained in the two-phase commit log may be obtained.

Before the security timestamp is updated for each log buffer, the security timestamp needs to be initialized to zero, and the security timestamp is updated according to the judgment of the current change log. The method for updating the secure timestamp is to compare the acquired transaction commit timestamp with the secure timestamp and take the larger value as a new secure timestamp. It will be appreciated that for this log buffer, the transaction may be considered committed before the new secure timestamp.

And d4, if the current change log is a two-stage pre-submitted log, judging whether the log buffer zone contains a two-stage submitted log or a two-stage rollback log which belongs to the same transaction with the current change log.

In this step, if the current change log is a two-stage pre-commit log, it is further required to determine whether the log buffer includes a two-stage commit log or a two-stage rollback log of the same transaction as the current change log, so as to further determine whether to update the secure timestamp.

And e4, if the two-stage commit log is included, not updating the security timestamp.

Specifically, if the log buffer area includes a two-stage commit log belonging to the same transaction as the current change log, the secure timestamp is not updated, and the transaction change log in the log buffer area needs to be continuously returned to be read, so that when the two-stage commit log of the same transaction is read, the secure timestamp is updated according to the transaction commit timestamp in the two-stage commit log.

And f4, if the two-stage rollback log is included, the security time stamp is not updated, and the processing is ended.

Specifically, if the log buffer contains a two-stage rollback log that belongs to the same transaction as the current change log, the security timestamp is not updated. And the two-stage rollback log indicates that no response success message of the feedback of the sub-database related to the transaction is received within a set time, which can also be understood that some sub-databases do not respond to the two-stage pre-commit instruction yet, so that the transaction does not complete commit yet, the secure timestamp is not updated and the process is finished.

And g4, taking the next transaction change log in the log buffer area as a new current change log, and returning to the step of continuously executing the update of the security time stamp corresponding to the log buffer area until the transaction change log in the log buffer area is traversed and finished.

Specifically, after the current change log is processed, the pointer needs to be moved to take the next transaction change log in the log buffer as a new current change log, the step of continuously executing the update of the security time stamp is returned until the transaction change log in the log buffer is traversed, and then the processing is ended. It will be appreciated that each log buffer will determine its corresponding secure timestamp for further use in determining the target secure timestamp.

For the sake of clarity, fig. 9 is an exemplary diagram illustrating the process of updating the secure timestamp according to the first embodiment of the present invention. The process can be expressed as:

s21, initializing a safety time stamp.

S22, aiming at each log buffer area, acquiring a current change log in the log buffer area, and executing a step S25 if the current change log is a one-stage submitted log; if the current change log is a two-stage rollback log, executing step S25; if the current change log is a two-stage commit log, executing step S23; if the current change log is the two-stage pre-commit log, step S24 is performed.

S23, taking the larger value of the transaction commit timestamp and the security timestamp contained in the current change log as a new security timestamp.

S24, judging whether a two-stage commit log or a two-stage rollback log of the same transaction exists in the log buffer, if so, executing the step S25, and if not, executing the step S27.

S25, judging whether the log buffer area has a non-traversed transaction change log, if so, executing the step S26, and if not, executing the step S27.

S26, taking the next transaction change log in the log buffer area as a new current change log.

S27, ending the processing.

With continued reference to fig. 8, further, the second determining module 22 includes: a third determining unit 221, configured to determine, for each log buffer, newly added data corresponding to the log buffer according to the log type of the transaction change log in the log buffer and in combination with the target security timestamp; the fourth determining unit 222 is configured to aggregate each new increment data as a target increment data corresponding to the transaction processing system 10, and synchronize the target increment data to the analysis database 11.

The second determining module 22 includes a third determining unit 221 and a fourth determining unit 222. In this alternative embodiment, a temporary space may be allocated. And determining newly added data corresponding to the current log buffer zone according to each transaction change log in each log buffer zone, and storing the newly added data into a temporary space. After determining the newly added data corresponding to each log buffer, the fourth determining unit 222 sums all the newly added data in the temporary space as the target incremental data. It can be appreciated that the newly added data is determined by combining the target security timestamp according to the log type of the transaction change log in the log buffer. It can be understood that determining the incremental data corresponding to each log buffer before the target secure timestamp can ensure the consistency of the transaction, and can also be understood that the target incremental data is the complete change data about the transaction.

Further, the third determining unit 221 is specifically configured to:

and a5, aiming at each log buffer area, acquiring the log type of the current change log in the log buffer area.

Specifically, the third determining unit 221 needs to determine the incremental data corresponding to each log buffer, that is, needs to perform steps a 5) to e5 for each log buffer. And pointing the reading pointer to the head of each log buffer area, reading the current change log pointed by the pointer, and obtaining the log type of the current change log. It is understood that the log type of the current change log may be one-phase commit, two-phase pre-commit, two-phase commit, and two-phase rollback.

And b5, if the current change log is a one-stage commit log, determining incremental data corresponding to the current change log according to the content contained in the current change log.

Specifically, if the current change log is a one-stage commit log, it indicates that the transaction associated in the current change log only involves one sub-database, the one-stage commit log includes data changes, and incremental data corresponding to the current change log can be determined according to content included in the current change log.

And c5, if the current change log is a two-stage commit log or a two-stage rollback log, determining that the current change log has no corresponding newly added data.

Specifically, if the current change log is a two-stage commit log and the content contained in the two-stage commit log is a transaction commit timestamp, determining that the current change log has no corresponding newly-added data. If the current change log is a two-stage rollback log, determining that the current change log has no corresponding newly added data.

d5, if the current change log is a two-stage pre-submitted log, determining incremental data corresponding to the current change log according to the log types of other transaction change logs contained in the buffer zone and combining the target security time stamp.

Specifically, if the current change log is a two-stage pre-commit log, the log type of the other transaction change log contained in the buffer area needs to be continuously determined, the other transaction change log is determined to be a two-stage rollback log or a two-stage commit log, and the incremental data corresponding to the current change log is determined by further combining a target security timestamp.

And e5, removing the current change log from the log buffer area, taking the next transaction change log in the log buffer area as a new current change log, and returning to the step of continuously executing the determination of the newly added data corresponding to the current change log until no transaction change log in the log buffer area is finished.

Specifically, after the current change log is processed, the current change log is removed from the log buffer area, a pointer is moved to take the next transaction change log in the log buffer area as a new current change log, the step of continuously executing the determination of the newly added data is returned until no transaction change log exists in the log buffer area, and the processing is ended. It will be appreciated that each log buffer will determine its corresponding newly added data for further use in determining the target newly added data.

Further, the third determining unit 221 is configured to perform a step of determining incremental data corresponding to the current change log according to the log type of the other transaction change log included in the buffer, and includes:

a6, acquiring other transaction change logs contained in the log buffer.

And b6, judging whether the other transaction change logs belong to two-stage commit logs or two-stage rollback logs of the same transaction as the current change log.

Specifically, the log type of the change log of other transactions is obtained, and whether the change log of other transactions belongs to the two-stage commit log or the two-stage rollback log of the same transaction as the current change log is judged.

And c6, if the log is rolled back in two stages, determining that the current change log has no corresponding newly-added data.

Specifically, if the other transaction change logs exist as two-stage rollback logs belonging to the same transaction with the current change log, the fact that the transaction is not finished to be submitted finally is indicated, and it is determined that the current change log has no corresponding newly added data.

And d6, if so, judging whether the transaction commit time stamp contained in the two-stage commit log is smaller than or equal to the target security time stamp.

Specifically, if there is a two-stage commit log in which the other transaction change log is the same transaction as the current change log, then it is necessary to obtain a transaction commit timestamp included in the two-stage commit log, and compare the transaction commit timestamp with a target secure timestamp.

And e6, if so, determining the incremental data corresponding to the current change log according to the content contained in the current change log.

Specifically, since the target secure timestamp indicates that each sub database involved in the transaction has completed data change, if the transaction commit timestamp is less than or equal to the target secure timestamp, the two-stage commit log is a log of the transaction completed to commit, and further, incremental data corresponding to the current change log is determined according to content included in the current change log.

And f6, if not, ending the processing.

Specifically, if the transaction commit timestamp is greater than the target secure timestamp, which indicates that all the sub databases related to the transaction are not completely committed, the process is ended. It will be appreciated that as the transaction is executed, a new transaction change log is stored in the log buffer for the next target delta data determination.

For the sake of clarity, fig. 10 is an exemplary diagram illustrating the incremental data determining process according to the first embodiment of the present invention. The process can be expressed as:

s31, aiming at each log buffer area, acquiring a current change log in the log buffer area. If the current change log is a two-stage commit log, executing step S36; if the current change log is a two-stage rollback log, executing step S36; if the current change log is a one-stage commit log, executing step S32; if the current change log is the two-stage pre-commit log, step S33 is performed.

S32, the incremental data contained in the current change log are put into a temporary space.

S33, judging whether a two-stage commit log or a two-stage rollback log of the same transaction exists in the log buffer, if so, executing the step S34, and if so, executing the step S36.

S34, if the corresponding transaction commit time stamp is smaller than or equal to the target security time stamp, executing the step S35 if yes, and executing the step S39 if no.

S35, the incremental data contained in the current change log are put into a temporary space.

S36, removing the current change log from the log buffer.

S37, judging whether a transaction change log exists in the log buffer area, if so, executing the step S38, and if not, executing the step S39.

S38, taking the next transaction change log in the log buffer area as a new current change log.

S39, ending the processing.

In this alternative embodiment, the transaction change log is converted into incremental data based on the target secure timestamp, so that it can be ensured that all changes included in one or more complete transactions are stored in each incremental storage file, and therefore, the analysis processing system can be ensured to have transaction consistency as long as the analysis processing system reads the complete and complete incremental file.

Example two

Fig. 11 is a flow chart of a transaction consistency management method according to a second embodiment of the present invention, where the method is applicable to a situation of implementing transaction consistency. The method may be performed by a transaction consistency management engine provided by the above embodiments, which may be implemented by hardware and/or software. The transaction consistency management engine comprises a transaction processing system, an analysis processing system and a transaction consistency management system, wherein the transaction processing system comprises at least one sub-database, and each sub-database corresponds to one log buffer area. As shown in fig. 11, the transaction consistency management method provided in the second embodiment specifically includes the following steps:

S410, generating a transaction change log containing a log type according to the received transaction submitting instruction through the sub database, and caching the transaction change log into a corresponding log buffer area.

It is known that the process of executing a transaction by a user includes: requests to open transactions, DML data operations, requests to commit transactions, which are issued to the transaction processing system after the user performs these operations. The transaction system receives these operations and executes corresponding content in accordance with these operations.

First, when a user requests to open a transaction, the transaction processing system generates a transaction ID for the current transaction. The "current transaction's set of partial databases" is now empty. Then, when the user executes the DML to perform each data operation, the transaction processing system firstly judges which sub-database the data operation relates to, and if the sub-database is already in the 'sub-database set of the current transaction', the data operation is directly executed on the sub-database; if this sub-database is not already in the "sub-database set for the current transaction," then the open sub-transaction request is performed on the sub-database and then the data operation is performed on the sub-database. Finally, when a user initiates a request for submitting a transaction, the transaction processing system issues different transaction submitting instructions to the fractional database according to the distribution of the change data, and the fractional database can generate different types of change logs.

It will be appreciated that if a transaction performed by a user involves only one database, the transaction processing system need only send a transaction commit instruction to one of the partial databases to which the transaction relates. If a transaction performed by a user involves at least two sub-databases, then the transaction processing system needs to send a transaction commit instruction to each sub-database to which the transaction relates. In addition, for a database involving only one of the sub-databases, or at least two of the sub-databases, it may send different types of transaction commit instructions to the sub-databases.

In this embodiment, after the sub database receives a transaction commit request sent by the transaction processing system, the sub database generates a transaction change log of a corresponding type according to a transaction commit instruction. Wherein, the format of the transaction change log can be expressed as: log header|log payload, the log header may contain the transaction ID, log type, length of the log payload, the log payload may contain the specifics of the transaction execution, etc. Exemplary log types for the transaction change log include: one-phase commit, two-phase pre-commit, two-phase commit, and two-phase rollback. For transactions involving only one sub-database, the sub-database receives a transaction commit instruction as a one-phase commit, and then the sub-database generates a one-phase commit log. For transactions involving multiple sub-databases, the transaction commit instruction received by the sub-databases is a two-phase commit.

In this embodiment, each sub-database corresponds to a log buffer, and the sub-databases may cache the transaction change log into the corresponding log buffer. The specific caching process can be expressed as follows: each sub-database corresponds to different log extraction threads respectively, the log extraction threads can be responsible for reading transaction change logs in different sub-databases, the log extraction threads continuously read new transaction change logs written in the responsible sub-databases, and the read logs are placed at the tail parts of the corresponding log buffers.

S420, determining target incremental data corresponding to the transaction processing system by the transaction consistency management system according to the log types of the transaction change logs in each log buffer area, and synchronizing the target incremental data to the analysis processing system.

In this embodiment, the transaction consistency management system is used as a subsystem of the transaction consistency management engine, and may process the transaction change log in the log buffer, and determine the incremental data corresponding to the transaction processing system according to the transaction change log, which is in this embodiment denoted as target incremental data. The transaction consistency management system synchronizes the determined target incremental data into the analysis processing system.

Considering that in the prior art, because the application time sequence of transaction change logs generated by different sub-databases of the same transaction may be inconsistent, when one part of the logs are applied first and the other part of the logs are not applied yet, incremental data can only be determined according to the logs applied first, and further the determined incremental data is not complete incremental data. Therefore, in order to ensure that the incremental data synchronized into the analysis processing system is complete, it is necessary to ensure that the final incremental data is determined after logs generated by the sub-databases related to the same transaction are all applied to the analysis processing system, and the incremental data is synchronized into the analysis processing system, so that a user is prevented from querying the data from the analysis processing system to generate an error result.

In this embodiment, the transaction change log may be a one-phase commit log, a two-phase pre-commit log, a two-phase commit log, and a two-phase rollback log. It will be appreciated that for an executing transaction involving only one sub-database, since only one sub-database is involved in generating a transaction change log, the transaction consistency management system need only obtain the contents of the transaction change log to determine the target delta data. In this embodiment, for a transaction involving a single sub-database, a single-phase commit is used, so that the sub-database generates a single-phase commit log when receiving a single-phase commit instruction.

In this embodiment, a transaction commit timestamp is added to the two-phase commit log, where the transaction commit timestamp may be understood as the time at which the transaction was eventually committed. Different log buffers contain transaction change logs generated by different partial databases. The transaction consistency management system repeatedly batch merges the data involved in the log buffer and then generates the target delta data.

In this embodiment, the transaction consistency management system may determine incremental data for each log buffer, and then combine the incremental data related to each buffer to determine the target incremental data. To ensure that the determined target incremental data is complete, it is necessary to ensure that the transaction change log of each of the partial databases involved in the execution of the parsed transaction is complete. Therefore, in this embodiment, a time node needs to be determined, and before the time node, the integrity of the transaction can be guaranteed, that is, the transaction change log corresponding to the transaction in each sub-database is already applied.

After the target delta data is determined, the target delta data can be synchronized to an analysis processing system. As transactions are executed, new target delta data is continually determined and synchronized to the analysis processing system. The analysis processing system adopts a common storage mode: and (5) incremental storage. The original version of data is stored in the base file, and after each batch combination, a new version is generated and stored in an incremental file. When the analysis processing system processes the analysis class request of the user, all contents of each basic file and increment file are read first. And then filtering the result according to the filtering condition generated by the query request of the user, and filtering the data which does not meet the filtering condition to generate the result required by the user.

The transaction consistency management method provided by the embodiment of the invention can be executed by the transaction consistency management engine provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims

1. A transaction consistency management engine, comprising: transaction processing system, analysis processing system and transaction consistency management system; wherein,,

the transaction consistency management system is used for determining target increment data corresponding to the transaction processing system according to the log type of the transaction change log in each log buffer area and synchronizing the target increment data to the analysis processing system;

the fractional database is specifically used for:

receiving a transaction commit instruction sent by the transaction processing system;

if the transaction commit instruction is a one-stage commit instruction, generating a one-stage commit log corresponding to the transaction commit instruction as a transaction change log;

if the transaction commit instruction is a two-stage pre-commit instruction, generating a two-stage pre-commit log corresponding to the transaction commit instruction as a transaction change log;

If the transaction commit instruction is a two-stage commit instruction, generating a two-stage commit log corresponding to the transaction commit instruction as a transaction change log;

if the transaction commit instruction is a two-stage rollback instruction, generating a two-stage rollback log corresponding to the transaction commit instruction as a transaction change log;

caching the transaction change log to a corresponding log cache area;

the transaction consistency management system comprises:

the first determining module is used for determining a target security time stamp according to the log type of the transaction change log in each log buffer;

and the second determining module is used for determining target increment data corresponding to the transaction processing system according to the log type of the transaction change log in each log buffer zone and combining the target security time stamp, and synchronizing the target increment data to the analysis processing system.

2. The engine of claim 1, wherein the transaction processing system further comprises a processing service process, the processing service process comprising:

the first subprocess is used for generating a transaction commit instruction according to the received transaction commit request;

the second subprocess is used for generating a transaction commit instruction according to the received response success messages sent by the sub-databases;

And the sending subprocess is used for sending the transaction commit instruction to each of the sub-databases.

3. The engine of claim 2, wherein the first sub-process is specifically configured to:

determining a fractional database associated with the transaction commit request according to the received transaction commit request;

if the number of the sub-databases is one, generating a one-stage commit instruction as a transaction commit instruction;

if the number of the sub-databases is at least two, generating a two-stage pre-commit instruction as a transaction commit instruction;

and sending the transaction commit instruction to the fractional database.

4. The engine according to claim 2, characterized in that said second sub-process is specifically configured to:

receiving a response success message sent by each sub-database to the two-stage pre-commit instruction;

judging whether response success information of each sub-database is received within set time;

if yes, acquiring a current timestamp as a transaction commit timestamp, and generating a two-stage commit instruction as a transaction commit instruction according to the transaction commit timestamp;

otherwise, a two-stage rollback instruction is generated as a transaction commit instruction.

5. The engine of claim 1, wherein the first determination module comprises:

The first determining unit is used for determining a security time stamp corresponding to each log buffer according to the log type of the transaction change log in the log buffer;

and the second determining unit is used for determining the security timestamp with the smallest value in the security timestamps as the target security timestamp.

6. The engine according to claim 5, characterized in that said first determining unit is specifically configured to:

aiming at each log buffer area, acquiring the log type of the current change log in the log buffer area;

if the current change log is a one-stage commit log or a two-stage rollback log, the security timestamp is not updated;

if the current change log is a two-stage commit log, taking a transaction commit timestamp and a larger value in the secure timestamp contained in the current change log as a new secure timestamp;

if the current change log is a two-stage pre-submitted log, judging whether the log buffer zone contains a two-stage submitted log or a two-stage rollback log which belongs to the same transaction with the current change log;

if the two-stage commit log is included, not updating the secure timestamp;

If the two-stage rollback log is included, the security time stamp is not updated and the processing is finished;

and taking the next transaction change log in the log buffer area as a new current change log, and returning to continuously executing the updating step of the security timestamp corresponding to the log buffer area until the transaction change log in the log buffer area is traversed and ended.

7. The engine of claim 1, wherein the second determination module comprises:

a third determining unit, configured to determine, for each log buffer, newly added data corresponding to the log buffer according to a log type of a transaction change log in the log buffer and in combination with the target security timestamp;

and a fourth determining unit, configured to aggregate each new increment data as target increment data corresponding to the transaction processing system, and synchronize the target increment data to the sub-database.

8. The engine according to claim 7, characterized in that said third determining unit is specifically configured to:

if the current change log is a one-stage commit log, determining incremental data corresponding to the current change log according to the content contained in the current change log;

If the current change log is a two-stage commit log or a two-stage rollback log, determining that the current change log has no corresponding newly added data;

if the current change log is a two-stage pre-submitted log, determining incremental data corresponding to the current change log according to the log type of other transaction change logs contained in the log buffer and the target security timestamp;

and removing the current change log from the log buffer area, taking the next transaction change log in the log buffer area as a new current change log, and returning to the determination step of continuously executing the new data corresponding to the current change log until the transaction-free change log in the log buffer area is finished.

9. The engine according to claim 8, wherein the third determining unit is configured to perform the step of determining the incremental data corresponding to the current change log in combination with the target security timestamp according to the log type of the other transaction change log included in the log buffer, and includes:

acquiring other transaction change logs contained in the log buffer;

judging whether the other transaction change logs belong to two-stage commit logs or two-stage rollback logs of the same transaction as the current change log;

If the log is rolled back in two stages, determining that the current change log has no corresponding newly added data;

if yes, judging whether the transaction commit time stamp contained in the two-stage commit log is smaller than or equal to the target security time stamp;

if yes, determining incremental data corresponding to the current change log according to the content contained in the current change log;

otherwise, the process is ended.

10. A method of transaction consistency management, performed by the transaction consistency management engine of any of claims 1-9, the transaction consistency management engine comprising a transaction processing system, an analysis processing system, and a transaction consistency management system, the transaction processing system including at least one sub-database, each of the sub-databases corresponding to a log buffer, the method comprising:

determining target incremental data corresponding to the transaction processing system by the transaction consistency management system according to the log types of transaction change logs in each log buffer area, and synchronizing the target incremental data to the analysis processing system;

The generating, by the sub database, a transaction change log including a log type according to a received transaction commit instruction, and caching the transaction change log to a corresponding log buffer area, includes:

caching the transaction change log to a corresponding log cache area;

the determining, by the transaction consistency management system, target incremental data corresponding to the transaction processing system according to log types of transaction change logs in each log buffer, and synchronizing the target incremental data to the analysis processing system includes:

Determining a target security time stamp according to the log type of the transaction change log in each log buffer;

and determining target increment data corresponding to the transaction processing system according to the log type of the transaction change log in each log buffer zone and combining the target security time stamp, and synchronizing the target increment data to the analysis processing system.