WO2022017347A1 - 分布式数据库系统及数据处理方法 - Google Patents

分布式数据库系统及数据处理方法 Download PDF

Info

Publication number
WO2022017347A1
WO2022017347A1 PCT/CN2021/107243 CN2021107243W WO2022017347A1 WO 2022017347 A1 WO2022017347 A1 WO 2022017347A1 CN 2021107243 W CN2021107243 W CN 2021107243W WO 2022017347 A1 WO2022017347 A1 WO 2022017347A1
Authority
WO
WIPO (PCT)
Prior art keywords
transaction
target
storage module
target transaction
execution
Prior art date
Application number
PCT/CN2021/107243
Other languages
English (en)
French (fr)
Inventor
孙梦石
楼江航
傅宇
吴学强
Original Assignee
阿里巴巴集团控股有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 阿里巴巴集团控股有限公司 filed Critical 阿里巴巴集团控股有限公司
Publication of WO2022017347A1 publication Critical patent/WO2022017347A1/zh
Priority to US18/101,049 priority Critical patent/US20230161758A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/466—Transaction processing

Definitions

  • This specification relates to the technical field of distributed databases, and in particular, to a distributed database system and data processing method, a computing device, and a computer-readable storage medium.
  • a distributed database consists of multiple storage nodes.
  • each DML (Data Manipulation Language) write operation will independently generate a binary log (binlog).
  • binary logs are stored in the storage node where they are located. stay in order.
  • two operations before and after time may occur on two different storage nodes, directly subscribing to the binary logs of the two storage nodes may cause the order of these two operations to be out of order, and eventually The data is inconsistent.
  • the embodiments of this specification provide a distributed database system.
  • This specification also relates to a data processing method, a computing device, and a computer-readable storage medium, so as to solve the technical defects existing in the prior art.
  • a distributed database system including: a computing module and a storage module;
  • the computing module is configured to receive a transaction opening request; determine a target transaction and a target storage module corresponding to the target transaction according to the transaction opening request; send an execution instruction to the target storage module, where the execution instruction carries The target transaction is to be executed; when the target storage module returns a message that the target transaction is successfully executed, a submission instruction is sent to the target storage module, and the submission instruction carries the target transaction to be submitted and the submission timestamp of the target transaction;
  • the target storage module is configured to receive the execution instruction, and execute the target transaction according to the execution instruction; if the target transaction is successfully executed, return a message that the target transaction is successfully executed to the computing module; receive The commit instruction commits the target transaction according to the commit instruction, and generates a binary log according to the commit timestamp of the target transaction.
  • system further includes: a plurality of execution modules and a sorting module, and each execution module in the plurality of execution modules corresponds to a storage module;
  • the execution module is configured to read the binary log of the storage module corresponding to itself, and generate a memory sequence according to the binary log;
  • the sorting module is configured to read the memory sequence of each execution module, and obtain the commit timestamp of the target transaction in each binary log in the memory sequence; Sort the binary log in the memory sequence described above; write the sorted binary log to the message queue.
  • the computing module is further configured to:
  • the computing module is further configured to:
  • the execution logic of the form processing event is determined.
  • the computing module is further configured to:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version
  • the first version is the version before the table processing event is executed
  • the second version is compatible with the second version.
  • the version is the version after the table processing event is executed
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version.
  • the computing module is further configured to:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version, then determine that the execution logic of the table processing event is subsequent execution;
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version, it is determined that the execution logic of the table processing event is pre-order execution.
  • the computing module is further configured to:
  • execution logic of the table processing event is pre-order execution, before generating the binary logs corresponding to all the data of the second version, obtain the timestamp of the table processing event;
  • the execution logic of the table processing event is post-sequence execution, after generating the binary logs corresponding to all the data of the first version, the timestamp of the table processing event is acquired.
  • the sorting module is further configured to:
  • a binary log of the table processing event is inserted into the message queue according to the timestamp of the table processing event.
  • system further includes a timestamp generator, and the computing module is further configured to:
  • the timestamp is determined as the commit timestamp of the target transaction.
  • the transaction opening request carries a transaction identifier and a corresponding storage module identifier
  • the computing module is further configured as:
  • the storage module indicated by the storage module identifier is determined as the corresponding target storage module.
  • the transaction opening request carries a transaction identifier
  • the computing module is further configured to:
  • the execution module is further configured to:
  • the sorting module is further configured to:
  • the computing module is further configured to:
  • the storage module identifier of the target storage module is recorded.
  • the computing module is further configured to:
  • the operation step of sending an execution instruction to the target storage module is performed.
  • the storage module is further configured to:
  • a binary log is generated, and the commit timestamp of the target transaction is recorded in a comment of the binary log.
  • a data processing method is provided, which is applied to the computing module in the distributed database system provided in the first aspect, and the method includes:
  • a commit instruction is sent to the target storage module, and the commit instruction carries the target transaction to be submitted and the commit timestamp of the target transaction.
  • the method further includes:
  • the determining the execution logic of the form processing event includes:
  • the execution logic of the form processing event is determined.
  • determining a compatible type corresponding to the table processing event according to the operation type including:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version
  • the first version is the version before the table processing event is executed
  • the second version is compatible with the second version.
  • the version is the version after the table processing event is executed
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version.
  • determining the execution logic of the table processing event according to the compatibility type includes:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version, then determine that the execution logic of the table processing event is subsequent execution;
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version, it is determined that the execution logic of the table processing event is pre-order execution.
  • generating the timestamp of the table processing event according to the execution logic includes:
  • execution logic of the table processing event is pre-order execution, before generating the binary logs corresponding to all the data of the second version, obtain the timestamp of the table processing event;
  • the execution logic of the table processing event is post-sequence execution, after generating the binary logs corresponding to all the data of the first version, the timestamp of the table processing event is acquired.
  • the method before the sending the commit instruction to the target storage module, the method further includes:
  • the timestamp is determined as the commit timestamp of the target transaction.
  • the transaction opening request carries a transaction identifier and a corresponding storage module identifier, and determining a target transaction and a target storage module corresponding to the target transaction according to the transaction opening request includes:
  • the storage module indicated by the storage module identifier is determined as the corresponding target storage module.
  • the transaction opening request carries a transaction identifier, and determining a target transaction and a target storage module corresponding to the target transaction according to the transaction opening request includes:
  • the method further includes:
  • the storage module identifier of the target storage module is recorded.
  • the method further includes:
  • the operation step of sending an execution instruction to the target storage module is performed.
  • a data processing method is provided, which is applied to the storage module in the distributed database system provided in the first aspect, and the method includes:
  • a commit instruction is received, the target transaction is committed according to the commit instruction, and a binary log is generated according to the commit timestamp of the target transaction, where the commit instruction carries the target transaction to be committed and the commit timestamp of the target transaction.
  • generating a binary log according to the commit timestamp of the target transaction includes:
  • a binary log is generated, and the commit timestamp of the target transaction is recorded in a comment of the binary log.
  • a data processing method is provided, which is applied to the sorting module in the distributed database system provided in the first aspect, and the method includes:
  • the method further includes:
  • the operation step of reading the memory sequence of each execution module is returned to be executed.
  • the method further includes:
  • the binary log added in the execution module is acquired, and the binary log is placed at the end of the message queue.
  • the method further includes:
  • the binary log of the table processing event is inserted into the message queue according to the timestamp of the table processing event.
  • a computing device including:
  • the memory is used to store computer-executable instructions
  • the processor is used to execute the computer-executable instructions:
  • a commit instruction is sent to the target storage module, and the commit instruction carries the target transaction to be submitted and the commit timestamp of the target transaction.
  • a computing device including:
  • the memory is used to store computer-executable instructions
  • the processor is used to execute the computer-executable instructions:
  • a commit instruction is received, the target transaction is committed according to the commit instruction, and a binary log is generated according to the commit timestamp of the target transaction, where the commit instruction carries the target transaction to be committed and the commit timestamp of the target transaction.
  • a computing device including:
  • the memory is used to store computer-executable instructions
  • the processor is used to execute the computer-executable instructions:
  • a computer-readable storage medium which stores computer-executable instructions, and when the instructions are executed by a processor, implements the steps of any of the data processing methods in the second aspect.
  • a computer-readable storage medium which stores computer-executable instructions, and when the instructions are executed by a processor, implements the steps of any of the data processing methods described in the third aspect.
  • a computer-readable storage medium which stores computer-executable instructions, and when the instructions are executed by a processor, implements the steps of any of the data processing methods described in the fourth aspect.
  • the distributed database system includes a computing module and a storage module, the computing module is configured to receive a transaction opening request; determine a target transaction and a target storage module corresponding to the target transaction according to the transaction opening request;
  • the target storage module sends an execution instruction; when the target storage module returns a message that the execution of the target transaction is successful, a submission instruction is sent to the target storage module, and the submission instruction carries the target transaction to be submitted and the submission of the target transaction timestamp;
  • the target storage module is configured to receive the execution instruction, and execute the target transaction according to the execution instruction; if the target transaction is successfully executed, return a message that the target transaction is successfully executed to the computing module ; Receive the commit instruction, commit the target transaction according to the commit instruction, and generate a binary log according to the commit timestamp of the target transaction.
  • each storage module is a binary log generated according to the commit timestamp of the target transaction sent by the computing module, and each binary log carries a globally ordered timestamp, so that the two data in the distributed database system can be Sub-operations can be arranged in order to avoid confusion in the order of operations.
  • FIG. 1 is a structural block diagram of a distributed database system provided by an embodiment of this specification
  • FIG. 2 is a flowchart of a first data processing method provided by an embodiment of this specification
  • FIG. 3 is a schematic diagram of a pre-order execution and a post-order execution provided by an embodiment of the present specification
  • FIG. 5 is a flowchart of a third data processing method provided by an embodiment of this specification.
  • FIG. 6 is a flowchart of a fourth data processing method provided by an embodiment of this specification.
  • FIG. 7 is a schematic diagram of a binary log generation process provided by an embodiment of this specification.
  • FIG. 8 is a schematic diagram of a sorting process provided by an embodiment of the present specification.
  • FIG. 9 is a structural block diagram of a first computing device provided by an embodiment of this specification.
  • FIG. 10 is a structural block diagram of a second computing device provided by an embodiment of this specification.
  • FIG. 11 is a structural block diagram of a third computing device provided by an embodiment of this specification.
  • SQL Structured Query Language
  • DDL Data Definition Language
  • DML Data Manipulation Language
  • DCL Data Control Language
  • TCL Transaction Control Language
  • Transaction It is the basic unit of recovery and concurrency control. It is an execution unit composed of multiple SQL statements. It has four characteristics of atomicity, consistency, isolation and durability. Atomicity means that a transaction is a An indivisible unit of work, the operations included in the transaction are either done or not done; Consistency means that the transaction must change the database from one consistent state to another consistent state, consistency and atomicity It is closely related; isolation means that the execution of a transaction cannot be interfered with by other transactions, that is, the operations and data used within a transaction are isolated from other concurrent transactions, and concurrently executed transactions cannot interact with each other. Interference; Durability, also known as permanence, means that once a transaction is committed, its changes to the data in the database should be permanent, and subsequent operations or failures should not have any impact on it.
  • MySQL It is a relational database management system. With the continuous maturity of MySQL, it is gradually used in more large-scale websites and applications.
  • PolarDB It is a new generation of relational cloud-native database. It not only has the low-cost advantage of distributed design, but also has centralized ease of use. It realizes the separation of computing modules and storage modules, and provides immediate scalability and operation and maintenance. Ability, software and hardware integrated design, to meet the needs of large-scale application scenarios.
  • PolarDB-X The distributed version of PolarDB, a distributed database based on MySQL sharding (database sharding), mainly composed of a storage module (MySQL server) and a computing module.
  • TSO A centralized timestamp generator that provides globally ordered timestamp services.
  • binlog A log file in binary format, used to record changes to the database within Mysql (only record changes to data), mainly used for master-slave replication and incremental recovery of the database.
  • MySQL officially supports three binlog data formats, the corresponding parameter is binlog_format, the optional values are ROW, STATEMENT, MIXED, and the corresponding default value is MIXED, which needs to be adjusted to ROW mode.
  • MIXED MIXED
  • the values of the rows affected by SQL before and after the change will be recorded in the binlog.
  • every DML (addition, deletion, modification, etc., including DDL) operation will generate a binlog (binary log) log. This log is ordered according to the operation time in MySQL, and downstream applications can do some subscription operations based on this log. (eg data synchronization).
  • Physical binlog The binlog of each storage module of PolarDB-X.
  • Logical binlog The binlog provided by the PolarDB-X layer, including the binlog of all storage modules, is displayed externally as a Kafka message queue.
  • Commit command A commit command is used to save the changes made by the transaction to the database.
  • commit_timestamp used to indicate the commit time of the transaction, abbreviated commit_ts.
  • Jingwei worker A module that executes work, corresponds to the storage module one-to-one, is responsible for reading the binlog file of the corresponding storage module, and is responsible for parsing, and at the same time generates a memory sequence from the parsing result.
  • Jingwei merger It is equivalent to a merge sort module, which merges multiple ordered queues into a global ordered queue.
  • each storage node will generate a binary log independently after executing a transaction (DML-related operations).
  • the binary log generated by each storage node is kept in order within the storage node.
  • the time sequence between the generated binary logs is not certain. Therefore, in a distributed database, two operations before and after time may occur on two different storage nodes, and downstream applications directly subscribe to the binary logs of the two storage nodes. , which may cause the order of these two operations to be out of order, resulting in inconsistent final data of downstream applications.
  • the distributed database system includes MySQL1, MySQL2 and MySQL3.
  • Two binary logs are generated in MySQL1: log A1 and log A2, and three binary logs are generated in MySQL2: log B1, log B2 and log B3, which are generated in MySQL3.
  • log A1 is earlier than log A2
  • log B1 is earlier than log B2 and earlier than log B3
  • log C1 is earlier than log C2
  • the sequence between C1 and log C2 is not certain, and there may be confusion, resulting in different data subscribed by downstream applications.
  • This specification provides a distributed database system, including a computing module and a storage module (such as MySQL).
  • the computing module controls the storage module to execute the target transaction. After the storage module executes successfully, it returns a success message to the computing module. At this time, the computing module will send a message to the computing module.
  • the storage module sends the commit timestamp of the target transaction, so that the storage module generates a binary log according to the timestamp. In this case, the binary log records the commit timestamp of the globally ordered target transaction, so that all All binary logs are in chronological order, thus avoiding disorder in the order of operations of transactions and ensuring the consistency of subscription data for downstream applications.
  • a distributed database system is provided, and this specification also relates to a data processing method, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
  • FIG. 1 shows a structural block diagram of a distributed database system according to an embodiment of the present specification, which specifically includes a computing module 102 and a storage module 104 .
  • the computing module 102 is configured to receive a transaction opening request; determine a target transaction and a storage module 104 corresponding to the target transaction according to the transaction opening request; send an execution instruction to the storage module 104, in which the execution instruction is Carrying the target transaction to be executed; when the storage module 104 returns a message that the target transaction is successfully executed, a commit instruction is sent to the storage module 104, and the commit instruction carries the target transaction to be submitted and the commit timestamp of the target transaction ;
  • the storage module 104 is configured to receive the execution instruction, and execute the target transaction according to the execution instruction; if the target transaction is successfully executed, return a message that the target transaction is successfully executed to the computing module 102; The commit instruction is received, the target transaction is committed according to the commit instruction, and a binary log is generated according to the commit timestamp of the target transaction.
  • system further includes: a plurality of execution modules and a sorting module, and each execution module of the plurality of execution modules corresponds to a storage module 104;
  • the execution module is configured to read the binary log of the storage module 104 corresponding to itself, and generate a memory sequence according to the binary log;
  • the sorting module is configured to read the memory sequence of each execution module, and obtain the commit timestamp of the target transaction in each binary log in the memory sequence; Sort the binary log in the memory sequence described above; write the sorted binary log to the message queue.
  • the computing module is further configured to:
  • the computing module is further configured to:
  • the execution logic of the form processing event is determined.
  • the computing module is further configured to:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version
  • the first version is the version before the table processing event is executed
  • the second version is compatible with the second version.
  • the version is the version after the table processing event is executed
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version.
  • the computing module is further configured to:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version, then determine that the execution logic of the table processing event is subsequent execution;
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version, it is determined that the execution logic of the table processing event is pre-order execution.
  • the computing module is further configured to:
  • execution logic of the table processing event is pre-order execution, before generating the binary logs corresponding to all the data of the second version, obtain the timestamp of the table processing event;
  • the execution logic of the table processing event is post-sequence execution, after generating the binary logs corresponding to all the data of the first version, the timestamp of the table processing event is acquired.
  • the sorting module is further configured to:
  • a binary log of the table processing event is inserted into the message queue according to the timestamp of the table processing event.
  • system further includes a timestamp generator, and the computing module 102 is further configured to:
  • the timestamp is determined as the commit timestamp of the target transaction.
  • the transaction opening request carries a transaction identifier and a corresponding storage module identifier
  • the computing module 102 is further configured as:
  • the storage module indicated by the storage module identifier is determined as the corresponding storage module 104 .
  • the transaction opening request carries a transaction identifier
  • the computing module 102 is further configured to:
  • the storage module 104 corresponding to the target transaction is determined.
  • the execution module is further configured to:
  • the sorting module is further configured to:
  • the computing module 102 is further configured to:
  • the storage module identifier of the storage module 104 is recorded.
  • the computing module 102 is further configured to:
  • the storage module 104 is further configured to:
  • a binary log is generated, and the commit timestamp of the target transaction is recorded in a comment of the binary log.
  • the distributed database system includes a computing module and a storage module, the computing module is configured to receive a transaction opening request; determine a target transaction and a target storage module corresponding to the target transaction according to the transaction opening request;
  • the target storage module sends an execution instruction; when the target storage module returns a message that the execution of the target transaction is successful, a submission instruction is sent to the target storage module, and the submission instruction carries the target transaction to be submitted and the submission of the target transaction timestamp;
  • the target storage module is configured to receive the execution instruction, and execute the target transaction according to the execution instruction; if the target transaction is successfully executed, return a message that the target transaction is successfully executed to the computing module ; Receive the commit instruction, commit the target transaction according to the commit instruction, and generate a binary log according to the commit timestamp of the target transaction.
  • each storage module is a binary log generated according to the commit timestamp of the target transaction sent by the computing module, and each binary log carries a globally valid timestamp, so that two times before and after the time in the distributed database system
  • the binary logs generated by the operation can be arranged in order, thus avoiding disorder of the operation order of the transaction and ensuring the consistency of the subscription data of downstream applications.
  • FIG. 2 shows a flowchart of a data processing method provided according to an embodiment of the present specification, which is applied to the computing module in the distributed database system shown in FIG. 1 , and specifically includes the following steps:
  • Step 202 Receive a transaction start request.
  • the application program can send a transaction opening request to the computing module in the distributed database system, and the computing module receives the transaction opening request, thereby enabling subsequent execution. the entire process of the transaction.
  • the distributed database system here may refer to PolarDB-X or other distributed databases, which are not limited in this specification.
  • Step 204 Determine a target transaction and a target storage module corresponding to the target transaction according to the transaction opening request.
  • a target transaction and a target storage module corresponding to the target transaction are determined.
  • the target transaction refers to a transaction that is indicated to be opened by a transaction opening request
  • the target storage module refers to a storage module involved in executing the target transaction.
  • the storage module here may be a monolithic database in a distributed database system, such as a monolithic MySQL.
  • the computing module After the computing module receives a transaction opening request, it first needs to determine the specific transaction that the request instructs to execute. Since the transaction is ultimately executed by the storage module, the computing module needs to further determine the storage module that executes the transaction. Since a transaction is an execution unit composed of multiple SQL statements, a transaction may include multiple SQL operations, and the multiple operations may be executed by different storage modules. Therefore, a transaction may involve multiple storage modules. That is, there may be multiple target storage modules determined.
  • a user when a user wants to execute a transaction in the distributed database system, he can directly specify a storage module to execute the transaction, and the application program will identify the transaction corresponding to the transaction to be executed. , and the identifier of the storage module that executes the transaction is carried in the transaction opening request and sent to the computing module, and the computing module can directly determine the target transaction and the corresponding storage module identifier according to the transaction identifier and the corresponding storage module identifier carried in the received transaction opening request.
  • the specific implementation process is as follows:
  • the storage module indicated by the storage module identifier is determined as the corresponding target storage module.
  • the transaction identifier is used to identify the transaction to be executed
  • the storage module identifier is used to identify the storage module that executes the transaction.
  • the user wants to execute transaction A in the distributed database system, and the user specifies that the storage module 1 and the storage module 2 should execute the transaction A.
  • the user can use an application program to execute the transaction A in the distributed database system.
  • Send a transaction opening command which will carry transaction A, storage module 1, and storage module 2.
  • the computing module After the computing module receives the transaction opening instruction, it can determine that the target transaction to be executed is transaction A, and the storage modules that execute transaction A are storage module 1 and storage module 2 .
  • the transaction opening request directly carries the transaction identifier and the corresponding storage module identifier, and the computing module can directly determine the target transaction and the target storage module corresponding to the target transaction according to the identifier.
  • the determination process is simple and easy, and does not require the computing module to perform complex operations. , which saves computing resources to a certain extent.
  • the user when the user wants to execute a certain transaction in the distributed database system, the user may not specify the storage module for executing the transaction, and the computing module can perform the transaction according to the performance attributes, load, etc. of the storage module.
  • the parameter specifies the corresponding storage module to execute the target transaction.
  • the application program only carries the transaction identifier corresponding to the transaction to be executed in the transaction opening request and sends it to the calculation module. After the calculation module determines the target transaction to be executed, it can specify the execution
  • the specific implementation process of the target storage module of the target transaction is as follows:
  • the preset allocation rules refer to the rules for allocating storage modules for the target transaction to be executed.
  • the rules are set in advance by technicians, which may be load balancing rules or storage module performance rules. This specification does not contain allocation rules. It can be specified as long as it can allocate a reasonable storage module to the target transaction.
  • the preset allocation rule is load balancing, and it is assumed that the distributed database system includes a storage module 1 and a storage module 2, the transactions being executed or to be executed by the storage module 1 include transaction A and transaction B, and the storage module 2 is executing or to be executed.
  • the transaction has transaction C.
  • a user sends a transaction opening instruction to the computing module in the distributed database system through an application program, and the instruction carries transaction D.
  • the computing module determines that the target transaction to be executed is a transaction D. Then, according to the load balancing rule, it is determined that the storage module that executes the transaction D is the storage module 2 .
  • the calculation module determines the target transaction, it allocates the corresponding target storage module for the target transaction according to the preset allocation rules, and can flexibly specify the execution transaction according to parameters such as performance attributes and load.
  • the storage module can adapt to the actual situation of each storage module in the distributed database system, ensuring that the transaction can be executed efficiently and accurately, and the adaptability is higher.
  • the target transaction and the target storage module for executing the target transaction can be directly specified in the transaction opening request, or the storage module to be executed by the calculation module is assigned to the target transaction, which is more flexible and can be adapted to different transactions. Execution features, so that the target transaction can be executed more efficiently and accurately.
  • Step 206 Send an execution instruction to the target storage module, where the execution instruction carries the target transaction to be executed.
  • an execution instruction is sent to the target storage module, and the execution instruction carries the information to be executed. target transaction.
  • the target storage module does not know that it has been selected to start executing the target transaction, so according to the The transaction opening request, after determining the target transaction and the target storage module corresponding to the target transaction, and before sending the execution instruction to the target storage module, the computing module can also notify the target storage module to open the target transaction.
  • the specific implementation process is as follows :
  • the operation step of sending an execution instruction to the target storage module is performed.
  • the computing module after the subsequent target storage module executes the target transaction, the computing module also needs to send a commit instruction to the target storage module, so that the target storage module submits the target transaction. After the target storage module sends the execution instruction, it is also necessary to record the storage module identifier of the target storage module, so as to facilitate the subsequent sending of the commit instruction to the target storage module.
  • Step 208 When the target storage module returns a message that the target transaction is successfully executed, a commit instruction is sent to the target storage module, and the commit instruction carries the target transaction to be submitted and the commit timestamp of the target transaction.
  • the target storage module on the basis of sending the execution instruction to the target storage module, further, when the target storage module returns a message that the target transaction is successfully executed, a commit instruction is sent to the target storage module, the The commit instruction carries the target transaction to be committed and the commit timestamp of the target transaction.
  • the target storage module After the target storage module receives the execution instruction, it will execute the target transaction. If the target transaction is successfully executed, it will return a successful execution message to the computing module to notify the computing module that it has successfully executed the target transaction. ready to submit. After the computing module receives the successful execution message, it can control the storage module to commit the target transaction.
  • the user makes the distributed database system execute the corresponding target transaction through the application program, and the target storage module returns a successful execution message to the computing module when the target transaction is successfully executed, Further, at this time, the computing module can also return to the application a message that the execution of the target transaction is completed, and the application will return a transaction commit message to the computing module, so that the computing module controls the target storage module to commit the transaction.
  • the computing module in order to make the binary logs generated when each storage module submits a transaction in chronological order, when the computing module sends a submission instruction to the target storage module, the instruction It carries the commit timestamp of the target transaction, so before sending the commit instruction to the target storage module, the computing module can also obtain the commit timestamp of the globally ordered target transaction.
  • the specific implementation is as follows:
  • the timestamp is determined as the commit timestamp of the target transaction.
  • the timestamp generator may be a TSO in a distributed database system, and the commit timestamp of the target transaction obtained from the TSO is commit_timestamp.
  • the computing module may also receive DDL events, such as changing column types, lengths, and adding constraints.
  • DDL events such as changing column types, lengths, and adding constraints.
  • the computing module may also receive DDL events, such as changing column types, lengths, and adding constraints.
  • DDL change process of a distributed database system at the same time, there will be two versions of the schema metadata of the database. For example, when adding a column, the new column is visible to some shards, but not yet visible to some shards. Generate two versions of the data at the same time.
  • the table processing event refers to a DDL event, for example, events such as adding a column, increasing the length of a column, increasing the default value of a column, deleting a column, and shortening the length of a column.
  • the execution logic of the table processing event is determined, and the specific implementation process may be:
  • the execution logic of the form processing event is determined.
  • the operation type is set, and different operation types correspond to different compatible types.
  • the compatibility type corresponding to the table processing event is determined according to the operation type, and the specific implementation process may be as follows:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version
  • the first version is the version before the table processing event is executed
  • the second version is compatible with the second version.
  • the version is the version after the table processing event is executed
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version.
  • the schema structure (including the table structure, etc.) at a moment is called a version
  • the version before the execution of the DDL event is called the first version
  • the version after the DDL event is executed is called the second version.
  • the Va version is compatible with the Vb version means: the data using the Vb version can be written to the target using the Va version; after the data is read from the target using the Va version, it contains enough information. After a certain conversion, it can be converted to Vb version data, which is completely equal to the original data.
  • the first type is the operation type that the first version is compatible with the second version.
  • operations such as deleting a column, shortening the length of a column, deleting a table, and reducing the precision of a column are all of the first type.
  • the second type is the operation type that the second version is compatible with the first version, for example, adding a column, increasing the length of the column, increasing the default value of the column, creating and deleting indexes, creating and deleting constraints (creating unique key constraints), Operations such as building a table and increasing the precision of a column are all of the second type.
  • the execution logic of the table processing event is determined according to the compatibility type, and the specific implementation process may be:
  • the compatibility type corresponding to the table processing event is that the first version is compatible with the second version, then determine that the execution logic of the table processing event is subsequent execution;
  • the compatibility type corresponding to the table processing event is that the second version is compatible with the first version, it is determined that the execution logic of the table processing event is pre-order execution.
  • the timestamp of the table processing event is obtained before the binary logs corresponding to all the data of the second version are generated; if the table processing event is executed If the logic is post-sequence execution, after generating all the binary logs corresponding to the data of the first version, the timestamp of the table processing event is obtained.
  • the pre-order execution refers to the binary log of the DDL event, which should be at the top of the binary log of all the second version (V2) data, thus generating all the second version (V2) data.
  • Post-order execution refers to the binary log of the DDL event, which should be at the end of the binary log of all the first version (V1) data. Therefore, after generating the binary log corresponding to all the first version (V1) data, a timestamp is obtained. , and use the timestamp as the timestamp of the DDL event, that is, the DDL event should be located after all the binary logs of the first version (V1).
  • the first version can be compatible with the second version, and the second version can also be compatible with the first version.
  • the execution logic of the table processing event can be executed in a subsequent sequence. , or preorder execution.
  • a timestamp of the table processing event is generated according to the execution logic of the table processing event, and then a binary log of the table processing event can be generated according to the timestamp, so that the table Binary logs for processing events can be in place to ensure consistency of subscription data for downstream applications.
  • the computing module receives a transaction opening request, and according to the transaction opening request, determines a target transaction and a target storage module corresponding to the target transaction; then sends an execution instruction to the target storage module; When the module returns a message that the target transaction is successfully executed, a commit instruction is sent to the target storage module, and the commit instruction carries the target transaction to be submitted and the commit timestamp of the target transaction.
  • the computing module can apply to the timestamp generator for a globally ordered timestamp, use the timestamp as the commit timestamp of the target transaction, and send it to the target storage module, so that the subsequent target storage module can generate a global ordered timestamp.
  • Binary logs with ordered timestamps so that the binary logs generated by two operations before and after in time in the distributed database system can be arranged in order, avoiding the disorder of the operation sequence of transactions, and ensuring the consistency of subscription data of downstream applications.
  • FIG. 4 shows a flowchart of a data processing method provided according to an embodiment of the present specification, which is applied to the storage module in the distributed database system shown in FIG. 1 , and specifically includes the following steps:
  • Step 402 Receive an execution instruction, and execute a target transaction according to the execution instruction, where the execution instruction carries the target transaction to be executed.
  • the target storage module itself does not know that it is selected to start executing the target transaction. Therefore, before the target storage module receives the execution instruction, it will also After receiving the target transaction opening request sent by the computing module, when the target storage module receives the target transaction opening request, it will start the target transaction and notify the computing module that it is ready to execute the target transaction. Before executing the target transaction according to the execution instruction, the target storage module should also return an open message to the computing module in the case of receiving the target transaction open request.
  • Step 404 In the case that the execution of the target transaction is successful, a message that the execution of the target transaction is successful is returned to the computing module.
  • a message that the execution of the target transaction is successful is returned to the computing module.
  • Step 406 Receive a commit instruction, submit the target transaction according to the commit instruction, and generate a binary log according to the commit timestamp of the target transaction, where the commit instruction carries the target transaction to be submitted and the commit of the target transaction timestamp.
  • a submission instruction will be received, and the target transaction will be submitted according to the submission instruction, and A binary log is generated according to the commit timestamp of the target transaction.
  • the binary log is the binlog
  • the binary log generated by each storage module is called the physical binlog
  • a binary log is generated according to the commit timestamp of the target transaction
  • a specific implementation process may be: generating a binary log, and recording the commit timestamp of the target transaction in the in the comments of the binary log.
  • the target storage module receives the execution instruction, and executes the target transaction according to the execution instruction; and when the target transaction is successfully executed, it returns to the computing module a message that the target transaction is successfully executed; then receive A commit instruction, the target transaction is committed according to the commit instruction, and a binary log is generated according to the commit timestamp of the target transaction.
  • each storage module is a binary log generated according to the commit timestamp of the target transaction sent by the computing module, and each binary log carries a globally ordered timestamp, which makes the time in the distributed database system
  • the binary logs generated by the two operations before and after can be arranged in order, so as to avoid the disorder of the operation order of the transaction and ensure the consistency of the subscription data of the downstream application.
  • FIG. 5 shows a flowchart of a data processing method provided according to an embodiment of the present specification, which is applied to a sorting module in a distributed database system, and specifically includes the following steps:
  • Step 502 Read the memory sequence of each execution module.
  • each execution module corresponds to a storage module. Before the sorting module reads the memory sequence of each execution module, the execution module can also read the binary log of its corresponding storage module, and according to the binary log Generate memory sequences.
  • the execution module here can refer to the Jingwei worker generated by PolarDB-X, a module that executes work.
  • the sorting module can refer to the Jingwei merger generated by PolarDB-X, which is used to manage and control the generated Jingwei workers.
  • a merge-sort module can merge multiple ordered queues into a globally ordered queue.
  • Each execution module in this manual generates a memory sequence, and the memory sequence includes all binary logs generated by its corresponding storage module.
  • the memory sequence includes all binary logs generated by its corresponding storage module.
  • Step 504 Obtain the commit timestamp of the target transaction in each binary log in the memory sequence.
  • the commit timestamp of the target transaction in each binary log in the memory sequence is obtained.
  • Step 506 Sort the binary logs in the memory sequence according to the commit timestamp of the target transaction.
  • Step 508 Write the sorted binary log into the message queue.
  • the sorted binary logs are further written into the message queue.
  • the message queue can refer to the kafka message queue, which is an open source message queue.
  • the sorted binary log is written into the message queue, that is, the logical binlog of the distributed database system is generated, and downstream applications need to subscribe to the distribution.
  • the logical binlog in the open source message queue can be directly read, and the binary logs of the storage modules in the distributed database system can be obtained, and the binary logs are arranged in order in the message queue. .
  • the storage module may execute a new transaction to generate a new binary log, so the execution module detects its corresponding storage module In the case of generating a new binary log, you can return to read the binary log of the corresponding storage module, and generate the memory sequence according to the operation steps of the binary log. At this time, the sorting module can reorder all the binary logs.
  • the specific implementation The process is as follows:
  • the operation step of reading the memory sequence of each execution module is returned to be executed.
  • the storage module may execute a new transaction to generate a new binary log, so the execution module detects its corresponding storage module In the case of generating a new binary log, you can return to read the binary log of the corresponding storage module, and generate the memory sequence according to the operation steps of the binary log, because the newly generated binary log must be later than the already sorted in time. Good binary log, so the sorting module can directly put the binary log added in the execution module into the tail of the message queue.
  • the storage module executes a new transaction and generates a new binary log
  • all binary logs can be reordered, or the newly generated binary log can be directly placed at the end of the message queue to achieve
  • the sorting method is flexible and can be applied to different situations, making the sorting result more efficient and accurate.
  • the computing module if it detects the table processing event, it can generate a binary log of the table processing event according to the time stamp of the table processing event.
  • the sorting module can process the table according to the time stamp of the table processing event. The timestamp of the event, the binary log of the table processing event is inserted into the message queue.
  • a timestamp of the form processing event can be generated according to the execution logic of the form processing event, and then a binary log of the form processing event can be generated according to the timestamp, and can be generated according to the timestamp.
  • the time stamp of the table processing event is inserted into the message queue, so that the table processing event can be placed in an appropriate position in the message queue to ensure that the downstream system can correctly receive the table processing event before and after execution. This version of data avoids the inconsistency of the same piece of data in the downstream system.
  • the sorting module can read the memory sequence of each execution module, and obtain the commit timestamp of the target transaction in each binary log in the memory sequence; then according to the commit timestamp of the target transaction , sort the binary logs in the memory sequence, and write the sorted binary logs into the message queue.
  • the message queue includes binary logs generated by all storage modules in the distributed database system, and they are arranged in chronological order. When downstream applications need to subscribe to the data in the distributed database, they can directly read the binary logs. The message queue can avoid the disorder of the operation sequence of the transaction and ensure the consistency of the subscription data.
  • FIG. 6 shows a processing flow chart of a data processing method provided by an embodiment of this specification, which specifically includes the following steps:
  • Step 602 The computing module receives the transaction opening request sent by the application, and determines a target transaction and a target storage module corresponding to the target transaction according to the transaction opening request.
  • a transaction when a transaction includes multiple operations, when the application sends a transaction opening request to the computing module, it can be sent multiple times, and the identifiers of multiple operations are carried in multiple requests, so the computing module
  • the transaction opening request may be received in multiple times, and each request carries an operation included in the transaction.
  • the target transaction is transaction A, including operation 1 and operation 2.
  • the computing module will first receive a request carrying the identifier of operation 1, and then control the corresponding storage module to execute operation 1. After operation 1 is completed, the computing module will It will receive another request carrying the identifier of operation 2, and then control the corresponding storage module to execute operation 2. After the execution is completed, the entire transaction A will be submitted.
  • Step 604 The computing module sends a target transaction opening request to the target storage module.
  • Step 606 The target storage module receives the target transaction open request, opens the target transaction, and returns an open message to the computing module.
  • Step 608 The computing module sends an execution instruction to the target storage module, and records the storage module identifier of the target storage module, and the execution instruction carries the target transaction to be executed.
  • Step 610 The target storage module receives the execution instruction, executes the target transaction according to the execution instruction, and returns a message that the target transaction is successfully executed to the computing module if the target transaction is successfully executed.
  • Step 612 The computing module returns to the application a message that the target transaction is successfully executed, and receives a commit instruction sent by the application.
  • Step 614 The computing module applies to the timestamp generator for a globally ordered timestamp, determines the timestamp as the commit timestamp of the target transaction, and sends a commit instruction to the target storage module, where the commit instruction carries The target transaction to be committed and the commit timestamp of the target transaction.
  • Step 616 The target storage module receives the commit instruction, commits the target transaction according to the commit instruction, and generates a binary log according to the commit timestamp of the target transaction.
  • Step 618 The execution module reads the binary log of its corresponding storage module, and generates a memory sequence according to the binary log.
  • Step 620 The sorting module reads the memory sequence of each execution module, and obtains the commit timestamp of the target transaction in each binary log in the memory sequence.
  • Step 622 The sorting module sorts the binary logs in the memory sequence according to the commit timestamp of the target transaction, and writes the sorted binary logs into the message queue.
  • a transaction may include multiple SQL operations
  • the SQL operations within a transaction are inherently ordered (that is, after one SQL operation is executed, the next SQL operation will be executed), and no further operations are required. Sorting, only the sequence between different transactions may be disordered, so the commit timestamp in this specification is for the whole transaction, that is, one transaction corresponds to one commit timestamp.
  • Step 624 The computing module detects a table processing event, and if the operation type of the table processing event belongs to the first type, determines that the compatible type corresponding to the table processing event is the first version compatible with the second version, and determines that the table is compatible with the second version.
  • the execution logic of the processing event is a post-sequence execution.
  • the timestamp of the table processing event is obtained; if the operation type of the table processing event belongs to the second type, then determine The compatibility type corresponding to the table processing event is that the second version is compatible with the first version, and it is determined that the execution logic of the table processing event is pre-order execution, and before generating all binary logs corresponding to the data of the second version, obtain the The timestamp of the table processing event; according to the timestamp, a binary log of the table processing event is generated.
  • Step 626 The sorting module inserts the binary log of the table processing event into the message queue according to the time stamp of the table processing event.
  • the application program sends a transaction opening request to the computing module, and the transaction opening request indicates that the target transaction to be opened is transaction A.
  • This transaction A includes two operations: operation 1 and operation 2, and operation 1 is performed by the storage module 1, and operation 2 is performed by the storage module.
  • Module 2 executes.
  • the application program sends operation 1 to the computing module, and the computing module sends an operation 1 opening request to the storage module 1.
  • the storage module 1 After receiving the opening request, the storage module 1 returns an opening message to the computing module.
  • the computing module receives the opening message, it sends an opening message to the storage module. 1.
  • the storage module 1 executes the operation 1. After the execution of the operation 1 is successful, it returns a message that the execution of the operation 1 is successful to the computing module.
  • the computing module After the computing module receives the message that the operation 1 is successfully executed, it can also return the message that the operation 1 is successfully executed to the application program, and the further application program will send the operation 2 to the computing module, and then the computing module can send the operation 2 to the storage module 2.
  • the storage module 2 After receiving the opening request, the storage module 2 returns an opening message to the computing module.
  • the computing module After the computing module receives the opening message, it sends the execution instruction of operation 2 to the storage module 2. After the storage module 2 receives the execution instruction , and execute operation 2. After operation 2 is successfully executed, a message indicating that operation 2 is successfully executed is returned to the computing module.
  • the computing module receives the message that operation 2 is successfully executed, and returns to the application a message that transaction A is completed (that is, the message that operation 2 is successfully executed). After the application receives the message, it sends a submission instruction to the computing module.
  • the stamp generator obtains a globally ordered timestamp, and then sends the timestamp as the commit timestamp of transaction A to storage module 1 and storage module 2, and storage module 1 and storage module 2 respectively generate transaction A's timestamp according to the commit timestamp.
  • Binary log and returns a message that the binary log was successfully generated.
  • Execution module 1, execution module 2, execution module 3, and execution module 4 generate corresponding memory sequences respectively, and the sorting module reads the memory sequences of the 4 execution modules to obtain 8 binary logs, which are sorted according to time stamps to obtain binary logs.
  • Log 2 Binary Log 4, Binary Log 1, Binary Log 6, Binary Log 8, Binary Log 3, Binary Log 5, Binary Log 7, put the 8 binary logs into the message queue in the sorted order, that is, generate Logical binlog.
  • the target storage module can generate binary logs according to the commit timestamp of the target transaction, and then the sorting module sorts all binary logs according to the commit timestamp of the target transaction, and sorts the sorted binary logs.
  • the log is written to the message queue.
  • each binary log carries a globally ordered timestamp
  • the message queue will include binary logs generated by all storage modules in the distributed database system, all of which are arranged in chronological order.
  • the application needs to subscribe to the data in the distributed database, it can directly read the message queue, so that the binary logs generated by the two operations before and after the time in the distributed database system are arranged in order, thus avoiding the operation sequence of the transaction. Confusion ensures the consistency of subscription data for downstream applications.
  • FIG. 9 shows a structural block diagram of a computing device 900 according to an embodiment of the present specification.
  • Components of the computing device 900 include, but are not limited to, memory 910 and processor 920 .
  • the processor 920 is connected with the memory 910 through the bus 930, and the database 950 is used for saving data.
  • Computing device 900 also includes access device 940 that enables computing device 900 to communicate via one or more networks 960 .
  • networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet.
  • Access device 940 may include one or more of any type of network interface (eg, network interface card (NIC)), wired or wireless, such as IEEE 802.11 wireless local area network (WLAN) wireless interface, World Interoperability for Microwave Access ( Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and the like.
  • NIC network interface card
  • computing device 900 and other components not shown in FIG. 9 may also be connected to each other, such as through a bus. It should be understood that the structural block diagram of the computing device shown in FIG. 9 is only for the purpose of example, rather than limiting the scope of this specification. Those skilled in the art can add or replace other components as required.
  • Computing device 900 may be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (eg, tablet computers, personal digital assistants, laptops, notebook computers, netbooks, etc.), mobile phones (eg, smart phones) ), wearable computing devices (eg, smart watches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs.
  • Computing device 900 may also be a mobile or stationary server.
  • the processor 920 is configured to execute the following computer-executable instructions:
  • a commit instruction is sent to the target storage module, and the commit instruction carries the target transaction to be submitted and the commit timestamp of the target transaction.
  • FIG. 10 shows a structural block diagram of a computing device 1000 according to an embodiment of the present specification.
  • Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020 .
  • the processor 1020 is connected with the memory 1010 through the bus 1030, and the database 1050 is used for saving data.
  • Computing device 1000 also includes access device 1040 that enables computing device 1000 to communicate via one or more networks 1060 .
  • networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet.
  • Access device 1040 may include one or more of any type of network interface (eg, network interface card (NIC)), wired or wireless, such as IEEE 802.11 wireless local area network (WLAN) wireless interfaces, World Interoperability for Microwave Access (Wi -MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and the like.
  • NIC network interface card
  • computing device 1000 may also be connected to each other, such as through a bus.
  • bus may also be connected to each other, such as through a bus.
  • FIG. 10 the structural block diagram of the computing device shown in FIG. 10 is only for the purpose of example, rather than limiting the scope of this specification. Those skilled in the art can add or replace other components as required.
  • Computing device 1000 may be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (eg, tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (eg, smart phones) ), wearable computing devices (eg, smart watches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs.
  • Computing device 1000 may also be a mobile or stationary server.
  • the processor 1020 is configured to execute the following computer-executable instructions:
  • a commit instruction is received, the target transaction is committed according to the commit instruction, and a binary log is generated according to the commit timestamp of the target transaction, where the commit instruction carries the target transaction to be committed and the commit timestamp of the target transaction.
  • FIG. 11 shows a structural block diagram of a computing device 1100 according to an embodiment of the present specification.
  • Components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120 .
  • the processor 1120 is connected with the memory 1110 through the bus 1130, and the database 1150 is used for storing data.
  • Computing device 1100 also includes access device 1140 that enables computing device 1100 to communicate via one or more networks 1160 .
  • networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet.
  • Access device 1140 may include one or more of any type of network interface (eg, network interface card (NIC)), wired or wireless, such as IEEE 802.11 wireless local area network (WLAN) wireless interfaces, World Interoperability for Microwave Access (Wi -MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and the like.
  • NIC network interface card
  • the above-described components of the computing device 1100 and other components not shown in FIG. 11 may also be connected to each other, such as via a bus. It should be understood that the structural block diagram of the computing device shown in FIG. 11 is only for the purpose of example, rather than limiting the scope of this specification. Those skilled in the art can add or replace other components as required.
  • Computing device 1100 may be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (eg, tablets, personal digital assistants, laptops, notebook computers, netbooks, etc.), mobile phones (eg, smartphones ), wearable computing devices (eg, smart watches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs.
  • Computing device 1100 may also be a mobile or stationary server.
  • the processor 1120 is configured to execute the following computer-executable instructions:
  • An embodiment of the present specification further provides a computer-readable storage medium, which stores computer instructions, which, when executed by a processor, are used for:
  • a commit instruction is sent to the target storage module, and the commit instruction carries the target transaction to be submitted and the commit timestamp of the target transaction.
  • An embodiment of the present specification further provides a computer-readable storage medium, which stores computer instructions, which, when executed by a processor, are used for:
  • a commit instruction is received, the target transaction is committed according to the commit instruction, and a binary log is generated according to the commit timestamp of the target transaction, where the commit instruction carries the target transaction to be committed and the commit timestamp of the target transaction.
  • An embodiment of the present specification further provides a computer-readable storage medium, which stores computer instructions, which, when executed by a processor, are used for:
  • the computer instructions include computer program code, which may be in source code form, object code form, an executable file, some intermediate form, or the like.
  • the computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable media Electric carrier signals and telecommunication signals are not included.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种分布式数据库系统及数据处理方法，其中所述系统包括：计算模块（102）和存储模块（104）；所述计算模块（102），被配置为接收事务开启请求；根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块（104）；向所述目标存储模块（104）发送执行指令；所述目标存储模块（104）返回目标事务执行成功的消息的情况下，向所述目标存储模块（104）发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳；所述目标存储模块（104），被配置为接收所述执行指令，根据所述执行指令执行所述目标事务；接收所述提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。

Description

分布式数据库系统及数据处理方法
本申请要求2020年07月24日递交的申请号为202010721315.0、发明名称为“分布式数据库系统及数据处理方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本说明书涉及分布式数据库技术领域，特别涉及一种分布式数据库系统及数据处理方法，一种计算设备，以及一种计算机可读存储介质。
背景技术
随着计算机技术的日益发展，为了满足用户日益增长的需求，分布式数据库应运而生。分布式数据库由多个存储节点组成，在每个存储节点中，每一次DML(Data Manipulation Language，数据操纵语言)写入操作都会独立生成一个二进制日志(binlog)，这些二进制日志在所在的存储节点内保持有序。然而，在分布式数据库中，时间上前后的两次操作可能会发生在两个不同的存储节点上，直接订阅两个存储节点的二进制日志，有可能导致这两次操作的顺序发生错乱，最终的数据出现不一致。
发明内容
有鉴于此，本说明书实施例提供了一种分布式数据库系统。本说明书同时涉及一种数据处理方法，一种计算设备，以及一种计算机可读存储介质，以解决现有技术中存在的技术缺陷。
根据本说明书实施例的第一方面，提供了一种分布式数据库系统，包括：计算模块和存储模块；
所述计算模块，被配置为接收事务开启请求；根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳；
所述目标存储模块，被配置为接收所述执行指令，根据所述执行指令执行所述目标事务；所述目标事务执行成功的情况下，向所述计算模块返回目标事务执行成功的消息；接收所述提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。
可选的，所述系统还包括：多个执行模块和一个排序模块，所述多个执行模块中每一个执行模块均对应有存储模块；
所述执行模块，被配置为读取自身对应的存储模块的二进制日志，并根据所述二进制日志生成内存序列；
所述排序模块，被配置为读取每一个所述执行模块的内存序列，获取所述内存序列 中每一个二进制日志中目标事务的提交时间戳；按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；将排序后的二进制日志写入消息队列中。
可选的，所述计算模块进一步被配置为：
若检测到表格处理事件，则确定所述表格处理事件的执行逻辑；
根据所述执行逻辑，生成所述表格处理事件的时间戳；
根据所述时间戳，生成所述表格处理事件的二进制日志。
可选的，所述计算模块进一步被配置为：
确定所述表格处理事件的操作类型；
根据所述操作类型，确定所述表格处理事件对应的兼容类型；
根据所述兼容类型，确定所述表格处理事件的执行逻辑。
可选的，所述计算模块进一步被配置为：
若所述操作类型属于第一类型，则确定所述表格处理事件对应的兼容类型为第一版本兼容第二版本，所述第一版本为所述表格处理事件执行前的版本，所述第二版本为所述表格处理事件执行后的版本；
若所述操作类型属于第二类型，则确定所述表格处理事件对应的兼容类型为第二版本兼容第一版本。
可选的，所述计算模块进一步被配置为：
若所述表格处理事件对应的兼容类型为第一版本兼容第二版本，则确定所述表格处理事件的执行逻辑为后序执行；
若所述表格处理事件对应的兼容类型为第二版本兼容第一版本，则确定所述表格处理事件的执行逻辑为前序执行。
可选的，所述计算模块进一步被配置为：
若所述表格处理事件的执行逻辑为前序执行，则在生成所有第二版本的数据对应的二进制日志之前，获取所述表格处理事件的时间戳；
若所述表格处理事件的执行逻辑为后序执行，则在生成所有第一版本的数据对应的二进制日志之后，获取所述表格处理事件的时间戳。
可选的，所述排序模块进一步被配置为：
根据所述表格处理事件的时间戳，将所述表格处理事件的二进制日志插入所述消息队列中。
可选的，所述系统还包括时间戳产生器，所述计算模块进一步被配置为：
向所述时间戳产生器申请全局有序的时间戳；
将所述时间戳确定为所述目标事务的提交时间戳。
可选的，所述事务开启请求中携带事务标识及对应的存储模块标识，所述计算模块进一步被配置为：
将所述事务标识指示的事务确定为目标事务；
将所述存储模块标识指示的存储模块确定为对应的目标存储模块。
可选的，所述事务开启请求中携带事务标识，所述计算模块进一步被配置为：
将所述事务标识指示的事务确定为目标事务；
根据预设分配规则，确定所述目标事务对应的目标存储模块。
可选的，所述执行模块进一步被配置为：
若检测到自身对应的存储模块生成新的二进制日志，则返回执行所述读取所述存储模块的二进制日志的操作步骤；
相应的，所述排序模块进一步被配置为：
返回执行所述读取每一个所述执行模块的内存序列的操作步骤；或者，
获取所述执行模块中增加的二进制日志，将所述二进制日志放入所述消息队列的队尾。
可选的，所述计算模块进一步被配置为：
记录所述目标存储模块的存储模块标识。
可选的，所述计算模块进一步被配置为：
向所述目标存储模块发送目标事务开启请求；
所述目标存储模块返回开启消息的情况下，执行所述向所述目标存储模块发送执行指令的操作步骤。
可选的，所述存储模块进一步被配置为：
生成二进制日志，并将所述目标事务的提交时间戳记录在所述二进制日志的注释中。
根据本说明书实施例的第二方面，提供了一种数据处理方法，应用于上述第一方面提供的分布式数据库系统中的计算模块，所述方法包括：
接收事务开启请求；
根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；
向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；
所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
可选的，所述方法还包括：
若检测到表格处理事件，则确定所述表格处理事件的执行逻辑；
根据所述执行逻辑，生成所述表格处理事件的时间戳；
根据所述时间戳，生成所述表格处理事件的二进制日志。
可选的，所述确定所述表格处理事件的执行逻辑，包括：
确定所述表格处理事件的操作类型；
根据所述操作类型，确定所述表格处理事件对应的兼容类型；
根据所述兼容类型，确定所述表格处理事件的执行逻辑。
可选的，所述根据所述操作类型，确定所述表格处理事件对应的兼容类型，包括：
若所述操作类型属于第一类型，则确定所述表格处理事件对应的兼容类型为第一版本兼容第二版本，所述第一版本为所述表格处理事件执行前的版本，所述第二版本为所述表格处理事件执行后的版本；
若所述操作类型属于第二类型，则确定所述表格处理事件对应的兼容类型为第二版本兼容第一版本。
可选的，所述根据所述兼容类型，确定所述表格处理事件的执行逻辑，包括：
若所述表格处理事件对应的兼容类型为第一版本兼容第二版本，则确定所述表格处理事件的执行逻辑为后序执行；
若所述表格处理事件对应的兼容类型为第二版本兼容第一版本，则确定所述表格处理事件的执行逻辑为前序执行。
可选的，所述根据所述执行逻辑，生成所述表格处理事件的时间戳，包括：
若所述表格处理事件的执行逻辑为前序执行，则在生成所有第二版本的数据对应的二进制日志之前，获取所述表格处理事件的时间戳；
若所述表格处理事件的执行逻辑为后序执行，则在生成所有第一版本的数据对应的二进制日志之后，获取所述表格处理事件的时间戳。
可选的，所述向所述目标存储模块发送提交指令之前，还包括：
向时间戳产生器申请全局有序的时间戳；
将所述时间戳确定为所述目标事务的提交时间戳。
可选的，所述事务开启请求中携带事务标识及对应的存储模块标识，所述根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块，包括：
将所述事务标识指示的事务确定为目标事务；
将所述存储模块标识指示的存储模块确定为对应的目标存储模块。
可选的，所述事务开启请求中携带事务标识，所述根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块包括：
将所述事务标识指示的事务确定为目标事务；
根据预设分配规则，确定所述目标事务对应的目标存储模块。
可选的，所述向所述目标存储模块发送执行指令之后，还包括：
记录所述目标存储模块的存储模块标识。
可选的，所述根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块之后，且所述向所述目标存储模块发送执行指令之前，还包括：
向所述目标存储模块发送目标事务开启请求；
所述目标存储模块返回开启消息的情况下，执行所述向所述目标存储模块发送执行 指令的操作步骤。
根据本说明书实施例的第三方面，提供了一种数据处理方法，应用于上述第一方面提供的分布式数据库系统中的存储模块，所述方法包括：
接收到执行指令后，根据所述执行指令执行目标事务，所述执行指令中携带有待执行目标事务；
所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息；
接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
可选的，所述根据所述目标事务的提交时间戳生成二进制日志，包括：
生成二进制日志，并将所述目标事务的提交时间戳记录在所述二进制日志的注释中。
根据本说明书实施例的第四方面，提供了一种数据处理方法，应用于上述第一方面提供的分布式数据库系统中的排序模块，所述方法包括：
读取每一个执行模块的内存序列；
获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；
按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；
将排序后的二进制日志写入消息队列中。
可选的，所述将排序后的二进制日志写入消息队列中之后，还包括：
若检测到执行模块的内存序列发生变化，则返回执行所述读取每一个所述执行模块的内存序列的操作步骤。
可选的，所述将排序后的二进制日志写入消息队列中之后，还包括：
若检测到执行模块的内存序列发生变化，则获取所述执行模块中增加的二进制日志，将所述二进制日志放入所述消息队列的队尾。
可选的，所述将排序后的二进制日志写入消息队列中之后，还包括：
根据表格处理事件的时间戳，将所述表格处理事件的二进制日志插入所述消息队列中。
根据本说明书实施例的第五方面，提供了一种计算设备，包括：
存储器和处理器；
所述存储器用于存储计算机可执行指令，所述处理器用于执行所述计算机可执行指令：
接收事务开启请求；
根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；
向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；
所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发 送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
根据本说明书实施例的第六方面，提供了一种计算设备，包括：
存储器和处理器；
所述存储器用于存储计算机可执行指令，所述处理器用于执行所述计算机可执行指令：
接收到执行指令后，根据所述执行指令执行目标事务，所述执行指令中携带有待执行目标事务；
所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息；
接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
根据本说明书实施例的第七方面，提供了一种计算设备，包括：
存储器和处理器；
所述存储器用于存储计算机可执行指令，所述处理器用于执行所述计算机可执行指令：
读取每一个执行模块的内存序列；
获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；
按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；
将排序后的二进制日志写入消息队列中。
根据本说明书实施例的第八方面，提供了一种计算机可读存储介质，其存储有计算机可执行指令，该指令被处理器执行时实现上述第二方面任意所述数据处理方法的步骤。
根据本说明书实施例的第九方面，提供了一种计算机可读存储介质，其存储有计算机可执行指令，该指令被处理器执行时实现上述第三方面任意所述数据处理方法的步骤。
根据本说明书实施例的十方面，提供了一种计算机可读存储介质，其存储有计算机可执行指令，该指令被处理器执行时实现上述第四方面任意所述数据处理方法的步骤。
本说明书提供的分布式数据库系统包括计算模块和存储模块，所述计算模块被配置为接收事务开启请求；根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；向所述目标存储模块发送执行指令；所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳；所述目标存储模块被配置为接收所述执行指令，根据所述执行指令执行所述目标事务；所述目标事务执行成功的情况下，向所述计算模块返回目标事务执行成功的消息；接收所述提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。如此，每个存储模块都是根据计算模块发送的目标事务的提交时间戳生成的二进制日志，每个二进制日志中 都携带一个全局有序的时间戳，使得分布式数据库系统中时间上前后的两次操作可以按序排列，避免了操作的顺序发生错乱。
附图说明
图1是本说明书一实施例提供的一种分布式数据库系统的结构框图；
图2是本说明书一实施例提供的第一种数据处理方法的流程图；
图3是本说明书一实施例提供一种前序执行和后序执行示意图；
图4是本说明书一实施例提供的第二种数据处理方法的流程图；
图5是本说明书一实施例提供的第三种数据处理方法的流程图；
图6是本说明书一实施例提供的第四种数据处理方法的流程图；
图7是本说明书一实施例提供的一种二进制日志生成过程示意图；
图8是本说明书一实施例提供的一种排序过程示意图；
图9是本说明书一实施例提供的第一种计算设备的结构框图；
图10是本说明书一实施例提供的第二种计算设备的结构框图；
图11是本说明书一实施例提供的第三种计算设备的结构框图。
具体实施方式
在下面的描述中阐述了很多具体细节以便于充分理解本说明书。但是本说明书能够以很多不同于在此描述的其它方式来实施，本领域技术人员可以在不违背本说明书内涵的情况下做类似推广，因此本说明书不受下面公开的具体实施的限制。
在本说明书一个或多个实施例中使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本说明书一个或多个实施例。在本说明书一个或多个实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本说明书一个或多个实施例中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。
应当理解，尽管在本说明书一个或多个实施例中可能采用术语第一、第二等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不脱离本说明书一个或多个实施例范围的情况下，第一也可以被称为第二，类似地，第二也可以被称为第一。取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。
首先，对本说明书一个或多个实施例涉及的名词术语进行解释。
SQL(Structured Query Language，结构化查询语言)：是一种特殊目的的编程语言，一种数据库查询和程序设计语言，用于存取数据以及查询、更新和管理关系数据库系统。SQL是高级的非过程化编程语言，允许用户在高层数据结构上工作，它不要求用户指定对数据的存放方法，也不需要用户了解具体的数据存放方式，所以具有完全不同底层结构的不同数据库系统,可以使用相同的SQL作为数据输入与管理的接口，SQL可以嵌套， 这使它具有极大的灵活性和强大的功能。SQL语言包括四种主要程序设计语言类别的语句：数据定义语言(DDL)，数据操作语言(DML)，数据控制语言(DCL)和事务控制语言(TCL)。其中，数据操作语言(DML)，例如增、删、改等语句；数据定义语言(Data Definition Language，DDL)，例如增加列、变更列类型、长度、增加约束等语句。
事务：是恢复和并发控制的基本单位，是由多条SQL语句组成的一个执行单元，具有原子性、一致性、隔离性、持久性4个特性，原子性(atomicity)是指一个事务是一个不可分割的工作单位，事务中包括的操作要么都做，要么都不做；一致性(consistency)是指事务必须是使数据库从一个一致性状态变到另一个一致性状态，一致性与原子性是密切相关的；隔离性(isolation)是指一个事务的执行不能被其他事务干扰，即一个事务内部的操作及使用的数据对并发的其他事务是隔离的，并发执行的各个事务之间不能互相干扰；持久性(durability)也称永久性(permanence)，指一个事务一旦提交，它对数据库中数据的改变就应该是永久性的，接下来的其他操作或故障不应该对其有任何影响。
MySQL：是一个关系型数据库管理系统，随着MySQL的不断成熟，逐渐用于更多大规模网站和应用。
PolarDB：是新一代关系型云原生数据库，既拥有分布式设计的低成本优势，又具有集中式的易用性，实现了计算模块及存储模块的分离，提供即时生效的可扩展能力和运维能力，软硬一体化设计，满足大规模应用场景需求。
PolarDB-X：PolarDB的分布式版，一个基于MySQL sharding(数据库分片)的分布式数据库，主要由存储模块(MySQL server)与计算模块组成。
TSO：一个中心化的时间戳产生器，提供全局有序的时间戳服务。
binlog：二进制格式的日志文件，用来记录Mysql内部对数据库的改动(只记录对数据的修改操作)，主要用于数据库的主从复制以及增量恢复。MySQL官方支持三种binlog数据格式，对应参数为binlog_format，可选值分别为ROW、STATEMENT、MIXED，对应的默认值为MIXED，需要调整为ROW模式，该模式的特点为：每个增删改的SQL，都会在binlog里记录SQL所影响的行在变更前和变更后的值。在MySQL中，每一次DML(增删改等写入，包括DDL)操作都会生成binlog(binary log)日志，此日志在MySQL中是按照操作时间有序的，下游应用可以基于此日志做一些订阅操作(例如数据同步)。
物理binlog：PolarDB-X的每个存储模块的binlog。
逻辑binlog：PolarDB-X层提供的binlog，包含所有存储模块的binlog，对外展现形式是一个kafka消息队列。
Commit指令：提交指令，用于把事务所做的修改保存到数据库。
commit_timestamp：用来标明事务的提交时间，缩写commit_ts。
精卫worker：一个执行工作的模块，与存储模块一一对应，负责读取对应的存储模块的binlog文件，并负责进行解析，同时将解析后的结果生成一个内存序列。
精卫merger：相当于一个归并排序的模块，将多个有序的队列，归并为一个全局有序的队列。
接下来，对本说明书的基本构思进行简述：
目前，在分布式数据库中，每个存储节点执行事务(DML相关的操作)后都会独立生成一个二进制日志，每个存储节点生成的二进制日志在该存储节点内保持有序，然而不同的存储节点生成的二进制日志之间时间先后顺序并不确定，因而在分布式数据库中，时间上前后的两次操作可能会发生在两个不同的存储节点上，下游应用直接订阅两个存储节点的二进制日志，有可能导致这两次操作的顺序发生错乱，从而导致下游应用最终的数据出现不一致。
例如，分布式数据库系统中包括MySQL1、MySQL2和MySQL3，MySQL1内生成了2个二进制日志：日志A1和日志A2，MySQL2内生成了3个二进制日志：日志B1、日志B2和日志B3，MySQL3内生成了2个二进制日志：日志C1和日志C2。现有技术中，只能确定日志A1早于日志A2，日志B1早于日志B2早于日志B3，日志C1早于日志C2，但日志A1、日志A2、日志B1、日志B2、日志B3、日志C1和日志C2之间的先后顺序并不确定，有可能会出现错乱，导致下游应用订阅的数据出现不一样。
本说明书中提供了一种分布式数据库系统，包括计算模块和存储模块(如MySQL)，计算模块控制存储模块执行目标事务，存储模块执行成功后向计算模块返回成功消息，此时计算模块会向存储模块发送目标事务的提交时间戳，以使存储模块根据该时间戳生成二进制日志，这种情况下的二进制日志中记录有全局有序的目标事务的提交时间戳，使得分布式数据库系统中所有的二进制日志都是有时间顺序的，从而避免了事务的操作顺序发生错乱，保证了下游应用订阅数据的一致性。
在本说明书中，提供了一种分布式数据库系统，本说明书同时涉及一种数据处理方法，一种计算设备，以及一种计算机可读存储介质，在下面的实施例中逐一进行详细说明。
图1示出了根据本说明书一实施例提供的一种分布式数据库系统的结构框图，具体包括计算模块102和存储模块104。
所述计算模块102，被配置为接收事务开启请求；根据所述事务开启请求，确定目标事务及所述目标事务对应的存储模块104；向所述存储模块104发送执行指令，所述执行指令中携带有待执行目标事务；所述存储模块104返回目标事务执行成功的消息的情况下，向所述存储模块104发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳；
所述存储模块104，被配置为接收所述执行指令，根据所述执行指令执行所述目标 事务；所述目标事务执行成功的情况下，向所述计算模块102返回目标事务执行成功的消息；接收所述提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。
一个可选的实施例中，所述系统还包括：多个执行模块和一个排序模块，所述多个执行模块中每一个执行模块均对应有存储模块104；
所述执行模块，被配置为读取自身对应的存储模块104的二进制日志，并根据所述二进制日志生成内存序列；
所述排序模块，被配置为读取每一个所述执行模块的内存序列，获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；将排序后的二进制日志写入消息队列中。
一个可选的实施例中，所述计算模块进一步被配置为：
若检测到表格处理事件，则确定所述表格处理事件的执行逻辑；
根据所述执行逻辑，生成所述表格处理事件的时间戳；
根据所述时间戳，生成所述表格处理事件的二进制日志。
一个可选的实施例中，所述计算模块进一步被配置为：
确定所述表格处理事件的操作类型；
根据所述操作类型，确定所述表格处理事件对应的兼容类型；
根据所述兼容类型，确定所述表格处理事件的执行逻辑。
一个可选的实施例中，所述计算模块进一步被配置为：
若所述操作类型属于第一类型，则确定所述表格处理事件对应的兼容类型为第一版本兼容第二版本，所述第一版本为所述表格处理事件执行前的版本，所述第二版本为所述表格处理事件执行后的版本；
若所述操作类型属于第二类型，则确定所述表格处理事件对应的兼容类型为第二版本兼容第一版本。
一个可选的实施例中，所述计算模块进一步被配置为：
若所述表格处理事件对应的兼容类型为第一版本兼容第二版本，则确定所述表格处理事件的执行逻辑为后序执行；
若所述表格处理事件对应的兼容类型为第二版本兼容第一版本，则确定所述表格处理事件的执行逻辑为前序执行。
一个可选的实施例中，所述计算模块进一步被配置为：
若所述表格处理事件的执行逻辑为前序执行，则在生成所有第二版本的数据对应的二进制日志之前，获取所述表格处理事件的时间戳；
若所述表格处理事件的执行逻辑为后序执行，则在生成所有第一版本的数据对应的二进制日志之后，获取所述表格处理事件的时间戳。
一个可选的实施例中，所述排序模块进一步被配置为：
根据所述表格处理事件的时间戳，将所述表格处理事件的二进制日志插入所述消息队列中。
一个可选的实施例中，所述系统还包括时间戳产生器，所述计算模块102进一步被配置为：
向所述时间戳产生器申请全局有序的时间戳；
将所述时间戳确定为所述目标事务的提交时间戳。
一个可选的实施例中，所述事务开启请求中携带事务标识及对应的存储模块标识，所述计算模块102进一步被配置为：
将所述事务标识指示的事务确定为目标事务；
将所述存储模块标识指示的存储模块确定为对应的存储模块104。
一个可选的实施例中，所述事务开启请求中携带事务标识，所述计算模块102进一步被配置为：
将所述事务标识指示的事务确定为目标事务；
根据预设分配规则，确定所述目标事务对应的存储模块104。
一个可选的实施例中，所述执行模块进一步被配置为：
若检测到自身对应的存储模块104生成新的二进制日志，则返回执行所述读取所述存储模块104的二进制日志的操作步骤；
相应的，所述排序模块进一步被配置为：
返回执行所述读取每一个所述执行模块的内存序列的操作步骤；或者，
获取所述执行模块中增加的二进制日志，将所述二进制日志放入所述消息队列的队尾。
一个可选的实施例中，所述计算模块102进一步被配置为：
记录所述存储模块104的存储模块标识。
一个可选的实施例中，所述计算模块102进一步被配置为：
向所述存储模块104发送目标事务开启请求；
所述存储模块104返回开启消息的情况下，执行所述向所述存储模块104发送执行指令的操作步骤。
一个可选的实施例中，所述存储模块104进一步被配置为：
生成二进制日志，并将所述目标事务的提交时间戳记录在所述二进制日志的注释中。
本说明书提供的分布式数据库系统包括计算模块和存储模块，所述计算模块被配置为接收事务开启请求；根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；向所述目标存储模块发送执行指令；所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交 目标事务和所述目标事务的提交时间戳；所述目标存储模块被配置为接收所述执行指令，根据所述执行指令执行所述目标事务；所述目标事务执行成功的情况下，向所述计算模块返回目标事务执行成功的消息；接收所述提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。如此，每个存储模块都是根据计算模块发送的目标事务的提交时间戳生成的二进制日志，每个二进制日志中都携带一个全局有效的时间戳，使得分布式数据库系统中时间上前后的两次操作生成的二进制日志可以按序排列，从而避免了事务的操作顺序发生错乱，保证了下游应用订阅数据的一致性。
图2示出了根据本说明书一实施例提供的一种数据处理方法的流程图，应用于图1所示的分布式数据库系统中的计算模块，具体包括如下步骤：
步骤202：接收事务开启请求。
实际应用中，当用户要在分布式数据库系统中执行某个事务时，可以通过应用程序向分布式数据库系统中的计算模块发送一个事务开启请求，计算模块接收该事务开启请求，从而开启后续执行事务的整个过程。此处的分布式数据库系统可以是指PolarDB-X，也可以是其他分布式数据库，本说明书对此不进行限制。
步骤204：根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块。
具体的，在接收事务开启请求的基础上，进一步的，将根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块。
其中，所述目标事务是指事务开启请求指示开启的事务，所述目标存储模块是指执行所述目标事务所需涉及到的存储模块。此处的存储模块可以是分布式数据库系统中的单片数据库，例如单片MySQL。
实际应用中，计算模块接收到事务开启请求后，首先需要确定该请求指示执行的具体事务，由于事务最终是由存储模块执行的，因而计算模块还需要进一步确定出执行该事务的存储模块。由于事务是由多条SQL语句组成的执行单元，因而一个事务中可能包括有多个SQL操作，该多个操作可能会由不同的存储模块执行，因而一个事务可能会涉及到多个存储模块，也即确定出的目标存储模块可能为多个。
在本实施例的一个或多个实施方式中，用户要在分布式数据库系统中执行某个事务时，可以直接指定执行该事务的存储模块，此时应用程序会将要执行的事务对应的事务标识，以及执行所述事务的存储模块的标识携带在事务开启请求中发送给计算模块，计算模块可以直接根据接收到的事务开启请求中携带的事务标识及对应的存储模块标识，确定目标事务及所述目标事务对应的目标存储模块，具体实现过程如下：
将所述事务标识指示的事务确定为目标事务；
将所述存储模块标识指示的存储模块确定为对应的目标存储模块。
具体的，事务标识用于标识要执行的事务，存储模块标识用于标识执行事务的存储模块。
例如，用户要在分布式数据库系统中执行事务A，且用户指定了由存储模块1和存储模块2来执行所述事务A，此时用户可以通过某应用程序向分布式数据库系统中的计算模块发送一个事务开启指令，该指令中会携带事务A、存储模块1和存储模块2。计算模块接收到该事务开启指令后，就可以确定出要执行的目标事务为事务A，执行事务A的存储模块为存储模块1和存储模块2。
在事务开启请求中直接携带事务标识及对应的存储模块标识，计算模块直接根据标识即可确定出目标事务及目标事务对应的目标存储模块，确定过程简单易行，不需要计算模块进行复杂的运算，一定程度上节省了计算资源。
在本实施例的一个或多个实施方式中，用户要在分布式数据库系统中执行某个事务时，也可以不指定执行该事务的存储模块，由计算模块根据存储模块的性能属性、负载等参数指定对应的存储模块执行所述目标事务，此时应用程序只将要执行的事务对应的事务标识携带在事务开启请求中发送给计算模块，计算模块确定出要执行的目标事务后，可以指定执行所述目标事务的目标存储模块，具体实现过程如下：
将所述事务标识指示的事务确定为目标事务；
根据预设分配规则，确定所述目标事务对应的目标存储模块。
具体的，预设分配规则是指为待执行的目标事务分配存储模块的规则，该规则由技术人员预先进行设置，可以是负载均衡规则，还可以是存储模块性能规则，本说明书中不对分配规则进行具体限定，只要能为目标事务分配合理的存储模块的规则均可。
例如，预设分配规则为负载均衡，且假设分布式数据库系统包括存储模块1和存储模块2，存储模块1正在执行或待执行的事务有事务A、事务B，存储模块2正在执行或待执行的事务有事务C。此时，某用户通过应用程序向分布式数据库系统中的计算模块发送了一个事务开启指令，该指令中携带事务D，计算模块接收到该事务开启指令后，确定出待执行的目标事务为事务D，然后根据负载均衡规则，确定执行事务D的存储模块为存储模块2。
只在事务开启请求中携带事务标识，计算模块确定出目标事务后，根据预先设置的分配规则，为所述目标事务分配对应的目标存储模块，可以根据性能属性、负载等参数灵活指定执行事务的存储模块，可以适应分布式数据库系统中各个存储模块自身的实际情况，保证了事务可以高效准确执行，适应性更高。
本说明书中既可以直接在事务开启请求中指定目标事务和执行目标事务的目标存储模块，还可以不指定，由计算模块为目标事务分配执行的存储模块，灵活性更高，可以适应不同事务的执行特点，从而可以更高效、更准确的执行目标事务。
步骤206：向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标 事务。
具体的，在根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块的基础上，进一步的，将向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务。
在本实施例的一个或多个实施方式中，计算模块确定出目标事务及执行目标事务的目标存储模块后，目标存储模块并不知道自身被选定要开始执行目标事务了，因此在根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块之后，且所述向所述目标存储模块发送执行指令之前，计算模块还可以通知目标存储模块开启目标事务，具体实现过程如下：
向所述目标存储模块发送目标事务开启请求；
所述目标存储模块返回开启消息的情况下，执行所述向所述目标存储模块发送执行指令的操作步骤。
在本实施例的一个或多个实施方式中，后续目标存储模块执行目标事务后，计算模块还需要向目标存储模块发送提交指令，以使目标存储模块对目标事务进行提交，因而计算模块在向目标存储模块发送执行指令后，还需要记录所述目标存储模块的存储模块标识，从而方便后续向目标存储模块发送提交指令。
步骤208：所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
具体的，在向所述目标存储模块发送执行指令的基础上，进一步的，将在所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
实际应用中，目标存储模块接收到执行指令后，会执行目标事务，在目标事务执行成功的情况下，会向计算模块返回执行成功的消息，以通知计算模块，其已经成功执行了目标事务，可以提交了。计算模块接收到该执行成功的消息后，就可以控制存储模块对目标事务进行提交。
在本实施例的一个或多个实施方式中，用户是通过应用程序使分布式数据库系统执行对应的目标事务，目标存储模块在目标事务执行成功的情况下，向计算模块返回执行成功的消息，进一步的此时计算模块还可以向应用程序返回目标事务执行完毕的消息，应用程序会向计算模块返回事务提交消息，以使计算模块控制目标存储模块对事务进行提交。
在本实施例的一个或多个实施方式中，为了使各个存储模块提交事务时生成的二进制日志之间是有时间先后顺序的，计算模块在向目标存储模块发送提交指令时，会在该指令中携带目标事务的提交时间戳，因而在向所述目标存储模块发送提交指令之前，计 算模块还可以获取全局有序的目标事务的提交时间戳，具体实现方式如下：
向时间戳产生器申请全局有序的时间戳；
将所述时间戳确定为所述目标事务的提交时间戳。
具体的，时间戳产生器可以是分布式数据库系统中的TSO，向TSO获取的目标事务的提交时间戳为commit_timestamp。
实际应用过程中，计算模块除了会接收到与增删改数据相关的DML操作，还有可能会接收到DDL事件，如变更列类型、长度，增加约束等事件。分布式数据库系统在DDL变更过程中，同一个时刻，数据库的Schema元数据会存在两个版本，例如加列操作，新列对部分分片已经可见，对部分分片还未可见，此时会同时产生两个版本的数据。这两个版本的数据在通过逻辑binlog队列往下游系统(例如单机MySQL)进行传递的过程中，需要将DDL事件放在消息队列适当的位置，确保下游系统能正确的接受这两个版本的数据，不会出现同一条数据在下游系统中出现不一致，具体实现过程如下：
若检测到表格处理事件，则确定所述表格处理事件的执行逻辑；
根据所述执行逻辑，生成所述表格处理事件的时间戳；
根据所述时间戳，生成所述表格处理事件的二进制日志。
其中，表格处理事件是指DDL事件，例如，加列、列的长度变长、增加列的默认值、删除列、列的长度变短等事件。
在本实施例的一个或多个实施方式中，确定所述表格处理事件的执行逻辑，具体实现过程可以为：
确定所述表格处理事件的操作类型；
根据所述操作类型，确定所述表格处理事件对应的兼容类型；
根据所述兼容类型，确定所述表格处理事件的执行逻辑。
其中，操作类型是设定好的，不同的操作类型对应不同的兼容类型。
在本实施例的一个或多个实施方式中，根据所述操作类型，确定所述表格处理事件对应的兼容类型，具体实现过程可以为：
若所述操作类型属于第一类型，则确定所述表格处理事件对应的兼容类型为第一版本兼容第二版本，所述第一版本为所述表格处理事件执行前的版本，所述第二版本为所述表格处理事件执行后的版本；
若所述操作类型属于第二类型，则确定所述表格处理事件对应的兼容类型为第二版本兼容第一版本。
具体的，一个时刻的Schema结构(包含表结构等)称为一个版本，DDL事件执行前的版本称为第一版本，DDL事件执行后的版本称为第二版本。需要说明的是，Va版本兼容Vb版本是指：使用Vb版本的数据可以被写入使用Va版本的目标端；从使用Va版本的目标端将该数据读出后，其包含足够的信息，在经过一定的转换后，可以转换为 Vb版本的数据，并且和原数据完全相等。
另外，第一类型即为第一版本可以兼容第二版本的操作类型，例如，删除列，列的长度变短，删表，减小列的精度等操作均为第一类型。第二类型即为第二版本可以兼容第一版本的操作类型，例如，加列，列的长度变长，增加列的默认值，创建、删除索引，创建、删除约束(创建唯一键约束)，建表，增加列的精度等操作均为第二类型。
在本实施例的一个或多个实施方式中，根据所述兼容类型，确定所述表格处理事件的执行逻辑，具体实现过程可以为：
若所述表格处理事件对应的兼容类型为第一版本兼容第二版本，则确定所述表格处理事件的执行逻辑为后序执行；
若所述表格处理事件对应的兼容类型为第二版本兼容第一版本，则确定所述表格处理事件的执行逻辑为前序执行。
具体的，若所述表格处理事件的执行逻辑为前序执行，则在生成所有第二版本的数据对应的二进制日志之前，获取所述表格处理事件的时间戳；若所述表格处理事件的执行逻辑为后序执行，则在生成所有第一版本的数据对应的二进制日志之后，获取所述表格处理事件的时间戳。
实际应用中，参见图3所示，前序执行是指DDL事件的二进制日志，应该在所有第二版本(V2)数据的二进制日志的最前面，因而在生成所有第二版本(V2)的数据对应的二进制日志之前，获取一个时间戳，将该时间戳作为DDL事件的时间戳，也即，DDL事件应该位于所有的第二版本(V2)的二进制日志之前。后序执行是指DDL事件的二进制日志，应该在所有第一版本(V1)数据的二进制日志的最后面，因而在生成所有第一版本(V1)的数据对应的二进制日志之后，获取一个时间戳，将该时间戳作为DDL事件的时间戳，也即DDL事件应该位于所有的第一版本(V1)的二进制日志之后。
另外，某些表格处理事件(DDL事件)的操作类型，第一版本可以兼容第二版本，第二版本也可以兼容第一版本，此时所述表格处理事件的执行逻辑既可以为后序执行，也可以为前序执行。
在检测到表格处理事件的情况下，根据表格处理事件的执行逻辑，生成所述表格处理事件的时间戳，然后可以根据所述时间戳，生成所述表格处理事件的二进制日志，从而使得该表格处理事件的二进制日志可以处于适当的位置，保证下游应用订阅数据的一致性。
本说明书中提供的数据处理方法，计算模块接收事务开启请求，并根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；然后向目标存储模块发送执行指令；在目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。这种情况下，计算模块可以向时间戳产生器申请一个全局有序的时间戳，将该时间戳作 为目标事务的提交时间戳，并发送给目标存储模块，使得后续目标存储模块可以生成带全局有序时间戳的二进制日志，从而使得分布式数据库系统中时间上前后的两次操作生成的二进制日志可以按序排列，避免了事务的操作顺序发生错乱，保证了下游应用订阅数据的一致性。
图4示出了根据本说明书一实施例提供的一种数据处理方法的流程图，应用于图1所示的分布式数据库系统中的存储模块，具体包括如下步骤：
步骤402：接收执行指令，根据所述执行指令执行目标事务，所述执行指令中携带有待执行目标事务。
实际应用中，计算模块确定出目标事务及执行目标事务的目标存储模块后，目标存储模块自身并不知道被选定要开始执行目标事务了，因此目标存储模块在接收到执行指令之前，还会接收到计算模块发送的目标事务开启请求，目标存储模块接收到该目标事务开启请求的情况下，会开启目标事务，并通知计算模块自己准备好执行目标事务了，因而在接收执行指令，根据所述执行指令执行目标事务之前，目标存储模块还应该在接收到目标事务开启请求的情况下，向计算模块返回开启消息。
步骤404：所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息。
具体的，在接收执行指令，根据所述执行指令执行目标事务的基础上，进一步的，将在所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息。
步骤406：接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
具体的，在所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息的基础上，进一步的，将接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。
实际应用中，二进制日志即binlog，每个存储模块生成的二进制日志被称作物理binlog。
在本实施例的一个或多个实施方式中，根据所述目标事务的提交时间戳生成二进制日志，具体实现过程可以为：生成二进制日志，并将所述目标事务的提交时间戳记录在所述二进制日志的注释中。
本说明书中提供的数据处理方法，目标存储模块接收执行指令，根据所述执行指令执行目标事务；并在所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息；之后接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。这种情况下，每个存储模块都是根据计算模块发送的目标事务的提交时间戳生成的二进制日志，每个二进制日志中都携带一个全局 有序的时间戳，使得分布式数据库系统中时间上前后的两次操作生成的二进制日志可以按序排列，从而避免了事务的操作顺序发生错乱，保证了下游应用订阅数据的一致性。
图5示出了根据本说明书一实施例提供的一种数据处理方法的流程图，应用于分布式数据库系统中的排序模块，具体包括如下步骤：
步骤502：读取每一个执行模块的内存序列。
实际应用中，每一个执行模块都对应有一个存储模块，在排序模块读取每一个执行模块的内存序列之前，执行模块还可以读取自身对应的存储模块的二进制日志，并根据所述二进制日志生成内存序列。
具体的，此处的执行模块可以指PolarDB-X生成的精卫worker，一个执行工作的模块。排序模块可以指PolarDB-X生成的精卫merger，用于管控生成的精卫worker，一个归并排序的模块，可以将多个有序的队列，归并为一个全局有序的队列。
本说明书中每个执行模块都生成了一个内存序列，该内存序列包括其对应的存储模块生成的全部二进制日志，后续对二进制日志进行排序时，只需要获取内存序列中的二进制日志即可，不需要再去分别一个个获取存储模块生成的每个二进制日志，操作简单方便，方便后续进行排序。
步骤504：获取所述内存序列中每一个二进制日志中目标事务的提交时间戳。
具体的，在读取每一个执行模块的内存序列的基础上，进一步的，将获取所述内存序列中每一个二进制日志中目标事务的提交时间戳。
步骤506：按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序。
具体的，在获取所述内存序列中每一个二进制日志中目标事务的提交时间戳的基础上，进一步的，将按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序。
步骤508：将排序后的二进制日志写入消息队列中。
具体的，在按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序的基础上，进一步的，将排序后的二进制日志写入消息队列中。
实际应用中，消息队列可以指kafka消息队列，该消息队列是一个开源的消息队列，将排序后的二进制日志写入消息队列中，即生成了分布式数据库系统的逻辑binlog，下游应用需要订阅分布式数据库中的数据时，可以直接读取该开源的消息队列中的逻辑binlog，即可获得分布式数据库系统中存储模块的各个二进制日志，且该二进制日志在消息队列中都是有序排列的。
在本实施例的一个或多个实施方式中，对存储模块当前各个二进制日志进行排序后，存储模块可能会执行新的事务，生成新的二进制日志，因而执行模块在检测到其对应的存储模块生成新的二进制日志的情况下，可以返回执行读取对应的存储模块的二进制日 志，并根据所述二进制日志生成内存序列的操作步骤，此时排序模块可以对所有二进制日志全部重新排序，具体实现过程如下：
若检测到执行模块的内存序列发生变化，则返回执行所述读取每一个所述执行模块的内存序列的操作步骤。
实际应用中，若检测到执行模块的内存序列发生变化，说明执行模块对应的存储模块执行了新的事务，生成了新的二进制日志，此时排序模块会返回执行步骤502-508，对所有二进制日志全部重新排序。
在本实施例的一个或多个实施方式中，对存储模块当前各个二进制日志进行排序后，存储模块可能会执行新的事务，生成新的二进制日志，因而执行模块在检测到其对应的存储模块生成新的二进制日志的情况下，可以返回执行读取对应的存储模块的二进制日志，并根据所述二进制日志生成内存序列的操作步骤，由于新生成的二进制日志在时间上一定是晚于已经排序好的二进制日志的，因而此时排序模块可以直接将执行模块中增加的二进制日志放入消息队列的队尾。
本说明书中在存储模块执行了新的事务，生成新的二进制日志的情况下，既可以对所有的二进制日志全部重新排序，也可以直接将新生成的二进制日志放入消息队列的队尾，实现了实时对存储模块的所有二进制日志进行排序，且排序方式灵活，可以适用不同情况，使得排序结果更高效准确。
在本实施例的一个或多个实施方式中，若计算模块检测到了表格处理事件，则可以根据表格处理事件的时间戳，生成表格处理事件的二进制日志，此情况下，排序模块可以根据表格处理事件的时间戳，将所述表格处理事件的二进制日志插入所述消息队列中。
在检测到表格处理事件的情况下，可以根据表格处理事件的执行逻辑，生成所述表格处理事件的时间戳，然后可以根据所述时间戳，生成所述表格处理事件的二进制日志，且可以根据该表格处理事件的时间戳，将该表格处理事件插入消息队列中，从而使得该表格处理事件可以放在消息队列中适当的位置，确保下游系统能正确的接收表格处理事件执行前和执行后两个版本的数据，避免了同一条数据在下游系统中出现不一致的情况。
本说明书提供的数据处理方法，排序模块可以读取每一个执行模块的内存序列，并获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；然后按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序，将排序后的二进制日志写入消息队列中。这种情况下，消息队列中包括分布式数据库系统中所有存储模块生成的二进制日志，且都是按时间先后顺序有序排列，下游应用需要订阅分布式数据库中的数据时，可以直接读取该消息队列，即可避免事务的操作顺序发生错乱，保证订阅数据的一致性。
以本说明书提供的数据处理方法在分布式数据库系统中整体的应用为例，对所述数据处理方法进行进一步说明。其中，图6示出了本说明书一实施例提供的一种数据处理 方法的处理流程图，具体包括以下步骤：
步骤602：计算模块接收应用程序发送的事务开启请求，并根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块。
实际应用中，在一个事务包括多个操作的情况下，应用程序向计算模块发送事务开启请求时，可以分多次发送，并将多个操作的标识分别携带在多个请求中，因而计算模块可以分多次接收到事务开启请求，每个请求中携带事务包括的一个操作。例如，目标事务为事务A，包括操作1和操作2，此时计算模块会先接收到一个携带操作1标识的请求，然后控制对应的存储模块执行操作1，在操作1执行完毕后，计算模块会再接收到一个携带操作2标识的请求，然后再控制对应的存储模块执行操作2，执行完毕后，再对整个事务A进行提交。
步骤604：计算模块向所述目标存储模块发送目标事务开启请求。
步骤606：目标存储模块接收所述目标事务开启请求，开启目标事务，并向计算模块返回开启消息。
步骤608：计算模块向所述目标存储模块发送执行指令，并记录所述目标存储模块的存储模块标识，所述执行指令中携带有待执行目标事务。
步骤610：目标存储模块接收所述执行指令，根据所述执行指令执行所述目标事务，并在所述目标事务执行成功的情况下，向所述计算模块返回目标事务执行成功的消息。
步骤612：计算模块向应用程序返回所述目标事务执行成功的消息，并接收应用程序发送的提交指令。
步骤614：计算模块向时间戳产生器申请全局有序的时间戳，将所述时间戳确定为所述目标事务的提交时间戳，并向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
步骤616：目标存储模块接收所述提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。
步骤618：执行模块读取自身对应的存储模块的二进制日志，并根据所述二进制日志生成内存序列。
步骤620：排序模块读取每一个执行模块的内存序列，并获取所述内存序列中每一个二进制日志中目标事务的提交时间戳。
步骤622：排序模块按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序，并将排序后的二进制日志写入消息队列中。
需要说明的是，虽然一个事务可能会包括多个SQL操作，但是一个事务内的SQL操作本就是有序的(即一个SQL操作执行完毕，才会开始执行下一个SQL操作)，不需要再进行排序，只有不同事务之间前后顺序才可能会发生错乱，因而本说明书中的提交时间戳是针对事务整体的，也即一个事务对应一个提交时间戳。
步骤624：计算模块检测到表格处理事件，若所述表格处理事件的操作类型属于第一类型，则确定所述表格处理事件对应的兼容类型为第一版本兼容第二版本，并确定所述表格处理事件的执行逻辑为后序执行，在生成所有第一版本的数据对应的二进制日志之后，获取所述表格处理事件的时间戳；若所述表格处理事件的操作类型属于第二类型，则确定所述表格处理事件对应的兼容类型为第二版本兼容第一版本，并确定所述表格处理事件的执行逻辑为前序执行，在生成所有第二版本的数据对应的二进制日志之前，获取所述表格处理事件的时间戳；根据所述时间戳，生成所述表格处理事件的二进制日志。
步骤626：排序模块根据表格处理事件的时间戳，将所述表格处理事件的二进制日志插入所述消息队列中。
接下来，参见图7，假设事务中包括2个SQL操作，对上述完整的二进制日志生成过程进行举例说明：
应用程序向计算模块发送事务开启请求，该事务开启请求指示开启的目标事务为事务A，该事务A包括2个操作：操作1和操作2，且操作1由存储模块1执行，操作2由存储模块2执行。应用程序向计算模块发送操作1，计算模块向存储模块1发送操作1开启请求，存储模块1接收到该开启请求后，向计算模块返回开启消息，计算模块接收到该开启消息后，向存储模块1发送操作1的执行指令，存储模块1接收到该执行指令后，执行操作1，操作1执行成功后，向计算模块返回操作1执行成功的消息。
计算模块接收到操作1执行成功的消息后，还可以将该操作1执行成功的消息返回给应用程序，进一步的应用程序将向计算模块发送操作2，之后计算模块可以向存储模块2发送操作2的开启请求，存储模块2接收到该开启请求后，向计算模块返回开启消息，计算模块接收到该开启消息后，向存储模块2发送操作2的执行指令，存储模块2接收到该执行指令后，执行操作2，操作2执行成功后，向计算模块返回操作2执行成功的消息。
计算模块接收到操作2执行成功的消息，向应用程序返回事务A执行完毕的消息(即操作2执行成功的消息)，应用程序接收到该消息后，向计算模块发送提交指令，计算模块向时间戳产生器获取全局有序的时间戳，然后将该时间戳作为事务A的提交时间戳发送给存储模块1和存储模块2，存储模块1和存储模块2分别根据该提交时间戳生成事务A的二进制日志，并返回成功生成二进制日志的消息。
接下来，结合附图8，对上述排序过程进行完整举例说明：
假设有4个存储模块，分别为存储模块1、存储模块2、存储模块3、存储模块4，其分别对应执行模块1、执行模块2、执行模块3、执行模块4。假设存储模块1中生成了二进制日志1(时间戳为0002)、二进制日志2(时间戳为0000)，存储模块2中生成了二进制日志3(时间戳为0005)、二进制日志4(时间戳为0001)，存储模块3中生成了二进制日志5(时间戳为0006)、二进制日志6(时间戳为0003)，存储模块4 中生成了二进制日志7(时间戳为0007)、二进制日志8(时间戳为0004)。执行模块1、执行模块2、执行模块3、执行模块4分别生成对应的内存序列，排序模块读取4个执行模块的内存序列，得到8个二进制日志，将其按照时间戳进行排序，得到二进制日志2、二进制日志4、二进制日志1、二进制日志6、二进制日志8、二进制日志3、二进制日志5、二进制日志7，将该8个二进制日志按照排好的顺序放入消息队列中，即生成逻辑binlog。
本说明书中提供的数据处理方法，目标存储模块可以根据目标事务的提交时间戳生成二进制日志，然后由排序模块按照所述目标事务的提交时间戳，对所有二进制日志进行排序，将排序后的二进制日志写入消息队列中。这种情况下，每个二进制日志中都携带一个全局有序的时间戳，消息队列中会包括分布式数据库系统中所有存储模块生成的二进制日志，且都是按时间先后顺序有序排列，下游应用需要订阅分布式数据库中的数据时，可以直接读取该消息队列，使得分布式数据库系统中时间上前后的两次操作生成的二进制日志都是按序排列，从而避免了事务的操作顺序发生错乱，保证了下游应用订阅数据的一致性。
需要说明的是，此实施例中的技术方案与上述的图2、图4和图5所示的数据处理方法的技术方案属于同一构思，该数据处理方法的技术方案未详细描述的细节内容，均可以参见上述图2、图4和图5所示的数据处理方法的技术方案的描述。
图9示出了根据本说明书一实施例提供的一种计算设备900的结构框图。该计算设备900的部件包括但不限于存储器910和处理器920。处理器920与存储器910通过总线930相连接，数据库950用于保存数据。
计算设备900还包括接入设备940，接入设备940使得计算设备900能够经由一个或多个网络960通信。这些网络的示例包括公用交换电话网(PSTN)、局域网(LAN)、广域网(WAN)、个域网(PAN)或诸如因特网的通信网络的组合。接入设备940可以包括有线或无线的任何类型的网络接口(例如，网络接口卡(NIC))中的一个或多个，诸如IEEE802.11无线局域网(WLAN)无线接口、全球微波互联接入(Wi-MAX)接口、以太网接口、通用串行总线(USB)接口、蜂窝网络接口、蓝牙接口、近场通信(NFC)接口，等等。
在本说明书的一个实施例中，计算设备900的上述部件以及图9中未示出的其他部件也可以彼此相连接，例如通过总线。应当理解，图9所示的计算设备结构框图仅仅是出于示例的目的，而不是对本说明书范围的限制。本领域技术人员可以根据需要，增添或替换其他部件。
计算设备900可以是任何类型的静止或移动计算设备，包括移动计算机或移动计算设备(例如，平板计算机、个人数字助理、膝上型计算机、笔记本计算机、上网本等)、移动电话(例如，智能手机)、可佩戴的计算设备(例如，智能手表、智能眼镜等)或 其他类型的移动设备，或者诸如台式计算机或PC的静止计算设备。计算设备900还可以是移动式或静止式的服务器。
其中，处理器920用于执行如下计算机可执行指令：
接收事务开启请求；
根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；
向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；
所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
图10示出了根据本说明书一实施例提供的一种计算设备1000的结构框图。该计算设备1000的部件包括但不限于存储器1010和处理器1020。处理器1020与存储器1010通过总线1030相连接，数据库1050用于保存数据。
计算设备1000还包括接入设备1040，接入设备1040使得计算设备1000能够经由一个或多个网络1060通信。这些网络的示例包括公用交换电话网(PSTN)、局域网(LAN)、广域网(WAN)、个域网(PAN)或诸如因特网的通信网络的组合。接入设备1040可以包括有线或无线的任何类型的网络接口(例如，网络接口卡(NIC))中的一个或多个，诸如IEEE 802.11无线局域网(WLAN)无线接口、全球微波互联接入(Wi-MAX)接口、以太网接口、通用串行总线(USB)接口、蜂窝网络接口、蓝牙接口、近场通信(NFC)接口，等等。
在本说明书的一个实施例中，计算设备1000的上述部件以及图10中未示出的其他部件也可以彼此相连接，例如通过总线。应当理解，图10所示的计算设备结构框图仅仅是出于示例的目的，而不是对本说明书范围的限制。本领域技术人员可以根据需要，增添或替换其他部件。
计算设备1000可以是任何类型的静止或移动计算设备，包括移动计算机或移动计算设备(例如，平板计算机、个人数字助理、膝上型计算机、笔记本计算机、上网本等)、移动电话(例如，智能手机)、可佩戴的计算设备(例如，智能手表、智能眼镜等)或其他类型的移动设备，或者诸如台式计算机或PC的静止计算设备。计算设备1000还可以是移动式或静止式的服务器。
其中，处理器1020用于执行如下计算机可执行指令：
接收到执行指令后，根据所述执行指令执行目标事务，所述执行指令中携带有待执行目标事务；
所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息；
接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
图11示出了根据本说明书一实施例提供的一种计算设备1100的结构框图。该计算设备1100的部件包括但不限于存储器1110和处理器1120。处理器1120与存储器1110通过总线1130相连接，数据库1150用于保存数据。
计算设备1100还包括接入设备1140，接入设备1140使得计算设备1100能够经由一个或多个网络1160通信。这些网络的示例包括公用交换电话网(PSTN)、局域网(LAN)、广域网(WAN)、个域网(PAN)或诸如因特网的通信网络的组合。接入设备1140可以包括有线或无线的任何类型的网络接口(例如，网络接口卡(NIC))中的一个或多个，诸如IEEE 802.11无线局域网(WLAN)无线接口、全球微波互联接入(Wi-MAX)接口、以太网接口、通用串行总线(USB)接口、蜂窝网络接口、蓝牙接口、近场通信(NFC)接口，等等。
在本说明书的一个实施例中，计算设备1100的上述部件以及图11中未示出的其他部件也可以彼此相连接，例如通过总线。应当理解，图11所示的计算设备结构框图仅仅是出于示例的目的，而不是对本说明书范围的限制。本领域技术人员可以根据需要，增添或替换其他部件。
计算设备1100可以是任何类型的静止或移动计算设备，包括移动计算机或移动计算设备(例如，平板计算机、个人数字助理、膝上型计算机、笔记本计算机、上网本等)、移动电话(例如，智能手机)、可佩戴的计算设备(例如，智能手表、智能眼镜等)或其他类型的移动设备，或者诸如台式计算机或PC的静止计算设备。计算设备1100还可以是移动式或静止式的服务器。
其中，处理器1120用于执行如下计算机可执行指令：
读取每一个执行模块的内存序列；
获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；
按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；
将排序后的二进制日志写入消息队列中。
上述为本实施例的三种计算设备的示意性方案。需要说明的是，该计算设备的技术方案与上述的数据处理方法的技术方案属于同一构思，计算设备的技术方案未详细描述的细节内容，均可以参见上述数据处理方法的技术方案的描述。
本说明书一实施例还提供一种计算机可读存储介质，其存储有计算机指令，该指令被处理器执行时以用于：
接收事务开启请求；
根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；
向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；
所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
本说明书一实施例还提供一种计算机可读存储介质，其存储有计算机指令，该指令被处理器执行时以用于：
接收到执行指令后，根据所述执行指令执行目标事务，所述执行指令中携带有待执行目标事务；
所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息；
接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
本说明书一实施例还提供一种计算机可读存储介质，其存储有计算机指令，该指令被处理器执行时以用于：
读取每一个执行模块的内存序列；
获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；
按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；
将排序后的二进制日志写入消息队列中。
上述为本实施例的三种计算机可读存储介质的示意性方案。需要说明的是，该存储介质的技术方案与上述的数据处理方法的技术方案属于同一构思，存储介质的技术方案未详细描述的细节内容，均可以参见上述数据处理方法的技术方案的描述。
上述对本说明书特定实施例进行了描述。其它实施例在所附权利要求书的范围内。在一些情况下，在权利要求书中记载的动作或步骤可以按照不同于实施例中的顺序来执行并且仍然可以实现期望的结果。另外，在附图中描绘的过程不一定要求示出的特定顺序或者连续顺序才能实现期望的结果。在某些实施方式中，多任务处理和并行处理也是可以的或者可能是有利的。
所述计算机指令包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质可以包括：能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、电载波信号、电信信号以及软件分发介质等。需要说明的是，所述计算机可读介质包含的内容可以根据司法管辖区内立法和专利实践的要求进行适当的增减，例如在某些司法管辖区，根据立法和专利实践，计算机可读介质不包括电载波信号和电信信号。
需要说明的是，对于前述的各方法实施例，为了简便描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本说明书并不受所描述的动作顺序的限制，因为依据本说明书，某些步骤可以采用其它顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一 定都是本说明书所必须的。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其它实施例的相关描述。
以上公开的本说明书优选实施例只是用于帮助阐述本说明书。可选实施例并没有详尽叙述所有的细节，也不限制该发明仅为所述的具体实施方式。显然，根据本说明书的内容，可作很多的修改和变化。本说明书选取并具体描述这些实施例，是为了更好地解释本说明书的原理和实际应用，从而使所属技术领域技术人员能很好地理解和利用本说明书。本说明书仅受权利要求书及其全部范围和等效物的限制。

Claims (38)

  1. 一种分布式数据库系统，包括计算模块和存储模块；
    所述计算模块，被配置为接收事务开启请求；根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳；
    所述目标存储模块，被配置为接收所述执行指令，根据所述执行指令执行所述目标事务；所述目标事务执行成功的情况下，向所述计算模块返回目标事务执行成功的消息；接收所述提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志。
  2. 根据权利要求1所述的分布式数据库系统，所述系统还包括：多个执行模块和一个排序模块，所述多个执行模块中每一个执行模块均对应有存储模块；
    所述执行模块，被配置为读取自身对应的存储模块的二进制日志，并根据所述二进制日志生成内存序列；
    所述排序模块，被配置为读取每一个所述执行模块的内存序列，获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；将排序后的二进制日志写入消息队列中。
  3. 根据权利要求2所述的分布式数据库系统，所述计算模块进一步被配置为：
    若检测到表格处理事件，则确定所述表格处理事件的执行逻辑；
    根据所述执行逻辑，生成所述表格处理事件的时间戳；
    根据所述时间戳，生成所述表格处理事件的二进制日志。
  4. 根据权利要求3所述的分布式数据库系统，所述计算模块进一步被配置为：
    确定所述表格处理事件的操作类型；
    根据所述操作类型，确定所述表格处理事件对应的兼容类型；
    根据所述兼容类型，确定所述表格处理事件的执行逻辑。
  5. 根据权利要求4所述的分布式数据库系统，所述计算模块进一步被配置为：
    若所述操作类型属于第一类型，则确定所述表格处理事件对应的兼容类型为第一版本兼容第二版本，所述第一版本为所述表格处理事件执行前的版本，所述第二版本为所述表格处理事件执行后的版本；
    若所述操作类型属于第二类型，则确定所述表格处理事件对应的兼容类型为第二版本兼容第一版本。
  6. 根据权利要求5所述的分布式数据库系统，所述计算模块进一步被配置为：
    若所述表格处理事件对应的兼容类型为第一版本兼容第二版本，则确定所述表格处理事件的执行逻辑为后序执行；
    若所述表格处理事件对应的兼容类型为第二版本兼容第一版本，则确定所述表格处理事件的执行逻辑为前序执行。
  7. 根据权利要求6所述的分布式数据库系统，所述计算模块进一步被配置为：
    若所述表格处理事件的执行逻辑为前序执行，则在生成所有第二版本的数据对应的二进制日志之前，获取所述表格处理事件的时间戳；
    若所述表格处理事件的执行逻辑为后序执行，则在生成所有第一版本的数据对应的二进制日志之后，获取所述表格处理事件的时间戳。
  8. 根据权利要求3所述的分布式数据库系统，所述排序模块进一步被配置为：
    根据所述表格处理事件的时间戳，将所述表格处理事件的二进制日志插入所述消息队列中。
  9. 根据权利要求1所述的分布式数据库系统，所述系统还包括时间戳产生器，所述计算模块进一步被配置为：
    向所述时间戳产生器申请全局有序的时间戳；
    将所述时间戳确定为所述目标事务的提交时间戳。
  10. 根据权利要求1所述的分布式数据库系统，所述事务开启请求中携带事务标识及对应的存储模块标识，所述计算模块进一步被配置为：
    将所述事务标识指示的事务确定为目标事务；
    将所述存储模块标识指示的存储模块确定为对应的目标存储模块。
  11. 根据权利要求1所述的分布式数据库系统，所述事务开启请求中携带事务标识，所述计算模块进一步被配置为：
    将所述事务标识指示的事务确定为目标事务；
    根据预设分配规则，确定所述目标事务对应的目标存储模块。
  12. 根据权利要求2所述的分布式数据库系统，所述执行模块进一步被配置为：
    若检测到自身对应的存储模块生成新的二进制日志，则返回执行所述读取所述存储模块的二进制日志的操作步骤；
    相应的，所述排序模块进一步被配置为：
    返回执行所述读取每一个所述执行模块的内存序列的操作步骤；或者，
    获取所述执行模块中增加的二进制日志，将所述二进制日志放入所述消息队列的队尾。
  13. 根据权利要求11所述的分布式数据库系统，所述计算模块进一步被配置为：
    记录所述目标存储模块的存储模块标识。
  14. 根据权利要求1所述的分布式数据库系统，所述计算模块进一步被配置为：
    向所述目标存储模块发送目标事务开启请求；
    所述目标存储模块返回开启消息的情况下，执行所述向所述目标存储模块发送执行指令的操作步骤。
  15. 根据权利要求1所述的分布式数据库系统，所述存储模块进一步被配置为：
    生成二进制日志，并将所述目标事务的提交时间戳记录在所述二进制日志的注释中。
  16. 一种数据处理方法，应用于分布式数据库系统中的计算模块，所述方法包括：
    接收事务开启请求；
    根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；
    向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；
    所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
  17. 根据权利要求16所述的数据处理方法，所述方法还包括：
    若检测到表格处理事件，则确定所述表格处理事件的执行逻辑；
    根据所述执行逻辑，生成所述表格处理事件的时间戳；
    根据所述时间戳，生成所述表格处理事件的二进制日志。
  18. 根据权利要求17所述的数据处理方法，所述确定所述表格处理事件的执行逻辑，包括：
    确定所述表格处理事件的操作类型；
    根据所述操作类型，确定所述表格处理事件对应的兼容类型；
    根据所述兼容类型，确定所述表格处理事件的执行逻辑。
  19. 根据权利要求18所述的数据处理方法，所述根据所述操作类型，确定所述表格处理事件对应的兼容类型，包括：
    若所述操作类型属于第一类型，则确定所述表格处理事件对应的兼容类型为第一版本兼容第二版本，所述第一版本为所述表格处理事件执行前的版本，所述第二版本为所述表格处理事件执行后的版本；
    若所述操作类型属于第二类型，则确定所述表格处理事件对应的兼容类型为第二版本兼容第一版本。
  20. 根据权利要求19所述的数据处理方法，所述根据所述兼容类型，确定所述表格处理事件的执行逻辑，包括：
    若所述表格处理事件对应的兼容类型为第一版本兼容第二版本，则确定所述表格处理事件的执行逻辑为后序执行；
    若所述表格处理事件对应的兼容类型为第二版本兼容第一版本，则确定所述表格处理事件的执行逻辑为前序执行。
  21. 根据权利要求20所述的数据处理方法，所述根据所述执行逻辑，生成所述表格处理事件的时间戳，包括：
    若所述表格处理事件的执行逻辑为前序执行，则在生成所有第二版本的数据对应的二进制日志之前，获取所述表格处理事件的时间戳；
    若所述表格处理事件的执行逻辑为后序执行，则在生成所有第一版本的数据对应的二进制日志之后，获取所述表格处理事件的时间戳。
  22. 根据权利要求16所述的数据处理方法，所述向所述目标存储模块发送提交指令之前，还包括：
    向时间戳产生器申请全局有序的时间戳；
    将所述时间戳确定为所述目标事务的提交时间戳。
  23. 根据权利要求16所述的数据处理方法，所述事务开启请求中携带事务标识及对应的存储模块标识，所述根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块，包括：
    将所述事务标识指示的事务确定为目标事务；
    将所述存储模块标识指示的存储模块确定为对应的目标存储模块。
  24. 根据权利要求16所述的数据处理方法，所述事务开启请求中携带事务标识，所述根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块包括：
    将所述事务标识指示的事务确定为目标事务；
    根据预设分配规则，确定所述目标事务对应的目标存储模块。
  25. 根据权利要求24所述的数据处理方法，所述向所述目标存储模块发送执行指令之后，还包括：
    记录所述目标存储模块的存储模块标识。
  26. 根据权利要求16所述的数据处理方法，所述根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块之后，且所述向所述目标存储模块发送执行指令之前，还包括：
    向所述目标存储模块发送目标事务开启请求；
    所述目标存储模块返回开启消息的情况下，执行所述向所述目标存储模块发送执行指令的操作步骤。
  27. 一种数据处理方法，应用于分布式数据库系统中的存储模块，所述方法包括：
    接收到执行指令后，根据所述执行指令执行目标事务，所述执行指令中携带有待执行目标事务；
    所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息；
    接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
  28. 根据权利要求27所述的数据处理方法，所述根据所述目标事务的提交时间戳生成二进制日志，包括：
    生成二进制日志，并将所述目标事务的提交时间戳记录在所述二进制日志的注释中。
  29. 一种数据处理方法，应用于分布式数据库系统中的排序模块，所述方法包括：
    读取每一个执行模块的内存序列；
    获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；
    按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；
    将排序后的二进制日志写入消息队列中。
  30. 根据权利要求29所述的数据处理方法，所述将排序后的二进制日志写入消息队列中之后，还包括：
    若检测到执行模块的内存序列发生变化，则返回执行所述读取每一个所述执行模块的内存序列的操作步骤。
  31. 根据权利要求29所述的数据处理方法，所述将排序后的二进制日志写入消息队列中之后，还包括：
    若检测到执行模块的内存序列发生变化，则获取所述执行模块中增加的二进制日志，将所述二进制日志放入所述消息队列的队尾。
  32. 根据权利要求29所述的数据处理方法，所述将排序后的二进制日志写入消息队列中之后，还包括：
    根据表格处理事件的时间戳，将所述表格处理事件的二进制日志插入所述消息队列中。
  33. 一种计算设备，包括：
    存储器和处理器；
    所述存储器用于存储计算机可执行指令，所述处理器用于执行所述计算机可执行指令：
    接收事务开启请求；
    根据所述事务开启请求，确定目标事务及所述目标事务对应的目标存储模块；
    向所述目标存储模块发送执行指令，所述执行指令中携带有待执行目标事务；
    所述目标存储模块返回目标事务执行成功的消息的情况下，向所述目标存储模块发送提交指令，所述提交指令携带有待提交目标事务和所述目标事务的提交时间戳。
  34. 一种计算设备，包括：
    存储器和处理器；
    所述存储器用于存储计算机可执行指令，所述处理器用于执行所述计算机可执行指令：
    接收到执行指令后，根据所述执行指令执行目标事务，所述执行指令中携带有待执行目标事务；
    所述目标事务执行成功的情况下，向计算模块返回目标事务执行成功的消息；
    接收提交指令，根据所述提交指令对所述目标事务进行提交，并根据所述目标事务的提交时间戳生成二进制日志，所述提交指令携带有待提交目标事务和所述目标事务的 提交时间戳。
  35. 一种计算设备，包括：
    存储器和处理器；
    所述存储器用于存储计算机可执行指令，所述处理器用于执行所述计算机可执行指令：
    读取每一个执行模块的内存序列；
    获取所述内存序列中每一个二进制日志中目标事务的提交时间戳；
    按照所述目标事务的提交时间戳，对所述内存序列中的二进制日志进行排序；
    将排序后的二进制日志写入消息队列中。
  36. 一种计算机可读存储介质，其存储有计算机指令，该指令被处理器执行时实现权利要求16至26任意一项所述数据处理方法的步骤。
  37. 一种计算机可读存储介质，其存储有计算机指令，该指令被处理器执行时实现权利要求27至28任意一项所述数据处理方法的步骤。
  38. 一种计算机可读存储介质，其存储有计算机指令，该指令被处理器执行时实现权利要求29至32任意一项所述数据处理方法的步骤。
PCT/CN2021/107243 2020-07-24 2021-07-20 分布式数据库系统及数据处理方法 WO2022017347A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/101,049 US20230161758A1 (en) 2020-07-24 2023-01-24 Distributed Database System and Data Processing Method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010721315.0 2020-07-24
CN202010721315.0A CN113297320B (zh) 2020-07-24 2020-07-24 分布式数据库系统及数据处理方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/101,049 Continuation US20230161758A1 (en) 2020-07-24 2023-01-24 Distributed Database System and Data Processing Method

Publications (1)

Publication Number Publication Date
WO2022017347A1 true WO2022017347A1 (zh) 2022-01-27

Family

ID=77318655

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/107243 WO2022017347A1 (zh) 2020-07-24 2021-07-20 分布式数据库系统及数据处理方法

Country Status (3)

Country Link
US (1) US20230161758A1 (zh)
CN (1) CN113297320B (zh)
WO (1) WO2022017347A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114691051A (zh) * 2022-05-30 2022-07-01 恒生电子股份有限公司 数据处理方法以及装置
CN115658805A (zh) * 2022-09-15 2023-01-31 星环信息科技(上海)股份有限公司 一种事务一致性管理引擎及方法
CN115905402A (zh) * 2022-09-26 2023-04-04 北京奥星贝斯科技有限公司 处理事务日志的方法及装置

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113448757B (zh) * 2021-08-30 2022-02-01 阿里云计算有限公司 消息处理方法、装置、设备、存储介质和系统
CN114003657A (zh) * 2021-10-11 2022-02-01 阿里云计算有限公司 分布式数据库的数据处理方法、系统、设备和存储介质
CN113656384B (zh) * 2021-10-18 2022-04-08 阿里云计算有限公司 数据处理方法、分布式数据库系统、电子设备及存储介质
CN114691307A (zh) * 2022-03-16 2022-07-01 阿里巴巴（中国）有限公司 事务处理方法及计算机系统
CN114925123B (zh) * 2022-04-24 2024-06-07 杭州悦数科技有限公司 一种分布式的图数据库与图计算系统间的数据传输方法
CN117131060B (zh) * 2023-07-26 2024-08-06 泽拓科技(深圳)有限责任公司 分布式数据库并发控制方法、系统、计算机设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016064575A1 (en) * 2014-10-19 2016-04-28 Microsoft Technology Licensing, Llc High performance transactions in database management systems
CN106991113A (zh) * 2015-12-18 2017-07-28 Sap欧洲公司 数据库环境中的表格复制
CN110019112A (zh) * 2017-08-25 2019-07-16 阿里巴巴集团控股有限公司 数据事务处理方法、装置以及电子设备
CN110018884A (zh) * 2019-03-19 2019-07-16 阿里巴巴集团控股有限公司 分布式事务处理方法、协调装置、数据库及电子设备
CN110457157A (zh) * 2019-08-05 2019-11-15 腾讯科技（深圳）有限公司 分布式事务异常处理方法、装置、计算机设备及存储介质

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8650155B2 (en) * 2008-02-26 2014-02-11 Oracle International Corporation Apparatus and method for log based replication of distributed transactions using globally acknowledged commits
US7962458B2 (en) * 2008-06-12 2011-06-14 Gravic, Inc. Method for replicating explicit locks in a data replication engine
US9984140B1 (en) * 2015-02-05 2018-05-29 Amazon Technologies, Inc. Lease based leader election system
CN106897123B (zh) * 2015-12-21 2021-07-16 阿里巴巴集团控股有限公司 数据库操作方法及装置
CN106897306B (zh) * 2015-12-21 2019-04-30 阿里巴巴集团控股有限公司 数据库操作方法及装置
US10671642B2 (en) * 2016-11-11 2020-06-02 International Business Machines Corporation Copying data changes to a target database
US10474695B1 (en) * 2016-12-20 2019-11-12 Gravic, Inc. Method for replacing a currently operating data replication engine with a new data replication engine without application downtime and while preserving target database consistency
US11645261B2 (en) * 2018-04-27 2023-05-09 Oracle International Corporation System and method for heterogeneous database replication from a remote server
EP3576368A1 (de) * 2018-05-29 2019-12-04 Siemens Aktiengesellschaft Verfahren und system zum steuern einer freigabe einer ressource
CN110196760B (zh) * 2018-07-12 2023-04-18 腾讯科技（深圳）有限公司 分布式事务一致性实现方法及装置
WO2021048572A1 (en) * 2019-09-09 2021-03-18 Citrix Systems, Inc Conversational ai platform using declarative graph model
CN111427966B (zh) * 2020-06-10 2020-09-22 腾讯科技（深圳）有限公司 数据库事务处理方法、装置及服务器
CN111858629B (zh) * 2020-07-02 2023-08-22 北京奥星贝斯科技有限公司 二阶段提交分布式事务更新数据库的实现方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016064575A1 (en) * 2014-10-19 2016-04-28 Microsoft Technology Licensing, Llc High performance transactions in database management systems
CN106991113A (zh) * 2015-12-18 2017-07-28 Sap欧洲公司 数据库环境中的表格复制
CN110019112A (zh) * 2017-08-25 2019-07-16 阿里巴巴集团控股有限公司 数据事务处理方法、装置以及电子设备
CN110018884A (zh) * 2019-03-19 2019-07-16 阿里巴巴集团控股有限公司 分布式事务处理方法、协调装置、数据库及电子设备
CN110457157A (zh) * 2019-08-05 2019-11-15 腾讯科技（深圳）有限公司 分布式事务异常处理方法、装置、计算机设备及存储介质

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114691051A (zh) * 2022-05-30 2022-07-01 恒生电子股份有限公司 数据处理方法以及装置
CN115658805A (zh) * 2022-09-15 2023-01-31 星环信息科技(上海)股份有限公司 一种事务一致性管理引擎及方法
CN115658805B (zh) * 2022-09-15 2023-10-17 星环信息科技(上海)股份有限公司 一种事务一致性管理引擎及方法
CN115905402A (zh) * 2022-09-26 2023-04-04 北京奥星贝斯科技有限公司 处理事务日志的方法及装置
CN115905402B (zh) * 2022-09-26 2023-07-18 北京奥星贝斯科技有限公司 处理事务日志的方法及装置

Also Published As

Publication number Publication date
CN113297320A (zh) 2021-08-24
US20230161758A1 (en) 2023-05-25
CN113297320B (zh) 2024-05-14

Similar Documents

Publication Publication Date Title
WO2022017347A1 (zh) 分布式数据库系统及数据处理方法
KR102141234B1 (ko) 분산된 데이터 스토어 내의 버젼형 계층 데이터 구조
CN109906448B (zh) 用于促进可插拔数据库上的操作的方法、设备和介质
CN111797121B (zh) 读写分离架构业务系统的强一致性查询方法、装置及系统
CN109491989B (zh) 数据处理方法及装置、电子设备、存储介质
CN111190935B (zh) 数据读取方法、装置、计算机设备及存储介质
WO2021238701A1 (zh) 数据迁移方法以及装置
US20150032695A1 (en) Client and server integration for replicating data
CN111324610A (zh) 一种数据同步的方法及装置
US10509773B2 (en) DBFS with flashback archive
WO2015062181A1 (zh) 用于实现多源异构数据资源自动同步的方法
WO2021254240A1 (zh) 数据处理方法及装置
JPWO2011108695A1 (ja) 並列データ処理システム、並列データ処理方法及びプログラム
US20190361999A1 (en) Data analysis over the combination of relational and big data
CN102254021A (zh) 基于虚拟机管理系统的数据库构建方法
CN107977446A (zh) 一种基于数据分区的内存网格数据加载方法
WO2023061249A1 (zh) 分布式数据库的数据处理方法、系统、设备和存储介质
WO2021254418A1 (zh) 数据读取方法及装置
CN108536833A (zh) 一种分布式、面向大数据的数据库及其构建方法
CN114860727A (zh) 拉链表更新方法及装置
CN104794567B (zh) 一种基于HBase的传染病数据管理方法
CN104572638B (zh) 数据读写方法及装置
CN112181950B (zh) 一种分布式对象数据库的构建方法
CN113297231A (zh) 数据库处理方法及装置
CN117056305A (zh) 多源同构数据库的构建方法、模型、数据库系统及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21846592

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21846592

Country of ref document: EP

Kind code of ref document: A1