WO2025219804A1 - Ddl语句处理 - Google Patents
Ddl语句处理Info
- Publication number
- WO2025219804A1 WO2025219804A1 PCT/IB2025/053633 IB2025053633W WO2025219804A1 WO 2025219804 A1 WO2025219804 A1 WO 2025219804A1 IB 2025053633 W IB2025053633 W IB 2025053633W WO 2025219804 A1 WO2025219804 A1 WO 2025219804A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- ddl
- thread
- target
- processing
- statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
- G06F16/2448—Query languages for particular applications; for extensibility, e.g. user defined types
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Definitions
- the present disclosure relates to the field of computer technology, and in particular to DDL statement processing.
- MySQL is a relational database management system.
- Sharding i.e., sharding
- sharding technology can be used to split the logical table containing the complete data of the distributed database into multiple physical shards corresponding to the logical table.
- the multiple physical shards are stored in the storage nodes of the distributed database.
- Data Definition Language is a type of Structured Query Language (SQL) used to define database objects such as tables, columns, and indexes.
- SQL Structured Query Language
- DDL statements are executed on logical tables, but due to the sharding mechanism in distributed databases, DDL statements need to be executed separately on different physical shards corresponding to the logical tables.
- DDL statements are used to modify the metadata of physical shards, such as adding or removing a column field, changing the byte length of a column field, or modifying the data format of a column field.
- the statement when executing a DDL statement in sequence on different physical shards, the statement may be executed successfully on some physical shards but not on other physical shards, resulting in inconsistent metadata corresponding to multiple physical shards, affecting the use of the database.
- the embodiments of the present disclosure provide a DDL statement processing method, system, device, and storage medium to achieve consistency of physical shard metadata during the execution of DDL statements on different physical shards.
- an embodiment of the present disclosure provides a DDL statement processing method, which is applied to a storage node in a distributed database system, wherein the storage node executes a control thread and at least one DDL thread corresponding to at least one locally stored physical shard, the method comprising: if a first indication signal is received from a computing node through the control thread to indicate entering a target processing phase of the DDL statement, then a first notification signal corresponding to the first indication signal is sent to the at least one DDL thread through the control thread, wherein the DDL statement is used to modify metadata of the at least one physical shard; a processing behavior corresponding to the target processing phase is executed through the at least one DDL thread, and after executing the processing behavior, a second notification signal indicating that a target synchronization point has been reached is sent to the control thread, wherein the target synchronization point corresponds to the target processing phase; and a second indication signal corresponding to the target synchronization point is sent to the computing node through the control
- an embodiment of the present disclosure provides a DDL statement processing device, which is applied to a storage node in a distributed database system, wherein the storage node executes a control thread and at least one DDL thread corresponding to at least one physical shard stored locally, and the device comprises: a first receiving module, which is used to send a first notification signal corresponding to the first indication signal to the at least one DDL thread through the control thread if a first indication signal is received from the computing node through the control thread to indicate entering a target processing stage of the DDL statement, wherein the DDL statement is used to modify the metadata of the at least one physical shard; a first processing module, which is used to execute a processing behavior corresponding to the target processing stage through the at least one DDL thread, and send a second notification signal indicating that a target synchronization point has been reached to the control thread after executing the processing behavior, wherein the target synchronization point corresponds to the target processing stage; a first sending module, which is also used to
- an embodiment of the present disclosure provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the DDL statement processing method as described in the first aspect.
- an embodiment of the present disclosure provides a non-transitory machine-readable storage medium having executable code stored thereon.
- the processor can at least implement the DDL statement processing method as described in the first aspect.
- an embodiment of the present disclosure provides a DDL statement processing method, which is applied to a computing node in a distributed database system, wherein the computing node communicates with a storage node in the distributed database system, and the storage node executes a control thread and at least one DDL thread corresponding to at least one physical shard stored locally.
- the method comprises: sending a first indication signal to the control thread for indicating entry into a target processing phase of the DDL statement, so that the control thread sends a first notification signal corresponding to the first indication signal to the at least one DDL thread respectively, causing the at least one DDL thread to execute a processing behavior corresponding to the target processing phase, and after executing the processing behavior, sending a second notification signal to the control thread indicating that a target synchronization point has been reached, wherein the target synchronization point corresponds to the target processing phase, and the DDL statement is used to modify the metadata of the at least one physical shard; receiving a second indication signal corresponding to the target synchronization point sent by the control thread, wherein the second indication signal indicates that the at least one DDL thread has issued the second notification signal; and determining the execution status of the DDL statement according to the second indication signal.
- an embodiment of the present disclosure provides a DDL statement processing device, which is applied to a computing node in a distributed database system, wherein the computing node communicates with a storage node in the distributed database system, wherein the storage node includes a control thread and at least one DDL thread corresponding to at least one locally stored physical shard, and the device includes: a second sending module, which is used to send a first indication signal to the control thread for indicating entry into a target processing phase of the DDL statement, so that the control thread sends a first notification signal corresponding to the first indication signal to the at least one DDL thread respectively, so that the at least one DDL thread executes a processing behavior corresponding to the target processing phase, and after executing the processing behavior, sends a second notification signal to the control thread indicating that a target synchronization point has been reached, wherein the target synchronization point corresponds to the target processing phase, and the DDL statement is used to modify the metadata of the at least one physical shard;
- an embodiment of the present disclosure provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the DDL statement processing method as described in the fifth aspect.
- an embodiment of the present disclosure provides a non-transitory machine-readable storage medium, on which executable code is stored.
- the processor can at least implement the DDL statement processing method described in the fifth aspect.
- an embodiment of the present disclosure provides a distributed database system, comprising: a computing node and a storage node, wherein the storage node executes a control thread and at least one DDL thread corresponding to at least one physical shard stored locally; the computing node is used to send a first indication signal to the control thread for indicating entry into a target processing phase of a DDL statement, wherein the DDL statement is used to modify the metadata of the at least one physical shard; the control thread is used to send a first notification signal corresponding to the first indication signal to each of the at least one DDL threads; the at least one DDL thread is used to execute a processing behavior corresponding to the target processing phase based on the first notification signal, and after executing the processing behavior, send a second notification signal to the control thread indicating that a target synchronization point has been reached, wherein the target synchronization point corresponds to the target processing phase; the control thread is also used to send a second indication signal corresponding to the target synchronization
- the solution provided by the embodiments of the present disclosure can be applied to a relational distributed database, which includes but is not limited to a computing node and a storage node connected in communication, wherein the storage node executes a control thread and at least one DDL thread corresponding to at least one physical shard stored locally.
- the storage node receives a first indication signal sent by the computing node through the control thread to indicate entry into the target processing stage of the DDL statement, and sends a first notification signal corresponding to the first indication signal to at least one DDL thread through the control thread to notify the at least one DDL thread to execute the processing behavior corresponding to the target processing stage.
- the storage node sends a second notification signal to the control thread indicating that the target synchronization point corresponding to the target processing stage has been reached.
- the storage node sends a second indication signal corresponding to the target synchronization point to the computing node through the control thread, so that the computing node determines the execution status of the DDL statement based on the second indication signal. That is, if the computing node receives the second indication signal, it indicates that at least one DDL thread has completed the processing behavior corresponding to the target processing stage; if the computing node does not receive the second indication signal, it indicates that at least one DDL thread has not completed the processing behavior corresponding to the target processing stage.
- the compute node receives the second indication signal during the last processing phase corresponding to the statement, it indicates that the meta information of at least one physical shard has been modified, i.e., the meta information of the physical shard has been consistently modified. If the compute node does not receive the second indication signal, it can notify at least one DDL thread to perform processing such as a rollback, i.e., return to the state before executing the DDL statement. This ensures that the meta information of the at least one physical shard remains in the state before executing the DDL statement, thus ensuring the consistency of the meta information of different physical shards.
- Figure 1 is a schematic diagram of a distributed database system provided in an embodiment of the present disclosure
- Figure 2 is an interactive flow chart of a DDL statement processing method provided in an embodiment of the present disclosure
- Figure 3 is an interactive flow chart of another DDL statement processing method provided in an embodiment of the present disclosure.
- Figure 4 is a schematic diagram of a DDL statement processing process provided by an embodiment of the present disclosure.
- FIG. 5 is a schematic diagram of another DDL statement processing process provided by an embodiment of the present disclosure.
- FIG. 6 is a flowchart of a DDL statement processing method provided in an embodiment of the present disclosure.
- FIG. 7 is a flowchart of another DDL statement processing method provided by an embodiment of the present disclosure.
- FIG8 is a schematic diagram of the structure of a DDL statement processing device provided in an embodiment of the present disclosure.
- FIG9 is a schematic diagram of the structure of an electronic device corresponding to the DDL statement processing device provided in the embodiment shown in FIG8;
- FIG10 is a schematic diagram of the structure of another DDL statement processing device provided in an embodiment of the present disclosure.
- FIG11 is a schematic diagram of the structure of an electronic device corresponding to the DDL statement processing device provided in the embodiment shown in FIG10.
- the user information including but not limited to user device information, user personal information, etc.
- data including but not limited to data used for analysis, stored data, displayed data, etc.
- the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
- MySQL is a relational database management system.
- Physical shards also known as physical tables, are used to store data in the database locally on the storage node.
- Sharding i.e., sharding
- sharding technology can be used to split the logical table containing the complete data of the distributed database (for example, vertical splitting, horizontal splitting, etc.) to obtain multiple physical shards corresponding to the logical table.
- Physical Shards Multiple physical shards are stored in the storage nodes of a distributed database. The data stored in all physical shards together represent the complete database data contained in the logical table.
- Logical meta-information i.e., meta-information corresponding to the logical table, records the storage locations of multiple physical shards corresponding to the logical table, the data format and column index information of the data stored in each physical shard, etc.
- DDL statements are a type of SQL statement used to define database objects such as tables, columns, and indexes. In this solution, DDL statements are used to modify the metadata of physical shards, such as adding or removing a column, changing the byte length of a column, or changing the data format of a column.
- DML Data Manipulation Language
- Metadata lock including MDL read lock and MDL write lock, is used to protect the structure of database objects such as tables, indexes, etc.
- MDL read locks are not mutually exclusive; MDL write locks and MDL write locks and MDL read locks are mutually exclusive, that is, they are exclusive.
- Data dictionary lock i.e. metadata lock at the storage engine layer in the MySQL logical system architecture (including the MySQL service layer and the storage engine layer)
- DDL thread that is, the thread that executes DDL statements on the storage node.
- Control thread i.e., the thread on the storage node that detects the execution status of DDL statements and sends notification signals to the DDL thread. This notification signal is used to notify the DDL thread to perform the processing actions associated with the DDL statement.
- the operation (execution) object of the DDL statement is the logical table.
- the DDL statement needs to be executed separately on different physical shards corresponding to the logical table.
- the metadata corresponding to the logical table that is, the logical metadata, is updated.
- the embodiments of the present disclosure provide a DDL statement processing solution. Based on the solution provided by the embodiments of the present disclosure, the DDL statement execution progress of multiple physical shards can be synchronized, and the consistency of the meta information of multiple physical shards can be achieved. This effectively solves the problem of inconsistent meta information of multiple physical shards and long-term inconsistency between logical meta information and physical shard meta information caused by failure to execute DDL statements on some physical shards.
- FIG1 is a schematic diagram of a distributed database system provided by the embodiments of the present disclosure. As shown in FIG1 , the distributed database system includes computing nodes and storage nodes, which are communicatively connected with each other.
- the compute node serves as the entry point of the distributed database system and can be used to receive DDL statements issued by the client and trigger the storage node to execute the DDL statements on the corresponding physical shard.
- the storage node executes a control thread and at least one DDL thread corresponding to at least one physical shard of local storage, for example: DDL thread 1 corresponding to physical shard 1 in Figure 1, DDL thread 2 corresponding to physical shard 2, etc.
- FIG2 is an interactive flow chart of a DDL statement processing method provided in an embodiment of the present disclosure. As shown in FIG2 , the method may include the following steps.
- the computing node sends a first indication signal to the control thread of the storage node for indicating entry into a target processing phase of a DDL statement, where the DDL statement is used to modify metadata of at least one physical shard stored locally by the storage node.
- the storage node If the storage node receives a first indication signal through the control thread, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread corresponding to at least one physical shard through the control thread.
- the storage nodes each execute the processing behavior corresponding to the target processing stage through at least one DDL thread.
- At least one DDL thread of the storage node After executing the processing behavior, at least one DDL thread of the storage node sends a second notification signal to the control thread indicating that a target synchronization point has been reached, where the target synchronization point corresponds to the target processing stage.
- the storage node sends a second indication signal corresponding to the target synchronization point to the computing node through the control thread, and the second indication signal indicates that at least one DDL thread sends a second notification signal.
- the computing node receives the second indication signal and determines the execution status of the DDL statement based on the second indication signal.
- the execution process of any DDL statement issued by the client can be generally divided into three processing stages: initial processing stage, preparation processing stage, and submission processing stage.
- Each processing stage requires completing different processing behaviors.
- the initial processing stage requires completing the acquisition of the MDL read lock and copying the existing data and incremental data in the physical shard
- the preparation processing stage requires completing the upgrade of the MDL read lock to the MDL write lock and performing data consistency verification
- the submission processing stage requires completing the acquisition of the data dictionary lock and switching between the old and new physical shards.
- the initial processing stage, the preparation processing stage and the submission processing stage are collectively referred to as the target processing stage in the embodiment shown in FIG2 , that is, the target processing stage can actually represent any one of the initial processing stage, the preparation processing stage and the submission processing stage.
- the target processing stage represents different processing stages of a DDL statement
- the first indication signal, the second indication signal, the first notification signal, the second notification signal, and the target synchronization point that directly or indirectly correspond to the target processing stage in the embodiment shown in FIG2 all have different meanings.
- the client sends a DDL statement to the computing node, where the DDL statement is used to modify the metadata of at least one physical shard stored locally on the storage node.
- the format of the DDL statement is "alter table info", where info represents the specific operation behavior, for example: alter table tl add column t2 int, which means adding a column named t2 to the physical shard tl, and the data type of the column is integer.
- the computing node After receiving the DDL statement, the computing node notifies the storage node that is in communication with the computing node to start the corresponding control thread, and informs the storage node of at least one physical shard on which the DDL statement needs to be executed, so that the storage node starts at least one corresponding DDL thread for the at least one physical shard.
- the computing node sends a first indication signal to the control thread of the storage node to indicate that it is entering the initial processing phase of the DDL statement.
- the storage node If the storage node receives a first indication signal sent by the computing node through the control thread to indicate entering the initial processing phase of the DDL statement, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread through the control thread.
- a processing stage flag is preset in the first notification signal, and the processing stage flag can be set to the target processing stage, so as to be used to notify the DDL thread that the processing stage currently corresponding to the DDL statement is the target processing stage.
- the processing stage indicated by the preset processing stage flag bit in the first notification signal being set to different fields can be pre-configured.
- the processing stage flag bit can be pre-configured to indicate that the DDL statement is currently in the initial processing stage when the processing stage flag bit is set to "start”; that the DDL statement is currently in the preparation processing stage when the processing stage flag bit is set to "prepare”; and that the DDL statement is currently in the commit processing stage when the processing stage flag bit is set to "commit”.
- the control thread sends the first notification signal to the DDL thread, it can set the preset processing stage flag bit in the first notification signal to the field indicating the target processing stage based on the pre-configured information.
- the processing stage flag bit preset in the first notification signal is set to the initial processing stage. Specifically, the processing stage flag bit preset in the first notification signal can be set to "start”.
- At least one DDL thread in the storage node executes the processing behavior corresponding to the initial processing phase.
- Each DDL thread executes the processing behavior corresponding to the initial processing phase of the DDL statement.
- a second notification signal is sent to the control thread indicating that the first synchronization point has been reached.
- the first synchronization point corresponds to the initial processing phase and is set for the initial processing phase to confirm whether at least one DDL thread has completed the initial processing phase.
- the control thread in the storage node After receiving the second notification signal sent by the at least one DDL thread, the control thread in the storage node sends a second indication signal corresponding to the first synchronization point to the compute node.
- the second indication signal indicates that the at least one DDL thread has sent the second notification signal, i.e., all DDL threads have completed the initial processing phase of the DDL statement.
- the computing node If the computing node receives the second indication signal sent by the control thread of the storage node, it indicates that at least one of the above DDL threads has completed the processing behavior corresponding to the initial processing phase of the DDL statement and can enter the preparation processing phase of the DDL statement.
- the computing node sends a first indication signal to the control thread of the storage node to indicate entering the preparation processing phase of the DDL statement.
- the storage node If the storage node receives a first indication signal sent by the computing node through the control thread to indicate entering the preparation processing phase of the DDL statement, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread through the control thread.
- the processing stage flag bit preset in the first notification signal is set to the preparation processing stage.
- the processing stage flag bit preset in the first notification signal can be set to "prepare”.
- At least one DDL thread in the storage node executes the processing behavior corresponding to the preparation processing phase.
- each DDL thread sends a second notification signal to the control thread indicating that a second synchronization point has been reached.
- the second synchronization point corresponds to the preparation processing phase, that is, the second synchronization point is a synchronization point set for the preparation processing phase and is used to confirm whether at least one DDL thread has completed the processing behavior of the preparation processing phase.
- the control thread in the storage node After receiving the second notification signal sent by the at least one DDL thread, the control thread in the storage node sends a second indication signal corresponding to the second synchronization point to the compute node.
- the second indication signal indicates that the at least one DDL thread has sent the second notification signal, i.e., all DDL threads have completed the preparation phase of the DDL statement.
- the computing node If the computing node receives the second indication signal sent by the control thread of the storage node, it indicates that at least one of the above DDL threads has completed the processing behavior corresponding to the preparation processing phase of the DDL statement and can enter the submission processing phase of the DDL statement.
- the computing node sends a first indication signal to the control thread of the storage node to indicate entering the commit processing phase of the DDL statement.
- the storage node If the storage node receives a first indication signal sent by the computing node through the control thread for indicating entry into the commit processing phase of the DDL statement, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread through the control thread.
- the processing stage flag bit preset in the first notification signal is set to the commit processing stage.
- the processing stage flag bit preset in the first notification signal can be set to "commit”.
- At least one DDL thread in the storage node executes the processing behavior corresponding to the commit processing phase.
- Each DDL thread executes the processing behavior corresponding to the commit processing phase of the DDL statement.
- a second notification signal is sent to the control thread indicating that the third synchronization point has been reached.
- the third synchronization point corresponds to the commit processing phase. Specifically, the third synchronization point is set for the commit processing phase and is used to confirm whether at least one DDL thread has completed the commit processing phase.
- the control thread in the storage node After receiving the second notification signal sent by the at least one DDL thread, the control thread in the storage node sends a second indication signal corresponding to the third synchronization point to the compute node.
- the second indication signal indicates that the at least one DDL thread has sent the second notification signal, i.e., all DDL threads have completed the commit processing phase of the DDL statement.
- the computing node receives the second indication signal sent by the control thread of the storage node, it indicates that the at least one DDL thread has completed the processing behavior corresponding to the submission processing stage of the DDL statement, that is, the execution of the DDL statement is completed, and the metadata of the at least one physical shard has been modified. At this time, the metadata of the at least one physical shard is consistent, and the logical metadata can be further modified.
- the control thread in the storage node confirms whether at least one DDL thread has completed the processing of the DDL statement in the corresponding processing stage at the first synchronization point corresponding to the initial processing stage, the second synchronization point corresponding to the preparation processing stage, and the third synchronization point corresponding to the commit processing stage.
- a second indication signal is sent to the computing node.
- the second indication signal is not sent to the computing node.
- the computing node determines that the DDL statement execution has failed and sends a rollback instruction to the at least one DDL thread in the storage node. If the storage node receives the rollback instruction sent by the computing node through at least one DDL thread, it performs a rollback process, that is, deletes all processing behaviors corresponding to the completed DDL statement and returns to the state before the DDL statement is executed.
- the storage node receives the rollback instruction sent by the computing node through at least one DDL thread at the first synchronization point, then at least one DDL thread deletes the processing behaviors executed in the initial processing stage of the DDL statement; if the storage node receives the rollback instruction sent by the computing node through at least one DDL thread at the second synchronization point, then at least one DDL thread deletes the processing behaviors executed in the initial processing stage and the preparation processing stage of the DDL statement.
- the computing node when it determines that at least one DDL thread has executed abnormally, it may also send a rollback instruction to the above-mentioned at least one DDL thread in the storage node, so that the storage node performs the rollback process after receiving the rollback instruction sent by the computing node through the at least one DDL thread.
- the storage node receives, through the control thread, a first indication signal sent by the computing node to indicate entry into the target processing stage of the DDL statement, and sends, through the control thread, a first notification signal corresponding to the first indication signal to at least one DDL thread, to notify the at least one DDL thread to execute the processing behavior corresponding to the target processing stage.
- the storage node After executing the processing behavior corresponding to the target processing stage through at least one DDL thread, the storage node sends a second notification signal to the control thread indicating that the target synchronization point corresponding to the target processing stage has been reached. If at least one DDL thread sends the second notification signal, the storage node sends, through the control thread, a second indication signal corresponding to the target synchronization point to the computing node, so that the computing node determines the execution status of the DDL statement based on the second indication signal. That is, if the computing node receives the second indication signal, it indicates that at least one DDL thread has completed the processing behavior corresponding to the target processing stage.
- the target processing stage is the last processing stage corresponding to the DDL statement, it indicates that the meta information of at least one physical shard has been modified, that is, the consistent modification of the physical shard meta information has been completed.
- the computing node does not receive the second indication signal, it indicates that there is a DDL thread in at least one DDL thread that has not completed the processing behavior corresponding to the target processing stage.
- at least one DDL thread can be notified to perform rollback processing, that is, return to the state before executing the DDL statement.
- rollback processing that is, return to the state before executing the DDL statement.
- the meta information of at least one physical shard is still the state before executing the DDL statement, ensuring the consistency of the meta information of the physical shard.
- setting corresponding synchronization points at different processing stages corresponding to the DDL statement can ensure that the DDL statement is consistent with the state before executing the DDL statement.
- timely processing is performed, such as consistent rollback, thereby improving the efficiency of modifying the metadata of at least one physical shard.
- FIG3 is an interactive flow chart of another DDL statement processing method provided by an embodiment of the present disclosure.
- DDL statement processing has three processing stages, namely, the initial processing stage, the preparation processing stage, and the submission processing stage. In each processing stage, the computing node and the storage node interact accordingly to complete the corresponding processing behavior.
- the DDL statement processing method may include the following steps.
- the computing node sends a first indication signal to the control thread of the storage node to indicate entering the initial processing phase of a DDL statement, where the DDL statement is used to modify the metadata of at least one physical shard stored locally in the storage node.
- the storage node If the storage node receives the first indication signal through the control thread, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread corresponding to at least one physical shard through the control thread, wherein a processing stage flag bit preset in the first notification signal is set to the initial processing stage.
- step 301 and step 302 can be referred to the aforementioned embodiment and will not be described in detail in this embodiment.
- the computing node sends the DDL statement to the target thread, the target DDL thread is any one of the at least one DDL thread, and the computing node sequentially sends the DDL statement to the at least one DDL thread.
- the storage node executes the initialization behavior and data copy behavior corresponding to the initial processing phase through the target DDL thread.
- the target DDL thread of the storage node After executing the data copy behavior, the target DDL thread of the storage node sends a second notification signal to the control thread indicating that the first synchronization point has been reached.
- the first synchronization point corresponds to the initial processing stage.
- the processing behaviors corresponding to the initial processing phase include: initialization behavior and data copy behavior.
- the computing node sends a DDL statement to at least one DDL thread of the storage node, so that at least one DDL thread executes the initialization behavior and data copy behavior corresponding to the initial processing phase.
- the computing node in the storage node sends the DDL statement to at least one DDL thread in sequence.
- the metadata of two physical shards, the first physical shard and the second physical shard, in the storage node need to be modified, where the first physical shard corresponds to the first DDL thread and the second physical shard corresponds to the second DDL thread.
- the computing node first sends a DDL statement to one of the first and second DDL threads.
- the computing node first sends the DDL statement to the first DDL thread, so that the storage node, in response to receiving the DDL statement through the first DDL thread, executes the initialization behavior and data copy behavior corresponding to the initial processing phase through the first DDL thread.
- the first DDL thread After executing the data copy, not only sends a second notification signal indicating that the first synchronization point has been reached to the control thread, but also sends a second notification signal indicating that the first synchronization point has been reached to the computing node.
- the computing node After receiving the second notification signal sent by the storage node through the first DDL thread, the computing node sends the DDL statement to the second DDL thread, so that the storage node, in response to receiving the DDL statement through the second DDL thread, executes the initialization behavior and data copy behavior corresponding to the initial processing phase through the second DDL thread.
- the storage node executes the initialization behavior and data copy behavior corresponding to the initial processing stage through the target DDL thread in the following manner: obtaining the MDL read lock through the target DDL thread; copying the existing data in the target physical shard to the target file through the target DDL thread, obtaining the log file that records the incremental data update information generated since the copying of the existing data began, and replaying the log file on the target file after the copying of the existing data is completed; when the amount of data remaining in the log file that has not been replayed does not exceed the set threshold, sending a second notification signal indicating that the first synchronization point has been reached to the control thread through the target DDL thread.
- the target physical shard is a physical shard in at least one physical shard that corresponds to the target DDL thread.
- Metadata locks include MDL read locks and MDL write locks. MDL read locks are not mutually exclusive, but MDL write locks and MDL write locks are mutually exclusive, that is, they are exclusive.
- a thread executing DML statements executes multiple DML statements, it can simultaneously obtain an MDL read lock for each DML statement through the thread, and can simultaneously perform add, delete, update, or query operations on the data in the physical shard through the DML statements.
- a DDL thread executes multiple DDL statements, it can only obtain an MDL write lock for one DDL statement at a time through the DDL thread. After obtaining the MDL write lock for a DDL statement, neither the thread executing DML statements nor the DDL thread can obtain an MDL write lock or an MDL read lock for other DML statements or DDL statements.
- the target DDL thread first acquires the MDL read lock. If the target DDL thread cannot acquire the MDL read lock, it indicates that other DDL statements are currently modifying the metadata of the target physical shard. It is necessary to continue waiting until the MDL read lock is acquired, or the DDL statement execution failure is confirmed when the wait timeout occurs. If the target DDL thread can acquire the MDL read lock, it indicates that no other DDL statements are currently modifying the metadata of the target physical shard. The DDL statement sent by the received compute node can be executed.
- the initialization behavior is considered to be completed. After that, the data copy behavior is performed.
- the data copy includes copying the existing data in the target physical shard and copying the incremental data in the target shard.
- the target DDL thread copies the existing data in the target physical shard to the target file, and obtains the log file (i.e., online log) that records the incremental data update information generated since the existing data copying began. Therefore, after the existing data copy is completed, the log file can be replayed on the target file to complete the incremental data copy.
- the target file can be understood as a temporary file.
- a threshold value may be customized.
- a second notification signal indicating that the first synchronization point has been reached is sent to the control thread through the target DDL thread.
- the target DDL thread After the storage node sends a second notification signal indicating that the first synchronization point has been reached to the control thread through the target DDL thread, the target DDL thread enters a sleep state.
- the computing node sequentially sends DDL statements to at least one DDL thread. Therefore, when there are multiple DDL threads, the times at which the multiple DDL threads send the second notification signal after executing the processing actions corresponding to the initial processing phase are different.
- the DDL thread that has sent the second notification signal is put into a sleep state, thereby waiting for the results of the processing actions corresponding to the initial processing phase executed by other DDL threads.
- the target DDL thread in response to a notification event indicating that the amount of data remaining in the log file that has not been replayed exceeds a set threshold, continues to replay the log file on the target file, so that the amount of data remaining in the log file that has not been replayed decreases to no more than the set threshold.
- the target DDL thread that has entered a dormant state is awakened, so that the target DDL thread continues to replay the log file on the target file.
- the storage node sends a second indication signal corresponding to the first synchronization point to the computing node through the control thread, and the second indication signal indicates that at least one DDL thread sends a second notification signal.
- the computing node receives a second indication signal corresponding to the first synchronization point, and determines the DDL statement according to the second indication signal. Execution status.
- the computing node If the computing node receives the second indication signal sent by the control thread of the storage node, it indicates that at least one of the above DDL threads has completed the processing behavior corresponding to the initial processing phase of the DDL statement and can enter the preparation processing phase of the DDL statement.
- the DDL statement processing method may include the following steps.
- the computing node sends a first indication signal to the storage node to indicate entering the preparation processing phase of the DDL statement.
- the storage node If the storage node receives a first indication signal through the control thread, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread corresponding to at least one physical shard through the control thread, wherein a processing stage flag bit preset in the first notification signal is set to the preparation processing stage.
- the first notification signal can be used to wake up the DDL thread that enters the sleep state during the initial processing phase.
- the storage node upgrades the MDL read lock to an MDL write lock through the target DDL thread, and performs a data consistency check after replaying the remaining unplayed data in the log file in the target file.
- Data consistency checking is used to ensure that the data in a physical shard is consistent before and after the physical shard's metadata is modified using DDL statements. For example, before modifying the physical shard's metadata, if each piece of data in a column of data in the physical shard is constrained to be non-duplicate, then after modifying the physical shard's metadata, each piece of data in the column of data should also be non-duplicate, thereby ensuring data consistency.
- a data consistency check can be performed as follows: First, a check mark field preset in the target physical shard is determined, and the initial state value corresponding to the check mark field is a first value, wherein the first value indicates that the data corresponding to the check mark field in the target physical shard is non-repeated. Then, a target state value corresponding to the check mark field is determined in the target file. Finally, based on the first value and the target state value, it is determined whether the data consistency check is successful. If the target state value is the second value, it is determined that the data consistency check is unsuccessful, and the second value is different from the first value; if the target state value is the first value, it is determined that the data consistency check is successful.
- the target status value corresponding to the check mark field in the target file will be automatically updated during the process of playing back the log file on the target file.
- the storage node After the data consistency check succeeds, the storage node sends a second notification signal to the control thread through the target DDL thread indicating that the second synchronization point has been reached.
- the second synchronization point corresponds to the preparation processing phase.
- the target DDL thread enters a sleep state.
- the storage node sends a second indication signal corresponding to the second synchronization point to the computing node through the control thread.
- the second indication signal indicates that at least one DDL thread has sent a second notification signal.
- the computing node receives a second indication signal corresponding to the second synchronization point, and determines the execution status of the DDL statement based on the second indication signal.
- the computing node receives the second indication signal sent by the control thread of the storage node, it indicates that at least one of the above-mentioned DDL threads has completed the processing behavior corresponding to the preparation processing phase of the DDL statement and can enter the submission processing phase of the DDL statement.
- the DDL statement processing method may include the following steps.
- the computing node sends a first indication signal to the storage node to indicate entering the commit processing phase of the DDL statement.
- the storage node If the storage node receives the first indication signal through the control thread, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread corresponding to at least one physical shard through the control thread, wherein a processing stage flag bit preset in the first notification signal is set to the commit processing stage.
- the first notification signal can be used to wake up the DDL thread that enters a sleep state during the preparation processing phase.
- the data dictionary lock is the metadata lock located in the storage engine layer of the MySQL logical system architecture (including the MySQL service layer and the storage engine layer).
- the MySQL logical system architecture can be referred to in related technologies and will not be described in detail in this implementation.
- Delete the target physical slice and set the target file as the new target physical slice that is, switch the old and new physical slices.
- the name of the target physical shard is used as the new target physical shard.
- the difference between the old target physical shard and the new target physical shard is that the new physical shard updates the data corresponding to the old target physical shard by replaying the log file that records the incremental data update information on the target file.
- the old target physical shard and the new target physical shard have different metadata.
- the storage node After setting the target file as the new target physical shard, the storage node sends a second notification signal to the control thread through the target DDL thread indicating that the third synchronization point has been reached.
- the third synchronization point corresponds to the commit processing phase.
- the storage node sends a second indication signal corresponding to the third synchronization point to the computing node through the control thread, and the second indication signal indicates that at least one DDL thread sends a second notification signal.
- the computing node receives a second indication signal corresponding to the third synchronization point, and determines the execution status of the DDL statement based on the second indication signal.
- the computing node receives the second indication signal sent by the control thread of the storage node, it indicates that the at least one DDL thread has completed the processing behavior corresponding to the submission processing stage of the DDL statement, that is, the DDL statement is executed, the metadata of at least one physical shard has been modified, the metadata of at least one physical shard in the storage node is consistent, and the logical metadata can be further modified.
- FIG4 is a schematic diagram of a DDL statement processing process provided by an embodiment of the present disclosure. As shown in FIG4, it is assumed that the metadata of physical shard 1 and physical shard 2 of the storage node need to be modified, wherein physical shard 1 corresponds to DDL thread 1, and physical shard 2 corresponds to DDL thread 2.
- the processing stage flag bit preset in the first notification signal is S0.
- the specific DDL statement processing process is as follows.
- the compute node sends a DDL statement to DDL thread 1, so that the storage node, in response to receiving the DDL statement through DDL thread 1, executes the initialization and data copying behaviors corresponding to the initial processing phase through DDL thread 1. That is, DDL thread 1 obtains the MDL read lock, copies the existing data in physical shard 1 to the target file, obtains the log file that records the incremental data update information generated since the copy of the existing data began, and replays the log file on the target file after the copy of the existing data is completed.
- DDL thread 1 When the amount of data remaining in the log file that has not been replayed does not exceed a set threshold, DDL thread 1 sends a second notification signal indicating that the first synchronization point has been reached to the control thread and the compute node respectively, and DDL thread 1 can enter a sleep state.
- the computing node After receiving the second notification signal sent by the storage node through DDL thread 1, the computing node sends a DDL statement to DDL thread 2, so that the storage node, in response to receiving the DDL statement through DDL thread 2, executes the initialization behavior and data copy behavior corresponding to the initial processing phase through DDL thread 2, namely, obtains the MDL read lock through DDL thread 2, copies the existing data in physical shard 2 to the target file, obtains the log file that records the incremental data update information generated since the copy of the existing data began, and replays the log file on the target file after the copy of the existing data is completed.
- the second notification signal indicating that the first synchronization point has been reached is sent to the control thread through DDL thread 2, and DDL thread 2 can enter a sleep state.
- the storage node sends a second indication signal corresponding to the first synchronization point to the computing node through the control thread, so that the computing node determines the execution status of the DDL statement based on the second indication signal.
- the second indication signal indicates that both DDL thread 1 and DDL thread 2 send the second notification signal.
- DDL thread 2 when DDL thread 2 is executing the initialization and data copying actions corresponding to the initial processing phase, if the amount of remaining unplayed data in the log file corresponding to physical shard 1 exceeds the set threshold, the target DDL thread, which has been in a dormant state, will be awakened, allowing DDL thread 1 to continue playing back the log file on the target file corresponding to physical shard 1.
- the computing node If the computing node receives the second indication signal corresponding to the first synchronization point sent by the control thread during the initial processing phase, it indicates that the above-mentioned DDL thread 1 and DDL thread 2 have completed the processing behavior corresponding to the initial processing phase of the DDL statement and can enter the preparation processing phase of the DDL statement.
- the storage node upgrades the MDL read lock to an MDL write lock through DDL thread 1, replays the remaining unplayed data in the log file in the target file corresponding to physical shard 1, and performs a data consistency check.
- a second notification signal indicating that the second synchronization point has been reached is sent to the control thread through DDL thread 1, and DDL thread 1 can enter a sleep state.
- the storage node upgrades the MDL read lock to an MDL write lock through DDL thread 2, replays the remaining unplayed data in the log file in the target file corresponding to physical shard 2, and performs a data consistency check. After the data consistency check is successful, a second notification signal indicating that the second synchronization point has been reached is sent to the control thread through DDL thread 2, and DDL thread 2 can enter a sleep state.
- the storage node sends a second indication signal corresponding to the second synchronization point to the computing node through the control thread, so that the computing node determines the execution status of the DDL statement based on the second indication signal.
- the second indication signal indicates that both DDL thread 1 and DDL thread 2 send the second notification signal.
- the computing node If the computing node receives the second indication signal corresponding to the first synchronization point sent by the control thread during the preparation processing phase, it indicates that both DDL thread 1 and DDL thread 2 have completed the processing behavior corresponding to the preparation processing phase of the DDL statement and can enter the submission processing phase of the DDL statement.
- the storage node acquires the data dictionary lock through DDL thread 1, deletes physical shard 1, sets the target file to the new physical shard 1, and sends a second notification signal indicating that the third synchronization point has been reached to the control thread through DDL thread 1.
- DDL thread 1 can enter a sleep state.
- DDL thread 2 Similar to DDL thread 1, after the storage node obtains the data dictionary lock through DDL thread 2, it deletes physical shard 2 and sets the target file to the new physical shard 2. It also sends a second notification signal indicating that the third synchronization point has been reached to the control thread through DDL thread 2, and DDL thread 2 can enter a sleep state.
- the storage node sends a second indication signal corresponding to the second synchronization point to the computing node through the control thread, so that the computing node determines the execution status of the DDL statement based on the second indication signal.
- the second indication signal indicates that both DDL thread 1 and DDL thread 2 send the second notification signal.
- the computing node If the computing node receives the second indication signal sent by the control thread of the storage node, it indicates that DDL thread 1 and DDL thread 2 have completed the processing behavior corresponding to the commit processing phase of the DDL statement, that is, the meta information of physical shard 1 and the meta information of physical shard 2 have been modified, and the logical meta information can be further modified.
- each processing stage corresponding to the DDL statement is provided with a corresponding synchronization point, so that the storage node can execute the corresponding processing behavior of each processing stage through at least one DDL thread.
- a second notification signal is sent to the control thread in the storage node, enabling the control thread to obtain the execution status of at least one DDL thread in the current processing phase. For example, at least one DDL thread has completed the processing behavior in the current processing phase and sent the second control signal.
- the control thread sends a second indication signal to the compute node, enabling the compute node to determine the execution status of at least one DDL thread in the current processing phase based on the second indication signal, thereby determining whether at least one DDL thread should continue to execute the processing behavior in the next processing phase or perform a unified rollback.
- This ensures that the execution of DDL statements by multiple DDL threads is synchronized in the processing phase, preventing the situation where the metadata of some physical shards has been modified while the metadata of others has not. This effectively ensures that during the process of modifying the metadata of at least one physical shard, the sibling information of each physical shard remains consistent, i.e., the metadata of each physical shard has either been modified or not.
- FIG5 is a schematic diagram of another DDL statement processing process provided by an embodiment of the present disclosure.
- DDL thread 1 can directly notify the computing node that the execution of the DDL statement has been completed, that is, the meta information modification of physical shard 1 has been completed;
- DDL thread 2 can also directly notify the computing node that the execution of the DDL statement has been completed, that is, the meta information modification of physical shard 2 has been completed.
- the computing node determines that the meta information of all physical shards has been modified, and can further instruct to modify the logical meta information.
- each storage node can send a second indication signal to the computing node through its control thread, so that the computing stage determines the execution status of the DDL statement by the DDL thread contained in each storage node according to the second indication signal, thereby ensuring the consistency of the metadata of the physical shards corresponding to the multiple storage nodes.
- the computing node can send a rollback notification to the storage node that is not down, so that the storage node that is not down can perform the rollback process through the DDL thread. This ensures that the meta information corresponding to the physical shards in the restarted storage node and the physical shards in the storage node that is not down are consistent, and are both the meta information before the DDL statement is executed.
- the non-crashed storage node can continue to execute the processing behavior of the commit processing phase through the DDL thread and feedback the processing results of the commit processing phase to the computing node after the execution is completed.
- the control thread can send a second indication signal corresponding to the third synchronization point to the computing node so that the computing node determines the execution status of the DDL statement based on the second indication signal, or directly notify the computing node through the control thread that the execution of the DDL statement has been completed.
- the computing node can resend the DDL statement to it after it is restarted so that it can execute the DDL statement through the DDL thread. Finally, after receiving the second indication signal corresponding to the third synchronization point or the notification of the completion of the execution of the DDL statement sent by the crashed storage node through the control thread, the computing node modifies the logical meta information.
- the computing node receives the second indication signal corresponding to the second synchronization point sent by the control thread of a storage node, and if it determines that the storage node is down, it resends the DDL statement to the storage node after the storage node is restarted; thereafter, if the storage node receives the DDL statement resent by the computing node, it executes the DDL statement based on the standard DDL statement execution process.
- the standard DDL statement execution process does not include the synchronization points provided by the embodiments of the present disclosure.
- the downed storage node receives the DDL statement resent by the computing node through the target DDL thread, it first obtains the MDL read lock through the target DDL thread and copies the existing data in the target physical shard through the target DDL thread.
- the initial processing phase is completed by obtaining the log file containing incremental data updates generated since the start of the existing data copy and replaying the log file on the target file after the existing data copy is complete.
- the target DDL thread is any one of the at least one DDL threads corresponding to at least one physical shard stored locally on the storage node.
- the storage node then upgrades the MDL read lock to an MDL write lock through the target DDL thread and, after replaying the remaining unplayed data in the log file, performs a data consistency check, completing the preparation phase. Finally, after obtaining the data dictionary lock through the target DDL thread, the storage node deletes the target physical shard and sets the target file to the new target physical shard, completing the commit phase.
- This scheme is a compensation strategy for achieving consistency of physical shard meta-information.
- the storage node that has been restarted after the crash will execute the DDL statement based on the standard DDL statement execution process after receiving the DDL statement resent by the computing node.
- the computing node does not determine the execution status of the DDL statement in each processing phase, thereby speeding up the execution efficiency of the DDL statement and completing the modification of the physical shard meta-information in the storage node that has been restarted after the crash as soon as possible, thereby ensuring the modification efficiency of the logical meta-information.
- FIG6 is a flowchart of a DDL statement processing method provided by an embodiment of the present disclosure. As shown in FIG6, the method may include the following steps:
- a storage node receives a first indication signal sent by a computing node through a control thread included therein for indicating entry into a target processing phase of a DDL statement, the storage node sends a first notification signal corresponding to the first indication signal to at least one DDL thread included in the storage node through the control thread, wherein the DDL statement is used to modify metadata of at least one physical shard stored locally, and the at least one DDL thread corresponds to the at least one physical shard.
- the storage node executes the processing behavior corresponding to the target processing stage through at least one DDL thread, and after executing the processing behavior, sends a second notification signal to the control thread indicating that the target synchronization point has been reached.
- the target synchronization point corresponds to the target processing stage.
- the storage node sends a second indication signal corresponding to the target synchronization point to the computing node through the control thread, so that the computing node determines the execution status of the DDL statement according to the second indication signal, and the second indication signal indicates that at least one DDL thread sends a second notification signal.
- This embodiment introduces the DDL statement processing process corresponding to the storage node in the distributed database system.
- the detailed execution process can be referred to the relevant description in the above embodiment and will not be repeated here.
- FIG7 is a flowchart of another DDL statement processing method provided in an embodiment of the present disclosure. As shown in FIG7 , the method may include the following steps.
- the computing node sends a first indication signal to the control thread included in the storage node for indicating entry into the target processing phase of the DDL statement, so that the control thread sends a first notification signal corresponding to the first indication signal to at least one DDL thread included in the storage node, instructing the at least one DDL thread to execute the processing behavior corresponding to the target processing phase, and after executing the processing behavior, sends a second notification signal to the control thread indicating that a target synchronization point has been reached, wherein the target synchronization point corresponds to the target processing phase, the DDL statement is used to modify the metadata of at least one physical shard stored locally, and the at least one DDL thread corresponds to the at least one physical shard.
- the computing node receives a second indication signal corresponding to the target synchronization point sent by the control thread, and the second indication signal indicates that at least one DDL thread sends a second notification signal.
- the computing node determines the execution status of the DDL statement according to the second indication signal.
- This embodiment introduces the DDL statement processing process corresponding to the computing node in the distributed database system.
- the detailed execution process can be referred to the relevant description in the above embodiment and will not be repeated here.
- FIG. 8 is a schematic diagram of the structure of a DDL statement processing device provided by an embodiment of the present disclosure.
- the DDL statement processing device is located in a storage node in a distributed database system.
- the storage node executes a control thread and at least one DDL thread corresponding to at least one physical shard of local storage.
- the device includes: a first receiving module 11, a first processing module 12, and a first sending module 13.
- a first receiving module 11 is configured to send a first notification signal corresponding to the first indication signal to each of the at least one DDL threads through the control thread if a first indication signal is received from the computing node through the control thread to indicate entering a target processing phase of a DDL statement, wherein the DDL statement is used to modify the metadata of the at least one physical shard.
- a first processing module 12 is configured to execute the processing behavior corresponding to the target processing stage through the at least one DDL thread, and after executing the processing behavior, send a second notification signal to the control thread indicating that a target synchronization point has been reached, where the target synchronization point corresponds to the target processing stage.
- the first sending module 13 is further used to send a second indication signal corresponding to the target synchronization point to the computing node through the control thread, so that the computing node determines the execution status of the DDL statement according to the second indication signal, and the second indication signal indicates that at least one DDL thread has issued the second notification signal.
- the target processing stage includes an initial processing stage; a processing stage flag bit preset in the first notification signal is set to the initial processing stage.
- the first processing module 12 is specifically configured to: in response to receiving the DDL statement sent by the computing node through a target DDL thread, execute the initialization behavior and data copy behavior corresponding to the initial processing stage through the target DDL thread, and after executing the data copy behavior, send a second notification signal to the control thread indicating that a first synchronization point has been reached, wherein the first synchronization point corresponds to the initial processing stage; wherein the target DDL thread is any one of the at least one DDL thread, and the computing node sequentially sends the DDL statement to the at least one DDL thread.
- the first processing module 12 is further specifically configured to: obtain an MDL read lock through the target DDL thread; copy the stock data in the target physical shard to the target file through the target DDL thread, obtain a log file recording incremental data update information generated after the stock data copying starts, and replay the log file on the target file after the stock data copying is completed; when the amount of data remaining in the log file that has not been replayed does not exceed a set threshold, send a second notification signal indicating that a first synchronization point has been reached to the control thread through the target DDL thread, the target physical shard corresponds to the target DDL thread,
- the first processing module 12 is further specifically configured to: in response to a notification event that the amount of remaining unplayed data in the log file exceeds the set threshold, continue to play back the log file on the target file through the target DDL thread, so that the amount of remaining unplayed data in the log file decreases to no more than the set threshold.
- the target processing phase includes a preparation processing phase; the processing phase flag bit preset in the first notification signal is set to the preparation processing phase.
- the first processing module 12 is further specifically configured to: upgrade the MDL read lock to an MDL write lock via the target DDL thread, perform a data consistency check after replaying the remaining unplayed data in the log file in the target file; and after the data consistency check succeeds, send a second notification signal to the control thread via the target DDL thread indicating that a second synchronization point has been reached, wherein the second synchronization point corresponds to the preparation processing phase.
- the first processing module 12 is further specifically used to: determine a check mark field preset in the target physical shard, the initial state value corresponding to the check mark field is a first value, and the first value indicates that the data corresponding to the check mark field in the target physical shard are not repeated; determine a target state value corresponding to the check mark field in the target file; if the target state value is the second value, determine that the data consistency check is unsuccessful; if the target state value is the first value, determine that the data consistency check is successful.
- the first receiving module 11 is further configured to execute the DDL statement based on a standard DDL statement execution process if the DDL statement resent by the computing node is received; wherein, after the computing node receives the second indication signal corresponding to the second synchronization point sent by the control thread, if it is determined that the storage node is down, the computing node resends the DDL statement to the storage node after the storage node is restarted.
- the target processing phase includes a commit processing phase; and a processing phase flag bit preset in the first notification signal is set to the commit processing phase.
- the first processing module 12 is further specifically configured to: after acquiring a data dictionary lock through the target DDL thread, delete the target physical shard and set the target file as a new target physical shard; and after setting the target file as the new target physical shard, send a second notification signal to the control thread through the target DDL thread indicating that a third synchronization point has been reached, wherein the third synchronization point corresponds to the commit processing phase.
- the first receiving module 11 is further used to: if a rollback instruction sent by the computing node is received through the at least one DDL thread, perform rollback processing, wherein the rollback instruction is issued when the computing node times out and does not receive the second indication signal or determines that the at least one DDL thread executes abnormally.
- the device shown in FIG8 can execute the steps executed by the storage node in the distributed database system in the aforementioned embodiment.
- the detailed execution process and technical effects are described in the aforementioned embodiment and will not be repeated here.
- the structure of the DDL statement processing device shown in FIG8 can be implemented as an electronic device, as shown in FIG9
- the electronic device may include: a first memory 21, a first processor 22, and a first communication interface 23.
- the first memory 21 stores executable code.
- the first processor 22 can at least implement the DDL statement processing method executed by the storage node in the distributed database system in the aforementioned embodiment.
- FIG10 is a schematic diagram of the structure of another DDL statement processing device provided in an embodiment of the present disclosure.
- the DDL statement processing device is located in a computing node in a distributed database system.
- the computing node communicates with a storage node in the distributed database system.
- the storage node executes a control thread and at least one DDL thread corresponding to at least one physical shard of local storage.
- the device includes: a second sending module 31, a second receiving module 32, and a second processing module 33.
- a second sending module 31 is configured to send a first indication signal to the control thread indicating entry into a target processing phase of a DDL statement, so that the control thread sends a first notification signal corresponding to the first indication signal to the at least one DDL thread, respectively, and instructs the at least one DDL thread to execute a processing behavior corresponding to the target processing phase, and after executing the processing behavior, sends a second notification signal to the control thread indicating arrival at a target synchronization point, wherein the target synchronization point corresponds to the target processing phase, and the DDL statement is used to modify the metadata of the at least one physical shard.
- a second receiving module 32 configured to receive a second indication signal sent by the control thread corresponding to the target synchronization point, wherein the second indication signal indicates that the at least one DDL thread has sent the second notification signal.
- a second processing module 33 configured to determine the execution status of the DDL statement according to the second indication signal.
- the device shown in FIG10 can execute the steps executed by the computing nodes in the distributed database system in the aforementioned embodiment.
- the detailed execution process and technical effects are described in the aforementioned embodiment and will not be repeated here.
- the structure of the data identification device shown in FIG10 can be implemented as an electronic device, as shown in FIG11 .
- the electronic device may include: a second memory 41, a second processor 42, and a second communication interface 43.
- the second memory 41 stores executable code.
- the second processor 42 can at least implement the DDL statement processing method executed by the computing node in the distributed database system in the above-mentioned embodiment.
- an embodiment of the present disclosure provides a non-transitory machine-readable storage medium, on which executable code is stored.
- the processor can at least implement the DDL statement processing method provided in the above embodiment.
- An embodiment of the present disclosure provides a computer program product, comprising: a computer program, which, when executed by a processor of an electronic device, enables the processor to execute the data processing method provided in the aforementioned embodiment.
- each embodiment can be implemented by adding a necessary general hardware platform, or of course, can be implemented by combining hardware and software.
- the above technical solution in essence, or the part that contributes to the existing technology, can be embodied in the form of a computer product.
- the present disclosure can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
提供一种 DDL 语句处理方法、系统、设备和存储介质,包括:计算节点向存储节点发送用于指示进入 DDL 语句的目标处理阶段的第一指示信号。存储节点通过控制线程分别向至少一个物理分片对应的至少一个 DDL 线程发送与第一指示信号对应的第一通知信号,以通过至少一个 DDL 线程执行目标处理阶段对应的处理行为,并在执行完处理行为后分别向控制线程发送第二通知信号;通过控制线程向计算节点发送指示至少一个 DDL 线程均发出第二通知信号的第二指示信号,计算节点根据第二指示信号确定 DDL 语句的执行状态。.
Description
DDL语句处理 技林 城
[01]本公开涉及计算机技术领域, 尤其涉及 DDL语句处理。 背景技术
[02] MySQL是一种关系型数据库管理系统。 在基于 MySQL Sharding架构的分布式数据库中, 可以 利用 Sharding (即分片 )技术, 将包含有分布式数据库完整数据的逻辑表进行数据切分, 以得到逻 辑表对应的多个物理分片, 其中, 多个物理分片存储于分布式数据库的存储节点。
[03]数据定义语言 ( Data Definition Language , 简称 DDL ) 是结构化查询语言 (Structured Query Language, 简称 SQL)的一种, 用于定义表、 列、 索引等数据库对象。 DDL语句的执行对象是逻辑 表, 但由于分布式数据库中的分片机制, DDL语句需要在逻辑表对应的不同物理分片上分别执行。 具体地, DDL语句用于修改物理分片的元信息, 比如: 增加或减少一列字段、 修改某列字段的字节 长度或修改某列字段的数据格式等 .
[04]实际应用中, 不同的物理分片在依次执行某一 DDL语句的过程中, 可能在某些物理分片上执行 成功, 但在另一些物理分片上没有执行成功, 从而导致多个物理分片对应的元信息不一致, 影响数 据库的使用 . 发明内容
[05]本公开实施例提供一种 DDL语句处理方法、 系统、 设备和存储介质, 用以实现在不同物理分片 上执行 DDL语句的过程中, 物理分片元信息的一致性。
[06]第一方面,本公开实施例提供一种 DDL语句处理方法,应用于分布式数据库系统中的存储节点, 所述存储节点执行控制线程和与本地存储的至少一个物理分片对应的至少一个 DDL线程,所述方法 包括:若通过所述控制线程接收到计算节点发送的用于指示进入 DDL语句的目标处理阶段的第一指 示信号,则通过所述控制线程分别向所述至少一个 DDL线程发送与所述第一指示信号对应的第一通 知信号, 所述 DDL语句用于修改所述至少一个物理分片的元信息; 分别通过所述至少一个 DDL线 程执行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送表示到 达目标同步点的第二通知信号, 所述目标同步点与所述目标处理阶段对应; 通过所述控制线程向所 述计算节点发送与所述目标同步点对应的第二指示信号, 以使所述计算节点根据所述第二指示信号 确定所述 DDL语句的执行状态, 所述第二指示信号指示了所述至少一个 DDL线程均发出所述第二 通知信号。
[07]第二方面,本公开实施例提供一种 DDL语句处理装置,应用于分布式数据库系统中的存储节点, 所述存储节点执行控制线程和与本地存储的至少一个物理分片对应的至少一个 DDL线程,所述装置 包括: 第一接收模块, 用于若通过所述控制线程接收到计算节点发送的用于指示进入 DDL语句的目 标处理阶段的第一指示信号,则通过所述控制线程分别向所述至少一个 DDL线程发送与所述第一指 示信号对应的第一通知信号, 所述 DDL语句用于修改所述至少一个物理分片的元信息; 第一处理模 块, 用于分别通过所述至少一个 DDL线程执行所述目标处理阶段对应的处理行为, 并在执行完所述 处理行为后向所述控制线程发送表示到达目标同步点的第二通知信号, 所述目标同步点与所述目标 处理阶段对应; 第一发送模块, 还用于通过所述控制线程向所述计算节点发送与所述目标同步点对 应的第二指示信号, 以使所述计算节点根据所述第二指示信号确定所述 DDL语句的执行状态, 所述 第二指示信号指示了所述至少一个 DDL线程均发出所述第二通知信号。
[08]第三方面, 本公开实施例提供一种电子设备, 包括: 存储器、 处理器、 通信接口; 其中, 所述 存储器上存储有可执行代码, 当所述可执行代码被所述处理器执行时, 使所述处理器至少可以实现 如第一方面所述的 DDL语句处理方法。
[09]第四方面, 本公开实施例提供了一种非暂时性机器可读存储介质, 所述非暂时性机器可读存储 介质上存储有可执行代码, 当所述可执行代码被电子设备的处理器执行时, 使所述处理器至少可以 实现如第一方面所述的 DDL语句处理方法。
[10]第五方面,本公开实施例提供一种 DDL语句处理方法,应用于分布式数据库系统中的计算节点, 所述计算节点与所述分布式数据库系统中的存储节点通信, 所述存储节点执行控制线程和与本地存 储的至少一个物理分片对应的至少一个 DDL线程, 所述方法包括: 向所述控制线程发送用于指示进 入 DDL语句的目标处理阶段的第一指示信号, 以使所述控制线程分别向所述至少一个 DDL线程发 送与所述第一指示信号对应的第一通知信号,令所述至少一个 DDL线程执行所述目标处理阶段对应 的处理行为,并在执行完所述处理行为后向所述控制线程发送表示到达目标同步点的第二通知信号, 所述目标同步点与所述目标处理阶段对应,所述 DDL语句用于修改所述至少一个物理分片的元信息; 接收所述控制线程发送与所述目标同步点对应的第二指示信号, 所述第二指示信号指示了所述至少 一个 DDL线程均发出所述第二通知信号; 根据所述第二指示信号确定所述 DDL语句的执行状态。
[11]第六方面,本公开实施例提供一种 DDL语句处理装置,应用于分布式数据库系统中的计算节点, 所述计算节点与所述分布式数据库系统中的存储节点通信, 所述存储节点中包含控制线程和与本地 存储的至少一个物理分片对应的至少一个 DDL线程, 所述装置包括: 第二发送模块, 用于向所述控 制线程发送用于指示进入 DDL语句的目标处理阶段的第一指示信号,以使所述控制线程分别向所述 至少一个 DDL线程发送与所述第一指示信号对应的第一通知信号, 令所述至少一个 DDL线程执行 所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送表示到达目标 同步点的第二通知信号, 所述目标同步点与所述 ■目标处理阶段对应, 所述 DDL语句用于修改所述至 少一个物理分片的元信息; 第二接收模块, 用于接收所述控制线程发送与所述目标同步点对应的第 二指示信号, 所述第二指示信号指示了所述至少一个 DDL线程均发出所述第二通知信号; 第二处理 模块, 用于根据所述第二指示信号确定所述 DDL语句的执行状态。
[12]第七方面, 本公开实施例提供一种电子设备, 包括: 存储器、 处理器、 通信接口; 其中, 所述 存储器上存储有可执行代码, 当所述可执行代码被所述处理器执行时, 使所述处理器至少可以实现 如第五方面所述的 DDL语句处理方法。
[13]第八方面, 本公开实施例提供了一种非暂时性机器可读存储介质, 所述非暂时性机器可读存储 介质上存储有可执行代码, 当所述可执行代码被电子设备的处理器执行时, 使所述处理器至少可以 实现如第五方面所述的 DDL语句处理方法。
[14]第九方面, 本公开实施例提供一种分布式数据库系统, 包括: 计算节点和存储节点, 其中, 所 述存储节点执行控制线程和与本地存储的至少一个物理分片对应的至少一个 DDL线程;所述计算节 点, 用于向所述控制线程发送用于指示进入 DDL语句的目标处理阶段的第一指示信号, 所述 DDL 语句用于修改所述至少一个物理分片的元信息; 所述控制线程, 用于分别向所述至少一个 DDL线程 发送与所述第一指示信号对应的第一通知信号; 所述至少一个 DDL线程, 用于基于所述第一通知信 号, 执行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送表示 到达目标同步点的第二通知信号, 所述目标同步点与所述目标处理阶段对应; 所述控制线程, 还用 于向所述计算节点发送与所述目标同步点对应的第二指示信号, 所述第二指示信号指示了所述至少 一个 DDL 线程均发出所述第二通知信号; 所述计算节点, 还用于根据所述第二指示信号确定所述 DDL语句的执行状态。
[15]本公开实施例提供的方案可以适用于关系型分布式数据库, 该分布式数据库包括但不限于通信 连接的计算节点和存储节点, 其中, 存储节点执行控制线程和与本地存储的至少一个物理分片对应 的至少一个 DDL线程。在通过 DDL语句修改至少一个物理分片的元信息的过程中,针对 DDL语句 对应任一个处理阶段(称为目标处理阶段) , 存储节点通过控制线程接收计算节点发送的用于指示 进入 DDL语句的目标处理阶段的第一指示信号, 并通过控制线程分别向至少一个 DDL线程发送与 第一指示信号对应的第一通知信号, 以通知至少一个 DDL线程执行目标处理阶段对应的处理行为。 存储节点通过至少一个 DDL线程在执行完目标处理阶段对应的处理行为后,向控制线程发送表示到 达与目标处理阶段对应的目标同步点的第二通知信号。 若至少一个 DDL线程均发出第二通知信号, 则存储节点通过控制线程向计算节点发送与目标同步点对应的第二指示信号, 以使计算节点根据第 二指示信号确定 DDL语句的执行状态。 也就是说, 计算节点若接收到第二指示信号, 则表明至少一 个 DDL线程执行完目标处理阶段对应的处理行为; 计算节点若未接收到第二指示信号, 则表明至少 一个 DDL线程中存在 DDL线程未执行完目标处理阶段对应的处理行为。 当目标处理阶段为 DDL
语句对应的最后一个处理阶段时, 计算节点若接收到第二指示信号, 则表明至少一个物理分片的元 信息均已修改完成, 即完成了物理分片元信息的一致修改。 当计算节点未接收到第二指示信号时,可 以通知至少一个 DDL线程进行例如回滚等处理, 即返回在执行 DDL语句之前的状态, 从而使得至 少一个物理分片的元信息仍是执行 DDL语句之前的状态, 保证不同物理分片的元信息的一致性。
[16]综上,本方案中,通过设置目标处理阶段对应的目标同步点,并使得控制线程在接收到全部 DDL 线程发送的表示到达目标同步点的第二通知信号之后, 再向计算节点发送第二指示信号的方式, 能 够同步 DDL语句在目标处理阶段的进度, 保证至少一个物理分片的元信息一致。 附图说明
[17]为了更清楚地说明本公开实施例中的技术方案, 下面将对实施例描述中所需要使用的附图作一 简单地介绍, 显而易见地, 下面描述中的附图是本公开的一些实施例, 对于本领域普通技术人员来 讲, 在不付出创造性劳动的前提下, 还可以根据这些附图获得其他的附图。
[18]图 1为本公开实施例提供的一种分布式数据库系统的示意图;
[19]图 2为本公开实施例提供的一种 DDL语句处理方法的交互流程图;
[20]图 3为本公开实施例提供的另一种 DDL语句处理方法的交互流程图;
[21]图 4为本公开实施例提供的一种 DDL语句处理过程的示意图;
[22]图 5为本公开实施例提供的另一种 DDL语句处理过程的示意图;
[23]图 6为本公开实施例提供的一种 DDL语句处理方法的流程图;
[24]图 7为本公开实施例提供的另一种 DDL语句处理方法的流程图;
[25]图 8为本公开实施例提供的一种 DDL语句处理装置的结构示意图;
[26]图 9为与图 8所示实施例提供的 DDL语句处理装置对应的电子设备的结构示意图;
[27]图 10为本公开实施例提供的另一种 DDL语句处理装置的结构示意图;
[28]图 11为与图 10所示实施例提供的 DDL语句处理装置对应的电子设备的结构示意图。 具体实施方式
[29]为使本公开实施例的目的、 技术方案和优点更加清楚, 下面将结合本公开实施例中的附图, 对 本公开实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例是本公开一部分实施例, 而不是全部的实施例 . 基于本公开中的实施例, 本领域普通技术人员在没有做出创造性劳动前提下 所获得的所有其他实施例, 都属于本公开保护的范围。
[30]需要说明的是, 本公开实施例中所涉及的用户信息 (包括但不限于用户设备信息、 用户个人信 息等)和数据(包括但不限于用于分析的数据、 存储的数据、 展示的数据等) , 均为经用户授权或 者经过各方充分授权的信息和数据, 并且相关数据的收集、 使用和处理需要遵守相关国家和地区的 相关法律法规和标准, 并提供有相应的操作入口, 供用户选择授权或者拒绝。
[31]另外, 下述各方法实施例中的步骤时序仅为一种举例, 而非严格限定.
[32]以下先对本公开实施例涉及的几个概念进行说明。
[33] MySQL是一种关系型数据库管理系统。
[34]物理分片,也可称为物理表,用于在存储节点本地存储数据库中的数据 °具体地,在基于 MySQL Sharding 架构的分布式数据库中, 可以利用 Sharding(即分片)技术, 将包含有分布式数据库完整 数据的逻辑表进行数据切分(比如: 垂直切分、 水平切分等) , 得到逻辑表对应的多个物理分片
(Physical Shards) . 多个物理分片存储于分布式数据库的存储节点, 所有物理分片中存储的数据, 共同代表逻辑表包含的数据库完整数据。
[35]逻辑元信息, 即逻辑表对应的元信息, 记录有逻辑表对应的多个物理分片的存储位置、 每个物 理分片所存储数据的数据格式和列索引信息等。
[36]物理分片的元信息, 用于记录该物理分片所存储数据的数据格式和列索引信息等。
[37] DD L语句, 是 SQL语句的一种, 用于定义表、 列、 索引等数据库对象。 本方案中, DDL语句 用于修改物理分片的元信息, 比如: 增加或减少一列字段、 修改某列字段的字节长度或修改某列字 段的数据格式等。
[38]数据操纵语言 ( Data Manipulation Language , 简称 DML) , 是 SQ L的一种, DML语句用于添 加、 删除、 更新和查询数据库中的数据。
[39]元数据锁 ( Mata Date Lock, 简称 MDL) , 包含有 MDL读锁和 MDL写锁, 用于保护表、 歹叭 索引等数据库对象的结构。 其中, MDL读锁之间不互斥; MDL写锁之间以及 MDL写锁与 MDL读 锁之间互斥, 即具有排他性。
[40]数据字典锁, 即 MySQL逻辑系统架构 (包括: MySQL服务层和存储引擎层) 中位于存储引擎 层的元数据锁 •
[41] DDL线程, 即存储节点上执行 DDL语句的线程 .
[42]控制线程,即存储节点上用于检测 DDL语句执行状态,以及向 DDL线程发送通知信号的线程, 该通知信号用于通知 DDL线程执行与 DDL语句相关联的处理行为 .
[43]实际应用中, DDL语句的操作(执行)对象是逻辑表,但由于分布式数据库中的分片机制, DDL 语句需要在逻辑表对应的不同物理分片上分别执行,并在所有的物理分片上均执行完成该 DDL语句 后, 更新逻辑表对应的元信息, 即逻辑元信息。
[44]然而,不同的物理分片在依次执行某一 DDL语句的过程中,可能在某些物理分片上执行成功(即 物理分片的元信息修改成功) , 但在另一些物理分片上没有执行成功 (即物理分片的元信息没有修 改成功),从而导致多个物理分片对应的元信息不一致, 并且, 由于部分物理分片上没有执行成功, 逻辑元信息也无法更新, 逻辑元信息与元信息修改成功的物理分片的元信息也会出现长时间的不一 致, 影响数据库的使用 .
[45]为解决上述至少一个技术问题, 本公开实施例提供了一种 DDL语句处理方案, 基于本公开实施 例提供的方案,可以同步多个物理分片的 DDL语句执行进度,实现多个物理分片的元信息的一致性, 有效解决在部分物理分片上执行 DDL语句失败导致的多个物理分片的元信息不一致、逻辑元信息与 物理分片的元信息长时间不一致的问题。
[46]本公开实施例提供的方案可以适用于关系型分布式数据库。 图 1 为本公开实施例提供的一种分 布式数据库系统的示意图, 如图 1所示, 该分布式数据库系统中包括计算节点和存储节点, 两者之 间通信连接。
[47]其中, 计算节点作为分布式数据库系统的入口, 能够用于接收客户端下发的 DDL语句, 并触发 存储节点在相应的物理分片上执行该 DDL语句。存储节点中执行有控制线程和与本地存储的至少一 个物理分片对应的至少一个 DDL线程, 比如: 图 1 中物理分片 1对应的 DDL线程 1 , 物理分片 2 对应的 DDL线程 2等。
[48]其中, 计算节点和存储节点的数量与分布式数据库的实际配置相关, 如图 1所示, 可以包含有 n 个计算节点 (比如: 计算节点 1 - 计算节点 2 计算节点 n)和 m个存储节点 (存储节点 1、 存 储节点 2 存储节点 m) , n和 m均为大于 1的整数, n与 m可以相等或不等。 实际应用中, 任意通信连接的计算节点和存储节点对应的 DDL语句处理过程相同。
[49]以下结合图 2示例性说明上述分布式数据库系统中存储节点和计算节点在本公开实施例提供的 DDL语句处理方案中的工作过程。
[50]图 2为本公开实施例提供的一种 DDL语句处理方法的交互流程图, 如图 2所示, 该方法可以包 括如下步骤。
[51] 201> 计算节点向存储节点的控制线程发送用于指示进入 DDL语句的目标处理阶段的第一指示 信号, DDL语句用于修改存储节点本地存储的至少一个物理分片的元信息。
[52] 202、 若存储节点通过控制线程接收到第一指示信号, 则通过控制线程分别向至少一个物理分片 对应的至少一个 DDL线程发送与第一指示信号对应的第一通知信号。
[53] 203、 存储节点分别通过至少一个 DDL线程执行目标处理阶段对应的处理行为。
[54] 204. 存储节点的至少一个 DDL线程在执行完处理行为后分别向控制线程发送表示到达目标同 步点的第二通知信号, 目标同步点与目标处理阶段对应。
[55] 205、 存储节点通过控制线程向计算节点发送与目标同步点对应的第二指示信号, 第二指示信号 指示了至少一个 DDL线程均发出第二通知信号。
[56] 206、 计算节点接收第二指示信号, 并根据第二指示信号确定 DDL语句的执行状态。
[57]实际应用中,针对客户端下发的任一条 DDL语句,其执行流程可以概括的划分为三个处理阶段, 依次分别为: 初始处理阶段、 准备处理阶段和提交处理阶段。 其中, 每个处理阶段需要完成不同的 处理行为, 比如: 初始处理阶段需要完成 MDL读锁的获取、 物理分片中的存量数据和增量数据的 拷贝; 准备处理阶段需要完成升级 MDL读锁为 MDL写锁、 进行数据一致性校验; 提交处理阶段需 要完成数据字典锁的获取、 新旧物理分片切换.
[58]在多个物理分片上执行 DDL语句的过程中, DDL语句的任一个处理阶段出现问题, 都会导致 DDL语句执行失败, 从而导致多个物理分片的元信息不一致。 本实施例通过在 DDL语句的上述三 个处理阶段引入相应的同步点并增加相应的处理逻辑的方式,同步多个物理分片的 DDL语句执行进 度, 保证多个物理分片元信息的一致性。
[59]以下先从处理阶段的维度,说明同步点的设置和多个物理分片的 DDL语句执行进度同步的实现 过程。 上述三个处理阶段对应的具体处理行为将在后续的实施例中进行展开说明。
[60]需要说明的是, 本实施例中, 为便于表述, 图 2所示实施例中将初始处理阶段、 准备处理阶段 和提交处理阶段统称为目标处理阶段, 即目标处理阶段实际上可以表示初始处理阶段、 准备处理阶 段和提交处理阶段中的任一个。
[61]可以理解的是, 当目标处理阶段表示 DDL语句的不同处理阶段时, 图 2所示实施例中与目标处 理阶段直接或间接对应的第一指示信号、 第二指示信号、 第一通知信号、 第二通知信号和目标同步 点均对应于不同的含义。
[62]以下将目标处理阶段代入具体的 DDL语句处理阶段, 对图 2所示实施例提供的 DDL语句处理 方法进行展开说明。
[63]首先, 客户端发送 DDL语句到计算节点, 其中, DDL语句用于修改存储节点本地存储的至少 一个物理分片的元信息, DDL语句的格式为“ alter table info”, 其中, info表示具体操作行为, 比如: alter table tl add column t2 int, 表示向物理分片 tl中添力口列名为 t2的列, 该列的数据类型为整型。
[64]计算节点接收到 DDL语句之后, 通知与计算节点通信连接的存储节点启动相应的控制线程, 并 告知存储节点需要执行 DDL语句的至少一个物理分片,以使存储节点为该至少一个物理分片启动对 应的至少一个 DDL线程。
[65]然后,计算节点向存储节点的控制线程发送用于指示进入 DDL语句的初始处理阶段的第一指示 信号 •
[66]存储节点若通过控制线程接收到计算节点发送的用于指示进入 DDL语句的初始处理阶段的第一 指示信号, 则通过控制线程分别向至少一个 DDL线程发送与该第一指示信号对应的第一通知信号。
[67]在一可选实施例中, 第一通知信号中预设有处理阶段标志位, 该处理阶段标志位可被设置为目 标处理阶段, 从而可用于通知 DDL线程 DDL语句当前对应的处理阶段为目标处理阶段。
[68]可选地, 可以预先配置第一通知信号中预设的处理阶段标志位被设置为不同字段时所表示的处 理阶段, 比如预先配置: 处理阶段标志位被设置为 “start” 时, 表示 DDL语句当前所处的处理阶段 为初始处理阶段; 处理阶段标志位被设置为 “prepare”时, 表示 DDL语句当前所处的处理阶段为准备 处理阶段; 处理阶段标志位被设置为 “commit”时,表示 DDL语句当前所处的处理阶段为提交处理阶 段。 从而, 控制线程在向 DDL线程发送第一通知信号时, 可以基于预先配置的信息, 将第一通知信 号中预设的处理阶段标志位设置为用于表示目标处理阶段的字段。
[69]当目标处理阶段为初始处理阶段时, 第一通知信号中预设的处理阶段标志位被设置为初始处理 阶段, 具体地, 可以将第一通知信号中预设的处理阶段标志位设置为 “start" 。
[70]存储节点中的至少一个 DDL线程在接收到控制线程发送的第一通知信号后, 至少一个 DDL线 程执行初始处理阶段对应的处理行为。 每个 DDL线程在执行完 DDL语句的初始处理阶段的处理行
为之后, 会向控制线程发送表示到达第一同步点的第二通知信号。 其中, 第一同步点与初始处理阶 段对应, 即第一同步点是针对初始处理阶段设置的同步点, 用于确认至少一个 DDL线程是否均完成 了初始处理阶段的处理行为。
[71]存储节点中的控制线程在接收到上述至少一个 DDL线程分别发送的第二通知信号后, 向计算节 点发送与第一同步点对应的第二指示信号。其中, 第二指示信号指示了上述至少一个 DDL线程均发 出第二通知信号, 即所有的 DDL线程均完成了 DDL语句的初始处理阶段的处理行为。
[72]可以理解的是,若至少一个 DDL线程中存在 DDL线程未完成 DDL语句的初始处理阶段的处理 行为, 则该 DDL线程不会向控制线程发送第二通知信号, 控制线程也不会向计算节点发送第二指示 信号。
[73]计算节点若接收到存储节点的控制线程发送的第二指示信号, 则表明上述至少一个 DDL线程均 已完成了 DDL语句的初始处理阶段对应的处理行为, 可以进入 DDL语句的准备处理阶段 .
[74]可见, 通过设置初始处理阶段对应的第一同步点, 并使得控制线程在接收到全部 DDL线程发送 的表示到达第一同步点的第二通知信号之后, 向计算节点发送第二指示信号的方式, 能够同步 DDL 语句在初始处理阶段的进度, 保证上述至少一个 DDL线程均完成 DDL语句的初始处理阶段的处理 行为之后, 再进入 DDL语句的下一处理阶段一一准备处理阶段。
[75]之后,计算节点向存储节点的控制线程发送用于指示进入 DDL语句的准备处理阶段的第一指示 信号。
[76]存储节点若通过控制线程接收到计算节点发送的用于指示进入 DDL语句的准备处理阶段的第一 指示信号, 则通过控制线程分别向至少一个 DDL线程发送与该第一指示信号对应的第一通知信号。
[77]其中, 当目标处理阶段为准备处理阶段时, 第一通知信号中预设的处理阶段标志位被设置为准 备处理阶段, 具体地, 可以将第一通知信号中预设的处理阶段标志位设置为 “prepare” 。
[78]存储节点中的至少一个 DDL线程在接收到控制线程发送的第一通知信号后, 至少一个 DDL线 程执行准备处理阶段对应的处理行为。 每个 DDL线程在执行完 DDL语句的准备处理阶段的处理行 为之后, 会向控制线程发送表示到达第二同步点的第二通知信号。 其中, 第二同步点与准备处理阶 役对应, 即第二同步点是针对准备处理阶役设置的同步点, 用于确认至少一个 DDL线程是否均完成 了准备处理阶段的处理行为。
[79]存储节点中的控制线程在接收到上述至少一个 DDL线程分别发送的第二通知信号后, 向计算节 点发送与第二同步点对应的第二指示信号。其中, 第二指示信号指示了上述至少一个 DDL线程均发 出第二通知信号, 即所有的 DDL线程均完成了 DDL语句的准备处理阶段的处理行为。
[80]可以理解的是,若至少一个 DDL线程中存在 DDL线程未完成 DDL语句的准备处理阶段的处理 行为, 则该 DDL线程不会向控制线程发送第二通知信号, 控制线程也不会像计算节点发送第二指示 信号。
[81]计算节点若接收到存储节点的控制线程发送的第二指示信号, 则表明上述至少一个 DDL线程均 已完成了 DDL语句的准备处理阶段对应的处理行为, 可以进入 DDL语句的提交处理阶段。
[82]可见, 通过设置准备处理阶段对应的第二同步点, 并使得控制线程在接收到全部 DDL线程发送 的表示到达第二同步点的第二通知信号之后, 向计算节点发送第二指示信号的方式, 能够同步 DDL 语句在准备处理阶段的进度, 保证上述至少一个 DDL线程均完成 DDL语句的准备处理阶段的处理 行为之后, 再进入 DDL语句的下一处理阶段 —提交处理阶段。
[83]最后,计算节点向存储节点的控制线程发送用于指示进入 DDL语句的提交处理阶段的第一指示 信号。
[84]存储节点若通过控制线程接收到计算节点发送的用于指示进入 DDL语句的提交处理阶段的第一 指示信号, 则通过控制线程分别向至少一个 DDL线程发送与该第一指示信号对应的第一通知信号。
[85]其中, 当目标处理阶段为提交处理阶段时, 第一通知信号中预设的处理阶段标志位被设置为提 交处理阶段, 具体地, 可以将第一通知信号中预设的处理阶段标志位设置为 “commit” 。
[86]存储节点中的至少一个 DDL线程在接收到控制线程发送的第一通知信号后, 至少一个 DDL线 程执行提交处理阶段对应的处理行为。 每个 DDL线程在执行完 DDL语句的提交处理阶段的处理行
为之后, 会向控制线程发送表示到达第三同步点的第二通知信号。 其中, 第三同步点与提交处理阶 段对应, 即第三同步点是针对提交处理阶段设置的同步点, 用于确认至少一个 DDL线程是否均完成 了提交处理阶段的处理行为。
[87]存储节点中的控制线程在接收到上述至少一个 DDL线程分别发送的第二通知信号后, 向计算节 点发送与第三同步点对应的第二指示信号。其中, 第二指示信号指示了上述至少一个 DDL线程均发 出第二通知信号, 即所有的 DDL线程均完成了 DDL语句的提交处理阶段的处理行为。
[88]可以理解的是,若至少一个 DDL线程中存在 DDL线程未完成 DDL语句的提交处理阶段的处理 行为, 则该 DDL线程不会向控制线程发送第二通知信号, 控制线程也不会像计算节点发送第二指示 信号。
[89]计算节点若接收到存储节点的控制线程发送的第二指示信号, 则表明上述至少一个 DDL线程均 已完成了 DDL语句的提交处理阶段对应的处理行为, 也即 DDL语句执行完成, 上述至少一个物理 分片的元信息均已修改完成, 此时, 至少一个物理分片的元信息一致, 可进一步修改逻辑元信息。
[90]可见, 通过设置提交处理阶段对应的第三同步点, 并使得控制线程在接收到全部 DDL线程发送 的表示到达第三同步点的第二通知信号之后, 向计算节点发送第二指示信号的方式, 能够同步 DDL 语句在提交处理阶段的进度, 保证上述至少一个 DDL线程均完成 DDL语句的提交处理阶段的处理 行为, 即多个物理分片的元信息均已修改完成。
[91]在上述 DDL语句处理过程中, 存储节点中的控制线程在初始处理阶段对应的第一同步点、 准备 处理阶段对应的第二同步点和提交处理阶段对应的第三同步点,均会确认至少一个 DDL线程是否均 已完成 DDL语句在相应的处理阶段的处理行为。当至少一个 DDL线程均已完成 DDL语句在相应的 处理阶段的处理行为时, 向计算节点发送第二指示信号; 当至少一个 DDL线程中存在未完成 DDL 语句在相应的处理阶段的处理行为的 DDL线程时, 不向计算节点发送第二指示信号。
[92]在一可选实施例中, 若计算节点在预设等待时间内未接收到控制线程发送的第二指示信号, 即 计算节点超时未收到第二指示信号,则确定 DDL语句执行失败,向存储节点中的上述至少一个 DDL 线程发送回滚指令。若存储节点通过至少一个 DDL线程接收到计算节点发送的回滚指令, 则执行回 滚处理,即删除所有已完成的 DDL语句对应的处理行为,返回在执行 DDL语句之前的状态。例如, 若存储节点通过至少一个 DDL线程在第一同步点接收到计算节点发送的回滚指令,则至少一个 DDL 线程均删除在 DDL语句的初始处理阶段所执行的处理行为; 若存储节点通过至少一个 DDL线程在 第二同步点接收到计算节点发送的回滚指令, 则至少一个 DDL线程均删除在 DDL语句的初始处理 阶段和准备处理阶段所执行的处理行为 .
[93]在另一可选实施例中, 当计算节点确定至少一个 DDL线程执行异常时, 也可向存储节点中的上 述 ■至少一个 DDL线程发送回滚指令, 以使存储节点通过至少一个 DDL线程接收到计算节点发送的 回潦指令后, 则执行回滚_处理。
[94]综上,在通过 DDL语句修改分布式数据库的存储节点本地存储的至少一个物理分片的元信息的 过程中, 针对 DDL语句对应初始处理阶段、 准备处理阶段和提交处理阶段中的任一个处理阶段(称 为目标处理阶段),存储节点通过控制线程接收计算节点发送的用于指示进入 DDL语句的目标处理 阶段的第一指示信号,并通过控制线程分别向至少一个 DDL线程发送与第一指示信号对应的第一通 知信号,以通知至少一个 DDL线程执行目标处理阶段对应的处理行为。存储节点通过至少一个 DDL 线程在执行完目标处理阶段对应的处理行为后, 向控制线程发送表示到达与目标处理阶段对应的目 标同步点的第二通知信号。 若至少一个 DDL线程均发出第二通知信号, 则存储节点通过控制线程向 计算节点发送与目标同步点对应的第二指示信号,以使计算节点根据第二指示信号确定 DDL语句的 执行状态。 也就是说, 计算节点若接收到第二指示信号, 则表明至少一个 DDL线程执行完目标处理 阶段对应的处理行为, 当目标处理阶段为 DDL语句对应的最后一个处理阶段时, 表明至少一个物理 分片的元信息均已修改完成, 即完成了物理分片元信息的一致修改. 计算节点若未接收到第二指示 信号, 则表明至少一个 DDL线程中存在 DDL线程未执行完目标处理阶段对应的处理行为, 此时可 以通知至少一个 DDL线程进行回滚处理, 即返回在执行 DDL语句之前的状态, 通过一致回滚的方 式, 使得至少一个物理分片的元信息仍是执行 DDL语句之前的状态, 保证了物理分片的元信息的一 致性。 另外, 在 DDL语句对应的不同处理阶段设置相应的同步点, 能够在 DDL语句在相应的处理
阶段不能正常执行时, 及时进行处理, 比如一致回滚, 从而提高至少一个物理分片的元信息的修改 效率。
[95]以上从处理阶段的维度, 对本公开实施例提供的 DDL语句处理方法进行了说明。 以下结合初始 处理阶段、 准备处理阶段和提交处理阶段三个处理阶段分别对应的具体处理行为, 对本公开实施例 提供的 DDL语句处理方法进行进一步的说明。
[96]图 3为本公开实施例提供的另一种 DDL语句处理方法的交互流程图, 如图 3所示, DDL语句 处理存在三个处理阶段, 分别为初始处理阶段、 准备处理阶段和提交处理阶段, 每个处理阶段计算 节点与存储节点进行相应的交互, 以完成相应的处理行为。
[97]以下按照处理阶段的先后顺序, 依次对每个处理阶段对应的处理过程进行展开说明.
[98]在初始处理阶段, 如图 3所示, DDL语句处理方法可以包括如下步骤。
[99] 301、 计算节点向存储节点的控制线程发送用于指示进入 DDL语句的初始处理阶段的第一指示 信号, DDL语句用于修改存储节点本地存储的至少一个物理分片的元信息。
[100] 302> 若存储节点通过控制线程接收到第一指示信号, 则通过控制线程分别向至少一个物理分 片对应的至少一个 DDL线程发送与第一指示信号对应的第一通知信号, 其中, 第一通知信号中预设 的处理阶段标志位被设置为初始处理阶段。
[101]其中, 步骤 301和步骤 302的具体过程, 可参考前述实施例, 本实施例中不再赘述。
[ 102] 303> 计算节点向目标线程发送 DDL语句, 目标 DDL线程为至少一个 DDL线程中任一个, 计 算节点顺序地将 DDL语句依次发送至至少一个 DDL线程。
[103]304,存储节点响应于通过目标 DDL线程接收到 DDL语句, 通过目标 DDL线程执行初始处理 阶段对应的初始化行为和数据拷贝行为。
[104] 305、 存储节点的目标 DDL线程在执行完数据拷贝行为后, 向控制线程发送表示到达第一同步 点的第二通知信号, 第 "-同步点与初始处理阶段对应。
[105]本实施例中, 概括来说, 初始处理阶段对应的处理行为包括: 初始化行为和数据拷贝行为。 计 算节点通过向存储节点的至少一个 DDL线程发送 DDL语句,使得至少一个 DDL线程分别执行初始 处理阶段对应的初始化行为和数据拷贝行为 .
[106]具体实施过程中,存储节点中计算节点将 DDL语句依次发送给至少一个 DDL线程。举例来说, 假设需要对存储节点中第一物理分片和第二物理分片这两个物理分片的元信息进行修改, 其中, 第 一物理分片对应于第一 DDL线程, 第二物理分片对应于第二 DDL线程。 此时, DDL线程的数量不 止一个, 计算节点先向第一 DDL线程和第二 DDL线程中的一个 DDL线程发送 DDL语句, 例如先 向第一 DDL线程发送 DDL语句, 以使存储节点响应于通过第一 DDL线程接收到 DDL语句, 通过 第一 DDL线程执行初始处理阶段对应的初始化行为和数据拷贝行为; 第一 DDL线程在在执行完数 据拷贝行后, 除了向控制线程发送表示到达第一同步点的第二通知信号之外, 还会向计算节点发送 表示到达第一同步点的第二通知信号;计算节点在接收到存储节点通过第一 DDL线程发送的第二通 知信号之后,向第二 DDL线程发送 DDL语句,以使存储节点响应于通过第二 DDL线程接收到 DDL 语句, 通过第二 DDL线程执行初始处理阶段对应的初始化行为和数据拷贝行为。
[107]可以理解的是, 当 DDL线程为多个时, 除最后一个 DDL线程外, 其他的 DDL线程均需要在 执行完数据拷贝行为后, 向计算节点发送表示当前 DDL线程到达第一同步点的第二通知信号, 以使 计算节点向与当前 DDL线程相邻的下一个 DDL线程发送 DDL语句。 可选地, 当存在多个 DDL线 程时, 计算节点可预先确定多个 DDL线程接收 DDL语句的顺序 .
[108]存储节点通过至少一个 DDL线程中的任一个 DDL线程(称为目标 DDL线程)接收到计算节 点发送的 DDL语句后, 具体地, 根据如下方式通过目标 DDL线程执行初始处理阶段对应的初始化 行为和数据拷贝行为: 通过目标 DDL线程获取到 MDL读锁; 通过目标 DDL线程将目标物理分片 中的存量数据拷贝到目标文件, 获取自存量数据开始拷贝后产生的记录增量数据更新信息的日志文 件, 以及在存量数据拷贝完成后在目标文件上回放日志文件; 在日志文件中剩余未被回放的数据量 不超过设定阈值时, 通过目标 DDL线程向控制线程发送表示到达第一同步点的第二通知信号。
[109]其中, 目标物理分片为至少一个物理分片中与目标 DDL线程对应的物理分片。
[110]为便于理解, 先对元数据锁 DML的获取机制进行说明 .
[111]元数据锁 DML包括: MDL读锁和 MDL写锁。 其中, MDL读锁之间不互斥, MDL写锁之间 以及 MDL写锁与 MDL读锁之间互斥, 即具有排他性。
[112]具体地, 针对同一个物理分片, 用于执行 DML语句的线程在执行的多个 DML语句时, 可以 同时通过该线程为每个 DML语句获取到 MDL读锁, 即可同时通过 DML语句对该物理分片中的数 据进行添加、 删除、 更新或查询操作。 而 DDL线程在执行多个 DDL语句时, 每次只能通过 DDL 线程为一个 DDL语句获取到 MDL写锁, 且当为一个 DDL语句获取到 MDL写锁后, 无论是用于执 行 DML语句的线程还是 DDL线程, 均无法再为其他的 DML语句或 DDL语句获取 MDL写锁和 MDL 读锁, 即同一时刻, 只允许通过一个 DDL语句对该物理分片的元信息进行修改, 且在执行该 DDL语句以修改物理分片的元信息的过程中, 不能对物理分片中的数据进行添加、 删除、 更新或查 询操作。
[113]基于上述 MDL读锁和 MDL写锁的获取机制,在初始化阶段,先通过目标 DDL线程获取 MDL 读锁。 若目标 DDL获取不到 MDL读锁, 则表明当前存在其他 DDL语句正在修改目标物理分片的 元信息, 需要继续等待直至获取到 MDL读锁, 或者在等待超时时确认 DDL语句执行失败; 若目标 DDL能够获取到 MDL读锁, 则表明当前不存在其他 DDL语句正在修改目标物理分片的元信息,可 以执行接收到的计算节点发送的 DDL语句 .
[114]当通过目标 DDL线程获取到 MDL读锁时,认为完成了初始化行为。之后,进行数据拷贝行为。 其中, 数据拷贝包括对目标物理分片中存量数据的拷贝, 以及对目标分片中增量数据的拷贝。
[115]在 MySQL提供的在线 DDL(即 online DDL)机制中, 当在目标物理分片上执行 DDL语句时, 会在目标物理分片中添加相应的标志位,以通过该标志位标识目标物理分片正在通过 DDL语句修改 元信息。 当对目标物理分片进行操作时, 若检查到该标志位, 则将对目标物理分片进行的操作作为 增量数据写入在线日志 (即 online log) , 比如: 在通过 DML语句对目标物理分片中的数据进行增 加、 删除等操作时, 若检查到该标志位, 则将通过 DML语句对目标物理分片执行的操作写入在线 日志, 这样就产生了增量数据。 之后, 可通过回放在线日志的方式拷贝增量数据。
[116]本实施例中, 通过目标 DDL线程将目标物理分片中的存量数据拷贝到目标文件, 并获取自存 量数据开始拷贝后产生的记录增量数据更新信息的日志文件(即在线日志) 。 从而, 可以在存量数 据拷贝完成后, 在目标文件上回放日志文件, 完成增量数据的拷贝。 其中, 目标文件可以理解为一 临时文件.
[117]可选地, 可以自定义一设定阈值, 在日志文件中剩余未被回放的数据量不超过设定阈值时, 通 过目标 DDL线程向控制线程发送表示到达第一同步点的第二通知信号。
[118]可选地, 在存储节点通过目标 DDL线程向控制线程发送表示到达第一同步点的第二通知信号 后, 目标 DDL线程进入睡眠状态。
[119]如上文所述, 在初始处理阶段, 计算节点依次发送 DDL语句给至少一个 DDL线程。 因此, 当 至少一个 DDL线程的数量为多个时, 多个 DDL线程在执行完初始处理阶段对应的处理行为后发送 第二通知信号的时间是不一样的。 本实施例中, 为了判断多个 DDL线程是否均能够完成初始处理阶 段对应的处理行为, 使已经发送了第二通知信号的 DDL线程进入睡眠状态, 从而等待其他的 DDL 线程执行初始处理阶段对应的处理行为的结果。
[120]可以理解的是, 当目标 DDL线程进入睡眠状态后, 其对应的记录增量数据更新信息的日志文 件中, 可能仍然有新的操作信息持续写入, 从而导致日志文件中剩余未被回放的数据量上升。
[121]在一可选实施例中, 响应于日志文件中的剩余未被回放的数据量升至超过设定阈值的通知事件, 通过目标 DDL线程继续在目标文件上回放日志文件,以使日志文件中剩余未被回放的数据量下降至 不超过设定阈值。 也即, 在日志文件中的剩余未被回放的数据量升至超过设定阈值时, 唤醒进入睡 眠状态的目标 DDL线程, 以使目标 DDL线程继续在目标文件上回放日志文件。
[122] 306, 存储节点通过控制线程向计算节点发送与第一同步点对应的第二指示信号, 第二指示信 号指示了至少一个 DDL线程均发出第二通知信号。
[ 123] 307、 计算节点接收与第一同步点对应的第二指示信号, 并根据第二指示信号确定 DDL语句的
执行状态。
[124]计算节点若接收到存储节点的控制线程发送的第二指示信号, 则表明上述至少一个 DDL线程 均已完成了 DDL语句的初始处理阶段对应的处理行为, 可以进入 DDL语句的准备处理阶段.
[125]在准备处理阶段, 如图 3所示, DDL语句处理方法可以包括如下步骤。
[126] 308> 计算节点向存储节点发送用于指示进入 DDL语句的准备处理阶段的第一指示信号。
[127] 309、 若存储节点通过控制线程接收到第一指示信号, 则通过控制线程分别向至少一个物理分 片对应的至少一个 DDL线程发送与第一指示信号对应的第一通知信号, 其中, 第一通知信号中预设 的处理阶段标志位被设置为准备处理阶段。
[128]其中, 第一通知信号可用于唤醒在初始处理阶段进入睡眠状态的 DDL线程 °
[129] 3010> 存储节点通过目标 DDL线程升级 MDL读锁为 MD L写锁, 在目标文件中回放日志文件 中剩余未被回放的数据量后, 进行数据一致性校验。
[130]其中, 数据一致性校验用于确保物理分片中的数据在通过 DDL语句修改物理分片的兄信息前 后, 具有一致性。 比如: 在修改物理分片的元信息之前, 物理分片的某列数据中各条数据被约束为 不能重复, 则在修改物理分片的元信息之后, 该列数据中各条数据中也应当不存在重复数据, 从而 保证数据的一致性。
[131]在一可选实施例中, 可以根据如下方式进行数据一致性校验: 首先, 确定目标物理分片中预设 的校验标志字段, 校验标志字段对应的初始状态值为第一数值, 其中, 第一数值表示目标物理分片 中核 ■验标志字段对应的各条数据不重复。之后,在目标文件中确定核验标志字段对应的目标状态值。 最后, 根据第一数值和目标状态值, 确定数据一致性校验是否成功。 若目标状态值为第二数值, 则 确定数据一致性校验不成功, 第二数值区别于第一数值; 若目标状态值为第一数值, 则确定数据一 致性校验成功。
[132]其中, 目标文件中校验标志字段对应的目标状态值, 会在目标文件上回放日志文件的过程中自 动更新。
[133] 3011> 存储节点在数据一致性校验成功后, 通过目标 DDL线程向控制线程发送表示到达第二 同步点的第二通知信号, 第二同步点与准备处理阶段对应 .
[134]与初始处理阶段类似地, 在准备处理阶段, 存储节点通过目标 DDL线程向控制线程发送表示 到达第二同步点的第二通知信号后, 目标 DDL线程进入睡眠状态。
[135] 3012、 存储节点通过控制线程向计算节点发送与第二同步点对应的第二指示信号, 第二指示信 号指示了至少一个 DDL线程均发出第二通知信号 •
[136] 3013、 计算节点接收与第二同步点对应的第二指示信号, 并根据第二指示信号确定 DDL语句 的执行状态。
[137]计算节点若接收到存储节点的控制线程发送的第二指示信号, 则表明上述至少一个 DDL线程 均已完成了 DDL语句的准备处理阶段对应的处理行为, 可以进入 DDL语句的提交处理阶段。
[138]在提交处理阶段, 如图 3所示, DDL语句处理方法可以包括如下步骤。
[139] 3014> 计算节点向存储节点发送用于指示进入 DDL语句的提交处理阶段的第一指示信号。
[140] 3015> 若存储节点通过控制线程接收到第一指示信号, 则通过控制线程分别向至少一个物理分 片对应的至少一个 DDL线程发送与第一指示信号对应的第一通知信号, 其中, 第一通知信号中预设 的处理阶段标志位被设置为提交处理阶段。
[141]其中, 第一通知信号可用于唤醒在准备处理阶段进入睡眠状态的 DDL线程。
[142] 3016、 存储节点通过目标 DDL线程获取数据字典锁后, 将目标物理分片删除, 并将目标文件 设置为新的目标物理分片 .
[143]其中, 数据字典锁, 即 MySQL逻辑系统架构(包括: MySQL服务层和存储引擎层)中位于存 储引擎层的元数据锁. MySQL逻辑系统架构可参考相关技术, 本实施中不作赘述.
[144]将目标物理分片删除并将目标文件设置为新的目标物理分片, 即进行新旧物理分片切换。 具体 地, 可以先将目标物理分片 (以下称为旧的目标物理分片)删除, 然后将目标文件命名为已删除的
目标物理分片的名字, 从而作为新的目标物理分片。 旧的目标物理分片与新的目标物理分片的区别 在于, 新的物理分片通过在目标文件上回放记录增量数据更新信息的日志文件的方式, 更新了旧的 目标物理分片对应的数据, 从而旧的目标物理分片与新的目标物理分片对应于不同的元信息。
[145] 3017. 存储节点在将目标文件设置为新的目标物理分片后, 通过目标 DDL线程向控制线程发 送表示到达第三同步点的第二通知信号, 第三同步点与提交处理阶段对应。
[146]另外, 在将目标文件设置为新的目标物理分片后, 还会释放 MDL写锁。
[147] 3018、 存储节点通过控制线程向计算节点发送与第三同步点对应的第二指示信号, 第二指示信 号指示了至少一个 DDL线程均发出第二通知信号。
[148] 3019> 计算节点接收与第三同步点对应的第二指示信号, 并根据第二指示信号确定 DDL语句 的执行状态。
[149]计算节点若接收到存储节点的控制线程发送的第二指示信号, 则表明上述至少一个 DDL线程 均已完成了 DDL语句的提交处理阶段对应的处理行为, 也即 DDL语句执行完成, 至少一个物理分 片的元信息均已修改完成,存储节点中至少一个物理分片的元信息一致,可进一步修改逻辑元信息。
[150]以上对本公开实施例提供的 DDL语句处理方法中, DDL语句的初始处理阶段、 准备处理阶段 和提交处理阶段, 以及每个处理阶段对应的处理行为进行了说明。
[151]为便于理解, 以下通过举例对本公开实施例提供的 DDL语句处理方法进行说明。
[152]图 4为本公开实施例提供的一种 DDL语句处理过程的示意图。 如图 4所示, 假设要对存储节 点的物理分片 1的元信息和物理分片 2的元信息进行修改, 其中, 物理分片 1对应于 DDL线程 1 , 物理分片 2对应于 DDL线程 2,计算节点在发送 DDL语句时先发送给 DDL线程 1 ,再发送给 DDL 线程 2; 第一通知信号中预设的处理阶段标志位为 S0, 当 SO=start时, 表示 DDL语句当前所处的处 理阶段为初始处理阶段, 当 SO=prepare 时, 表示 DDL语句当前所处的处理阶段为准备处理阶段; 当 S0=commit, 表示 DDL语句当前所处的处理阶段为提交处理阶段。 具体地的 DDL语句处理过程 如下。
[153] 1、 初始处理阶段 °
[154]计算节点接收到客户端发送的 DDL语句之后, 向存储节点的控制线程发送用于指示进入 DDL 语句的初始处理阶段的第一指示信号。存储节点若通过控制线程接收到该指示进入 DDL语句的初始 处理阶段的第一指示信号, 则通过控制线程分别向 DDL线程 1和 DDL线程 2发送与该第一指示信 号对应的第一通知信号, 且通知信号中 SO=start.
[155]之后, 计算节点向 DDL线程 1发送 DDL语句, 以使存储节点响应于通过 DDL线程 1接收到 DDL语句, 通过 DDL线程 1执行初始处理阶段对应的初始化行为和数据拷贝行为, 即通过 DDL线 程 1获取到 MDL读锁, 将物理分片 1 中的存量数据拷贝到目标文件, 获取自存量数据开始拷贝后 产生的记录增量数据更新信息的日志文件,以及在存量数据拷贝完成后在目标文件上回放日志文件。 并在日志文件中剩余未被回放的数据量不超过设定阈值时,通过 DDL线程 1向控制线程和计算节点 分别发送表示到达第一同步点的第二通知信号, DDL线程 1可进入睡眠状态。
[156]计算节点在接收到存储节点通过 DDL线程 1发送的第二通知信号后,向 DDL线程 2发送 DDL 语句, 以使存储节点响应于通过 DDL线程 2接收到 DDL语句,通过 DDL线程 2执行初始处理阶段 对应的初始化行为和数据拷贝行为, 即通过 DDL线程 2获取到 MDL读锁, 将物理分片 2中的存量 数据拷贝到目标文件, 获取自存量数据开始拷贝后产生的记录增量数据更新信息的日志文件, 以及 在存量数据拷贝完成后在目标文件上回放日志文件。 并在日志文件中剩余未被回放的数据量不超过 设定阈值时, 通过 DDL线程 2向控制线程发送表示到达第一同步点的第二通知信号, DDL线程 2 可进入睡眠状态。
[157]之后, 存储节点通过控制线程向计算节点发送与第一同步点对应的第二指示信号, 以使计算节 点根据第二指示信号确定 DDL语句的执行状态。 其中, 第二指示信号指示了 DDL线程 1和 DDL 线程 2均发出第二通知信号。
[158]另夕卜, 在初始处理阶段, 如图 4所示, 在接收到计算节点发送用于指示进入 DDL语句的准备 处理阶段的第一指示信号之前,针对 DDL线程 1和 DDL线程 2中的任一个 DDL线程,响应于日志
文件中的剩余未被回放的数据量升至超过设定阈值的通知事件, 均会继续在目标文件上回放日志文 件, 以使日志文件中剩余未被回放的数据量下降至不超过设定阈值, 比如: 存储节点在通过 DDL线 程 2执行初始处理阶段对应的初始化行为和数据拷贝行为的过程中, 若物理分片 1对应的日志文件 中剩余未被回放的数据量升至超过设定阈值时, 则唤醒进入睡眠状态的目标 DDL线程, 以使 DDL 线程 1继续在物理分片 1对应的目标文件上回放日志文件。
[159]2、 准备处理阶段。
[ 160]若计算节点在初始处理阶段接收到了控制线程发送的与第一同步点对应的第二指示信号, 则表 明上述 DDL线程 1和 DDL线程 2均已完成了 DDL语句的初始处理阶段对应的处理行为,可以进入 DDL语句的准备处理阶段。
[ 161 ]计算节点向存储节点的控制线程发送用于指示进入 DDL语句的准备处理阶段的第一指示信号 . 存储节点若通过控制线程接收到该指示进入 DDL语句的准备处理阶段的第一指示信号,则通过控制 线程分别向 DDL线程 1和 DDL线程 2发送与该第一指示信号对应的第一通知信号, 以唤醒处于睡 眠状态的 DDL线程 1和 DDL线程 2, 其中, 第一通知信号中 SO=prepare.
[162]之后, 存储节点通过 DDL线程 1升级 MDL读锁为 MD L写锁, 在物理分片 1对应的目标文件 中回放日志文件中剩余未被回放的数据量后, 并进彳亍数据一致性校验, 在数据一致性校验成功后, 通过 DDL线程 1向控制线程发送表示到达第二同步点的第二通知信号, DDL线程 1可进入睡眠状 态。
[163]与 DDL线程 1类似地,存储节点通过 DDL线程 2升级 MDL读锁为 MDL写锁, 在物理分片 2 对应的目标文件中回放日志文件中剩余未被回放的数据量后, 并进行数据一致性校验, 在数据一致 性校验成功后,通过 DDL线程 2向控制线程发送表示到达第二同步点的第二通知信号, DDL线程 2 可进入睡眠状态。
[164]之后, 存储节点通过控制线程向计算节点发送与第二同步点对应的第二指示信号, 以使计算节 点根据第二指示信号确定 DDL语句的执行状态。 其中, 第二指示信号指示了 DDL线程 1和 DDL 线程 2均发出第二通知信号。
[165]3、 提交处理阶段。
[ 166]若计算节点在准备处理阶段接收到了控制线程发送的与第一同步点对应的第二指示信号, 则表 明上述 DDL线程 1和 DDL线程 2均已完成了 DDL语句的准备处理阶段对应的处理行为,可以进入 DDL语句的提交处理阶段.
[ 167]计算节点向存储节点的控制线程发送用于指示进入 DDL语句的提交处理阶段的第一指示信号。 存储节点若通过控制线程接收到该指示进入 DDL语句的提交处理阶段的第一指示信号,则通过控制 线程分别向 DDL线程 1和 DDL线程 2发送与该第一指示信号对应的第一通知信号, 以唤醒处于睡 眠状态的 DDL线程 1和 DDL线程 2. 其中, 第一通知信号中 S0=commit.
[168]之后, 存储节点通过 DDL线程 1获取数据字典锁后, 将物理分片 1删除, 并将目标文件设置 为新的物理分片 1, 并通过 DDL线程 1向控制线程发送表示到达第三同步点的第二通知信号, DDL 线程 1可进入睡眠状态,
[169]与 DDL线程 1类似地, 存储节点通过 DDL线程 2获取数据字典锁后, 将物理分片 2删除, 并 将目标文件设置为新的物理分片 2, 并通过 DDL线程 2向控制线程发送表示到达第三同步点的第二 通知信号, DDL线程 2可进入睡眠状态。
[170]之后, 存储节点通过控制线程向计算节点发送与第二同步点对应的第二指示信号, 以使计算节 点根据第二指示信号确定 DDL语句的执行状态。 其中, 第二指示信号指示了 DDL线程 1和 DDL 线程 2均发出第二通知信号。
[171]计算节点若接收到存储节点的控制线程发送的第二指示信号, 则表明 DDL线程 1和 DDL线程 2均已完成了 DDL语句的提交处理阶段对应的处理行为, 也即物理分片 1的元信息和物理分片 2的 元信息均已修改完成, 可进一步修改逻辑元信息。
[172]在上述方案中, DDL语句对应的任一个处理阶段, 均设置有相应的同步点, 从而可以使得存储 节点通过至少一个 DDL线程在执行各处理阶段对应的处理行为的过程中,当到达于当前处理阶段对
应的同步点时, 向存储节点中的控制线程发送第二通知信号, 以使得控制线程能够获取到至少一个 DDL线程在当前处理阶段的执行情况, 比如: 至少一个 DDL线程均已完成当前处理阶段的处理行 为并发送第二控制信号。 进一步地, 控制线程根据接收到的第二通知信号, 向计算节点发送第二指 示信号, 以使得计算节点根据第二指示信号确定至少一个 DDL在当前处理阶段的执行状态, 从而确 定至少一个 DDL线程是要继续执行下一个处理阶段的处理行为, 还是统一进行回滚 ■处理, 从而保证 多个 DDL线程对 DDL语句的执行在处理阶段上是同步的, 不会存在有的物理分片的元信息已经修 改完成, 而有的物理分片的元信息还未修改完成的情况, 能够有效保证在修改至少一个物理分片的 元信息的过程中, 各物理分片的兄信息始终是一致的, 即各物理分片的元信息要么均进行了修改, 要么均未进行修改。
[173]实际应用中, 通常若 DDL语句的准备处理阶段的处理行为能够被完成, 则之后的提交处理阶 段的处理行为大概率也能够被顺利完成。
[174]因此, 在一可选实施例中, 可以选择不设置提交处理阶段对应的第三同步点。
[175]图 5为本公开实施例提供的另一种 DDL语句处理过程的示意图, 如图 5所示, 在提交处理阶 段, DDL线程 1在完成 DDL语句的提交处理阶段的处理行为之后,可直接通知计算节点已完成 DDL 语句的执行, 即已完成对物理分片 1的元信息修改; DDL线程 2在完成 DDL语句的提交处理阶段 的处理行为之后, 也可直接通知计算节点已完成 DDL语句的执行, 即已完成对物理分片 2的元信息 修改。 计算节点在接收到 DDL线程 1和 DDL线程 2发送的通知, 确定全部的物理分片的元信息均 已修改完成, 可进一步指示修改逻辑元信息。
[176]本实施例中, 通过减少提交处理阶段对应的第三同步点的方式, 能够简化计算节点和存储节点 执行 DDL语句处理时的操作步骤, 提高物理分片元信息的修改效率。
[177]可以理解的是, 分布式数据库中, 可能同时存在多个存储节点需要同时对本地存储的物理分片 的元信息进行修改, 这多个存储节点各自的控制线程均与同一个计算节点连接, 从而, 各存储节点 可以通过其包含控制线程向计算节点发送第二指示信号, 以使计算阶段根据第二指示信号确定各存 储节点中包含的 DDL线程对 DDL语句的执行状态, 保证多个存储节点对应的物理分片的元信息的 —致性,
[178]实际应用中, 可能会出现部分存储节点在进入 DDL语句的初始处理阶段、 准备处理阶段或提 交处理阶段之后发生宕机, 而其他的存储节点正常的情况。
[179]针对在初始处理阶段、 准备处理阶段发生宕机的情形, 可通过计算节点向未宕机的存储节点发 送回滚通知的方式,使得未宕机的存储节点通过 DDL线程进行回滚处理, 从而使得宕机重启的存储 节点中的物理分片和未宕机的存储节点中的物理分片所对应的元信息一致,均是执行 DDL语句前的 元信息 .
[180]针对在提交处理阶段发生宕机的情况, 由于无论是宕机还是未宕机的存储节点均已通过控制线 程接收到了计算节点发送的用于指示进入 DDL语句的提交处理阶段的第一指示信号, 因此, 未宕机 的存储节点可继续通过 DDL线程执行提交处理阶段的处理行为,并在执行完成后向计算节点反馈提 交处理阶段的处理结果。比如:可以通过控制线程向计算节点发送第三同步点对应的第二指示信号, 以使计算节点根据第二指示信号确定 DDL语句的执行状态, 或者, 直接通过控制线程通知计算节点 已完成 DDL语句的执行。而宕机的存储节点,计算节点可以在其重启后,重新向其发送 DDL语句, 以使其通过 DDL线程执行 DDL语句。 最后, 计算节点在接收到宕机的存储节点通过控制线程发送 的第三同步点对应的第二指示信号或已完成 DDL语句的执行的通知后, 修改逻辑元信息。
[ 181]作为一种可选的实施方式, 若计算节点在收到某一存储节点的控制线程发送的与第二同步点对 应的第二指示信号后, 若确定该存储节点宕机, 则在该存储节点重启后重新发送 DDL语句至该存储 节点; 之后, 该存储节点若接收到计算节点重新发送的 DDL语句, 则基于标准的 DDL语句执行流 程执行 DDL语句。
[182]其中, 标准的 DDL语句执行流程中, 不包含有本公开实施例提供的同步点。
[183]具体地, 宕机的存储节点通过目标 DDL线程接收到计算节点重新发送的 DDL语句后, 首先, 先通过目标 DDL线程获取到 MDL读锁, 并通过目标 DDL线程将目标物理分片中的存量数据拷贝
到目标文件, 获取自存量数据开始拷贝后产生的记录增量数据更新信息的日志文件, 以及在存量数 据拷贝完成后在目标文件上回放日志文件, 即完成初始处理阶段对应的处理行为。 其中, 目标 DDL 线程为存储节点本地存储的至少一个物理分片对应的至少一个 DDL线程中的任一个。 然后,存储节 点通过目标 DDL线程升级 MDL读锁为 MDL写锁, 并在目标文件中回放日志文件中剩余未被回放 的数据量后, 进行数据一致性校验, 即完成准备处理阶段对应的处理行为。 最后, 存储节点通过目 标 DDL线程获取数据字典锁后, 将目标物理分片删除, 并将目标文件设置为新的目标物理分片, 即 完成提交处理阶段对应的处理行为。
[184]本方案是一种实现物理分片元信息一致性的补偿策略, 为了保证物理分片的元信息尽可能一致, 针对在提交处理阶段发生宕机的情况,宕机重启的存储节点在接收到计算节点重新发送的 DDL语句 后,基于标准的 DDL语句执行流程执行 DDL语句,计算节点未在每个处理阶段对 DDL语句的执行 状态进行确定,从而可以加快 DDL语句的执行效率,尽快完成宕机重启的存储节点中物理分片元信 息的修改, 保证逻辑元信息的修改效率。
[185]图 6为本公开实施例提供的一种 DDL语句处理方法的流程图, 如图 6所示, 可以包括如下步 骤 •
[186] 601、存储节点若通过包含的控制线程接收到计算节点发送的用于指示进入 DDL语句的目标处 理阶段的第一指示信号,则通过控制线程分别向存储节点包含的至少一个 DDL线程发送与第一指示 信号对应的第一通知信号, DDL语句用于修改本地存储的至少一个物理分片的元信息, 至少一个 DDL线程与至少一个物理分片对应。
[ 187] 602> 存储节点分别通过至少一个 DDL线程执行目标处理阶段对应的处理行为, 并在执行完处 理行为后向控制线程发送表示到达目标同步点的第二通知信号, 目标同步点与目标处理阶段对应。
[ 188] 603> 存储节点通过控制线程向计算节点发送与目标同步点对应的第二指示信号, 以使计算节 点根据第二指示信号确定 DDL语句的执行状态, 第二指示信号指示了至少一个 DDL线程均发出第 二通知信号。
[189]本实施例介绍的是分布式数据库系统中的存储节点对应的 DDL语句处理过程, 其详细的执行 过程可以参考前述实施例中的相关说明, 在此不赘述。
[190]图 7为本公开实施例提供的另一种 DDL语句处理方法的流程图, 如图 7所示, 可以包括如下 步骤。
[191]701> 计算节点向存储节点包含的控制线程发送用于指示进入 DDL语句的目标处理阶段的第一 指示信号,以使控制线程分别向存储节点包含的至少一个 DDL线程发送与第一指示信号对应的第一 通知信号, 令至少一个 DDL线程执行目标处理阶段对应的处理行为, 并在执行完处理行为后向控制 线程发送表示到达目标同步点的第二通知信号, 目标同步点与目标处理阶段对应, DDL语句用于修 改本地存储的至少一个物理分片的元信息, 至少一个 DDL线程与至少一个物理分片对应。
[192] 702. 计算节点接收控制线程发送与目标同步点对应的第二指示信号, 第二指示信号指示了至 少一个 DDL线程均发出第二通知信号。
[ 193] 703、 计算节点根据第二指示信号确定 DDL语句的执行状态 .
[194]本实施例介绍的是分布式数据库系统中的计算节点对应的 DDL语句处理过程, 其详细的执行 过程可以参考前述实施例中的相关说明, 在此不赘述。
[195]以下将详细描述本公开的一个或多个实施例的 DDL语句处理装置。本领域技术人员可以理解, 这些装置均可使用市售的硬件组件通过本方案所教导的步骤进行配置来构成。
[196]图 8为本公开实施例提供的一种 DDL语句处理装置的结构示意图, 该 DDL语句处理装置位于 分布式数据库系统中的存储节点, 其中, 存储节点中执行控制线程和与本地存储的至少一个物理分 片对应的至少一个 DDL线程。 如图 8所示, 该装置包括: 第一接收模块 11、 第一处理模块 12、 第 一发送模块 13.
[197]第一接收模块 11 , 用于若通过所述控制线程接收到计算节点发送的用于指示进入 DDL语句的 目标处理阶段的第一指示信号,则通过所述控制线程分别向所述至少一个 DDL线程发送与所述第一 指示信号对应的第一通知信号, 所述 DDL语句用于修改所述至少一个物理分片的元信息。
[198]第一处理模块 12, 用于分别通过所述至少一个 DDL线程执行所述目标处理阶段对应的处理行 为, 并在执行完所述处理行为后向所述控制线程发送表示到达目标同步点的第二通知信号, 所述目 标同步点与所述目标处理阶段对应 .
[199]第一发送模块 13 ,还用于通过所述控制线程向所述计算节点发送与所述目标同步点对应的第二 指示信号, 以使所述计算节点根据所述第二指示信号确定所述 DDL语句的执行状态, 所述第二指示 信号指示了所述至少一个 DDL线程均发出所述第二通知信号。
[200]可选地, 所述目标处理阶段包括初始处理阶段; 在所述第一通知信号中预设的处理阶段标志位 被设置为所述初始处理阶段。 所述第一处理模块 12具体用于: 响应于通过目标 DDL线程接收到所 述计算节点发送的所述 DDL语句, 通过所述目标 DDL线程执行所述初始处理阶段对应的初始化行 为和数据拷贝行为, 在执行完所述数据拷贝行为后向所述控制线程发送表示到达第一同步点的第二 通知信号, 所述第一同步点与所述初始处理阶段对应; 其中, 所述目标 DDL 线程为所述至少一个 DDL线程中任一个, 所述计算节点顺序地将所述 DDL语句依次发送至所述至少一个 DDL线程。
[201]可选地, 所述第一处理模块 12还具体用于: 通过所述目标 DDL线程获取到 MDL读锁; 通过 所述目标 DDL线程将目标物理分片中的存量数据拷贝到目标文件,获取自所述存量数据开始拷贝后 产生的记录增量数据更新信息的日志文件, 以及在所述存量数据拷贝完成后在所述目标文件上回放 所述日志文件; 在所述日志文件中剩余未被回放的数据量不超过设定阈值时, 通过所述目标 DDL线 程向所述控制线程发送表示到达第一同步点的第二通知信号,所述目标物理分片与所述目标 DDL线 程对应,
[202]可选地, 所述第一处理模块 12还具体用于: 响应于所述日志文件中的剩余未被回放的数据量 升至超过所述设定阈值的通知事件,通过所述目标 DDL线程继续在所述目标文件上回放所述日志文 件, 以使所述日志文件中剩余未被回放的数据量下降至不超过所述设定阈值。
[203]可选地, 所述目标处理阶段包括准备处理阶段; 在所述第一通知信号中预设的处理阶段标志位 被设置为所述准备处理阶段。 所述第一处理模块 12还具体用于: 通过所述目标 DDL线程升级所述 MDL 读锁为 MDL写锁, 在所述目标文件中回放所述日志文件中剩余未被回放的数据量后, 进行数 据一致性校验; 在所述数据一致性校验成功后,通过所述目标 DDL线程向所述控制线程发送表示到 达第二同步点的第二通知信号, 所述第二同步点与所述准备处理阶段对应 .
[204]可选地, 所述第一处理模块 12还具体用于: 确定所述目标物理分片中预设的校验标志字段, 所述校验标志字段对应的初始状态值为第一数值, 所述第一数值表示所述目标物理分片中所述校验 标志字段对应的各条数据不重复; 在所述目标文件中确定所述校验标志字段对应的目标状态值; 若 所述目标状态值为第二数值, 则确定数据一致性校验不成功; 若所述目标状态值为所述第一数值, 则确定数据一致性校验成功。
[205]可选地, 所述第一接收模块 11 , 还用于若接收到所述计算节点重新发送的所述 DDL语句, 则 基于标准的 DDL语句执行流程执行所述 DDL语句; 其中, 所述计算节点在收到所述控制线程发送 的与所述第二同步点对应的第二指示信号后, 若确定所述存储节点宕机, 则在所述存储节点重启后 重新发送所述 DDL语句至所述存储节点。
[206]可选地, 所述目标处理阶段包括提交处理阶段; 在所述第一通知信号中预设的处理阶段标志位 被设置为所述提交处理阶段。 所述第一处理模块 12还具体用于: 通过所述目标 DDL线程获取数据 字典锁后, 将所述目标物理分片删除, 并将所述目标文件设置为新的目标物理分片; 在将所述目标 文件设置为新的目标物理分片后,通过所述目标 DDL线程向所述控制线程发送表示到达第三同步点 的第二通知信号, 所述第三同步点与所述提交处理阶段对应。
[207]可选地, 所述第一接收模块 11还用于: 若通过所述至少一个 DDL线程接收到所述计算节点发 送的回滚指令, 则执行回滚处理, 其中, 所述回滚指令是所述计算节点超时未收到所述第二指示信 号或确定所述至少一个 DDL线程执行异常时发出的。
[208]图 8所示装置可以执行前述实施例中分布式数据库系统中存储节点所执行的步骤, 详细的执行 过程和技术效果参见前述实施例中的描述, 在此不再赘述。
[209]在一个可能的设计中, 上述图 8所示 DDL语句处理装置的结构可实现为一电子设备, 如图 9
所示, 该电子设备可以包括: 第一存储器 21、 第一处理器 22、 第一通信接口 23。 其中, 第一存储 器 21上存储有可执行代码, 当所述可执行代码被第一处理器 22执行时, 使第一处理器 22至少可以 实现如前述实施例中分布式数据库系统中存储节点所执行的 DDL语句处理方法。
[210]图 10为本公开实施例提供的另一种 DDL语句处理装置的结构示意图, 该 DDL语句处理装置 位于分布式数据库系统中的计算节点, 其中, 计算节点与分布式数据库系统中的存储节点通信, 存 储节点中执行控制线程和与本地存储的至少一个物理分片对应的至少一个 DDL线程。如图 10所示, 该装置包括: 第二发送模块 31、 第二接收模块 32、 第二处理模块 33。
[211]第二发送模块 31 , 用于向所述控制线程发送用于指示进入 DDL语句的目标处理阶段的第一指 示信号,以使所述控制线程分别向所述至少一个 DDL线程发送与所述第一指示信号对应的第一通知 信号,令所述至少一个 DDL线程执行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为 后向所述控制线程发送表示到达目标同步点的第二通知信号, 所述目标同步点与所述目标处理阶段 对应, 所述 DDL语句用于修改所述至少一个物理分片的元信息。
[212]第二接收模块 32, 用于接收所述控制线程发送与所述目标同步点对应的第二指示信号, 所述第 二指示信号指示了所述至少一个 DDL线程均发出所述第二通知信号。
[213]第二处理模块 33, 用于根据所述第二指示信号确定所述 DDL语句的执行状态。
[214]图 10 所示装置可以执行前述实施例中分布式数据库系统中计算节点所执行的步骤, 详细的执 行过程和技术效果参见前述实施例中的描述, 在此不再赘述。
[215]在一个可能的设计中,上述图 10所示数据识别装置的结构可实现为一电子设备,如图 11所示, 该电子设备可以包括: 第二存储器 41、 第二处理器 42、 第二通信接口 43。 其中, 第二存储器 41上 存储有可执行代码, 当所述可执行代码被第二处理器 42执行时, 使第二处理器 42至少可以实现如 前述实施例中分布式数据库系统中计算节点所执行的 DDL语句处理方法。
[216]另外, 本公开实施例提供了一种非暂时性机器可读存储介质, 所述非暂时性机器可读存储介质 上存储有可执行代码, 当所述可执行代码被电子设备的处理器执行时, 使所述处理器至少可以实现 如前述实施例中提供的 DDL语句处理方法,
[217]本公开实施例提供了一种计算机程序产品, 包括: 计算机程序, 当所述计算机程序被电子设备 的处理器执行时, 使所述处理器执行如前述实施例中提供的数据处理方法。
[218]以上所描述的装置实施例仅仅是示意性的, 其中所述作为分离部件说明的单元可以是或者也可 以不是物理上分开的。 可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目 的。 本领域普通技术人员在不付出创造性的劳动的情况下, 即可以理解并实施。
[219]通过以上的实施方式的描述, 本领域的技术人员可以清楚地了解到各实施方式可借助加必需的 通用硬件平台的方式来实现, 当然也可以通过硬件和软件结合的方式来实现。 基于这样的理解, 上 述技术方案本质上或者说对现有技术做出贡献的部分可以以计算机产品的形式体现出来, 本公开可 采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质 (包括但不限于磁盘存储 器、 CD-ROM、 光学存储器等)上实施的计算机程序产品的形式。
[220]最后应说明的是: 以上实施例仅用以说明本公开的技术方案, 而非对其限制; 尽管参照前述实 施例对本公开进行了详细的说明, 本领域的普通技术人员应当理解: 其依然可以对前述各实施例所 记载的技术方案进行修改, 或者对其中部分技术特征进行等同替换; 而这些修改或者替换, 并不使 相应技术方案的本质脱离本公开各实施例技术方案的精神和范围。
Claims
1、 一种 DDL语句处理方法, 应用于分布式数据库系统中的存储节点, 所述存储节点执行控制 线程和与本地存储的至少一个物理分片对应的至少一个 DDL线程, 所述方法包括: 若通过所述控制线程接收到计算节点发送的用于指示进入 DDL 语句的目标处理阶段的第一指 示信号,则通过所述控制线程分别向所述至少一个 DDL线程发送与所述第一指示信号对应的第一通 知信号, 所述 DDL语句用于修改所述至少一个物理分片的元信息; 分别通过所述至少一个 DDL线程执行所述目标处理阶段对应的处理行为,并在执行完所述处理 行为后向所述控制线程发送表示到达目标同步点的第二通知信号, 所述目标同步点与所述目标处理 阶段对应; 通过所述控制线程向所述计算节点发送与所述目标同步点对应的第二指示信号, 以使所述计算 节点根据所述第二指示信号确定所述 DDL语句的执行状态,所述第二指示信号指示了所述至少一个 DDL线程均发出所述第二通知信号。
2、 根据权利要求 1所述的方法, 其中, 所述目标处理阶段包括初始处理阶段; 在所述第一通知 信号中预设的处理阶段标志位被设置为所述初始处理阶段;所述分别通过所述至少一个 DDL线程执 行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送表示到达目 标同步点的第二通知信号, 包括: 响应于通过目标 DDL线程接收到所述计算节点发送的所述 DDL语句,通过所述目标 DDL线程 执行所述初始处理阶段对应的初始化行为和数据拷贝行为, 在执行完所述数据拷贝行为后向所述控 制线程发送表示到达第一同步点的第二通知信号, 所述第一同步点与所述初始处理阶段对应; 其中,所述目标 DDL线程为所述至少一个 DDL线程中任一个,所述计算节点顺序地将所述 DDL 语句依次发送至所述至少一个 DDL线程。
3、 根据权利要求 2所述的方法, 其中, 所述通过所述目标 DDL线程执行所述初始处理阶段对 应的初始化行为和数据拷贝行为, 在执行完所述数据拷贝行为后向所述控制线程发送表示到达第一 同步点的第二通知信号, 包括: 通过所述目标 DDL线程获取到 MDL读锁; 通过所述目标 DDL线程将目标物理分片中的存量数据拷贝到目标文件,获取自所述存量数据开 始拷贝后产生的记录增量数据更新信息的日志文件, 以及在所述存量数据拷贝完成后在所述目标文 件上回放所述 .日志文件; 在所述日志文件中剩余未被回放的数据量不超过设定阈值时,通过所述目标 DDL线程向所述控 制线程发送表示到达第一同步点的第二通知信号, 所述目标物理分片与所述目标 DDL线程对应 .
4、 根据权利要求 3所述的方法, 其中, 所述目标 DDL线程向所述控制线程发送表示到达第一 同步点的第二通知信号后, 所述目标 DDL线程进入睡眠状态; 所述方法还包括: 响应于所述日志文件中的剩余未被回放的数据量升至超过所述设定阈值的通知事件, 通过所述 目标 DDL线程继续在所述目标文件上回放所述日志文件,以使所述日志文件中剩余未被回放的数据 量下降至不超过所述设定阈值。
5、 根据权利要求 3所述的方法, 其中, 所述目标处理阶段包括准备处理阶段; 在所述第一通知 信号中预设的处理阶段标志位被设置为所述准备处理阶段;所述分别通过所述至少一个 DDL线程执 行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送表示到达目 标同步点的第二通知信号, 包括: 通过所述目标 DDL线程升级所述 MDL读锁为 MDL写锁, 在所述目标文件中回放所述日志文 件中剩余未被回放的数据量后, 进行数据一致性校验; 在所述数据一致性校验成功后,通过所述目标 DDL线程向所述控制线程发送表示到达第二同步 点的第二通知信号, 所述第二同步点与所述准备处理阶段对应。
6、 根据权利要求 5所述的方法, 其中, 所述进行数据一致性校验, 包括: 确定所述目标物理分片中预设的校验标志字段, 所述校验标志字段对应的初始状态值为第一数 值, 所述第一数值表示所述目标物理分片中所述校验标志字段对应的各条数据不重复; 在所述目标文件中确定所述校验标志字段对应的目标状态值; 若所述目标状态值为第二数值, 则确定数据一致性校验不成功;
若所述目标状态值为所述第一数值, 则确定数据一致性校验成功。
7、 根据权利要求 5所述的方法, 还包括: 若接收到所述计算节点重新发送的所述 DDL语句, 则基于标准的 DDL语句执行流程执行所述 DDL语句; 其中, 所述计算节点在收到所述控制线程发送的与所述第二同步点对应的第二指示信号后, 若 确定所述存储节点宕机, 则在所述存储节点重启后重新发送所述 DDL语句至所述存储节点。
8、 根据权利要求 5所述的方法, 其中, 所述目标处理阶段包括提交处理阶段; 在所述第一通知 信号中预设的处理阶段标志位被设置为所述提交处理阶段;所述分别通过所述至少一个 DDL线程执 行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送表示到达目 标同步点的第二通知信号, 包括: 通过所述目标 DDL线程获取数据字典锁后, 将所述目标物理分片删除, 并将所述目标文件设置 为新的目标物理分片; 在将所述目标文件设置为新的目标物理分片后,通过所述目标 DDL线程向所述控制线程发送表 示到达第三同步点的第二通知信号, 所述第三同步点与所述提交处理阶段对应 .
9、 根据权利要求 1-8中任一项所述的方法, 还包括: 若通过所述至少一个 DDL线程接收到所述计算节点发送的回滚指令, 则执行回滚处理, 其中, 所述回滚指令是所述计算节点超时未收到所述第二指示信号或确定所述至少一个 DDL 线程执行异 常时发出的。
10、 一种 DDL语句处理方法, 应用于分布式数据库系统中的计算节点, 所述计算节点与所述分 布式数据库系统中的存储节点通信, 所述存储节点执行控制线程和与本地存储的至少一个物理分片 对应的至少一个 DDL线程, 所述方法包括: 向所述控制线程发送用于指示进入 DDL语句的目标处理阶段的第一指示信号,以使所述控制线 程分别向所述至少一个 DDL 线程发送与所述第一指示信号对应的第一通知信号, 令所述至少一个 DDL线程执行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送 表示到达目标同步点的第二通知信号, 所述目标同步点与所述目标处理阶段对应, 所述 DDL语句用 于修改所述至少一个物理分片的元信息; 接收所述控制线程发送与所述目标同步点对应的第二指示信号, 所述第二指示信号指示了所述 至少一个 DDL线程均发出所述第二通知信号; 根据所述第二指示信号确定所述 DDL语句的执行状态。
11、 一种分布式数据库系统, 包括: 计算节点和存储节点, 其中, 所述存储节点执行控制线程和与本地存储的至少一个物理分片对 应的至少一个 DDL线程; 所述计算节点,用于向所述控制线程发送用于指示进入 DDL语句的目标处理阶段的第一指示信 号, 所述 DDL语句用于修改所述至少一个物理分片的元信息; 所述控制线程,用于分别向所述 ■至少一个 DDL线程发送与所述第一指示信号对应的第一通知信 号; 所述至少一个 DDL线程,用于基于所述第一通知信号,执行所述目标处理阶段对应的处理行为, 并在执行完所述处理行为后向所述控制线程发送表示到达目标同步点的第二通知信号, 所述目标同 步点与所述目标处理阶段对应; 所述控制线程, 还用于向所述计算节点发送与所述目标同步点对应的第二指示信号, 所述第二 指示信号指示了所述至少一个 DDL线程均发出所述第二通知信号; 所述计算节点, 还用于根据所述第二指示信号确定所述 DDL语句的执行状态。
12、 一种电子设备, 包括: 存储器、 处理器、 通信接口; 其中, 所述存储器上存储有可执行代 码, 当所述可执行代码被所述处理器执行时, 使所述处理器执行如权利要求 1至 9中任一项所述的 DDL语句处理方法。
13、 一种非暂时性机器可读存储介质, 其中, 所述非暂时性机器可读存储介质上存储有可执行 代码, 当所述可执行代码被电子设备的处理器执行时, 使所述处理器执行如权利要求 1至 9中任一 项所述的 DDL语句处理方法, 或者执行如权利要求 10所述的 DDL语句处理方法。
19
14、 一种计算机程序产品, 包括: 计算机程序, 其中, 当所述计算机程序被电子设备的处理器 执行时,使所述处理器执行如权利要求 1至 9中任一项所述的 DDL语句处理方法, 或者执行如权利 要求 10所述的 DDL语句处理方法。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410471926.2A CN120832358A (zh) | 2024-04-17 | 2024-04-17 | Ddl语句处理方法、系统、设备和存储介质 |
| CN202410471926.2 | 2024-04-17 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2025219804A1 true WO2025219804A1 (zh) | 2025-10-23 |
Family
ID=97398176
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/IB2025/053633 Pending WO2025219804A1 (zh) | 2024-04-17 | 2025-04-07 | Ddl语句处理 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN120832358A (zh) |
| WO (1) | WO2025219804A1 (zh) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656384A (zh) * | 2021-10-18 | 2021-11-16 | 阿里云计算有限公司 | 数据处理方法、分布式数据库系统、电子设备及存储介质 |
| CN114816682A (zh) * | 2022-04-19 | 2022-07-29 | 泽拓科技(深圳)有限责任公司 | 分布式事务处理方法、系统及装置 |
| CN115114260A (zh) * | 2021-08-24 | 2022-09-27 | 腾讯科技(深圳)有限公司 | 数据处理方法、装置、电子设备及存储介质 |
| CN117435600A (zh) * | 2023-06-13 | 2024-01-23 | 金篆信科有限责任公司 | 分布式数据库数据定义操作原子性的实现方法及装置 |
| CN117687831A (zh) * | 2023-12-26 | 2024-03-12 | 金篆信科有限责任公司 | 一种分布式数据库的ddl处理方法、装置、系统及介质 |
-
2024
- 2024-04-17 CN CN202410471926.2A patent/CN120832358A/zh active Pending
-
2025
- 2025-04-07 WO PCT/IB2025/053633 patent/WO2025219804A1/zh active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115114260A (zh) * | 2021-08-24 | 2022-09-27 | 腾讯科技(深圳)有限公司 | 数据处理方法、装置、电子设备及存储介质 |
| CN113656384A (zh) * | 2021-10-18 | 2021-11-16 | 阿里云计算有限公司 | 数据处理方法、分布式数据库系统、电子设备及存储介质 |
| CN114816682A (zh) * | 2022-04-19 | 2022-07-29 | 泽拓科技(深圳)有限责任公司 | 分布式事务处理方法、系统及装置 |
| CN117435600A (zh) * | 2023-06-13 | 2024-01-23 | 金篆信科有限责任公司 | 分布式数据库数据定义操作原子性的实现方法及装置 |
| CN117687831A (zh) * | 2023-12-26 | 2024-03-12 | 金篆信科有限责任公司 | 一种分布式数据库的ddl处理方法、装置、系统及介质 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN120832358A (zh) | 2025-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10503699B2 (en) | Metadata synchronization in a distrubuted database | |
| KR101840996B1 (ko) | 파일 시스템에 대한 체크포인트 | |
| CN110209735B (zh) | 数据库备份方法、数据库备份装置、计算设备和存储介质 | |
| US8965840B2 (en) | Automatic synchronization conflict resolution | |
| US9575849B2 (en) | Synchronized backup and recovery of database systems | |
| JP2022511084A (ja) | ブロックチェーン技術を用いてデータベースアプリケーションを増強するためのシステムおよび方法 | |
| US20240211488A1 (en) | Transaction commitment systems, methods, and apparatuses based on distributed database systems | |
| US20070220328A1 (en) | Shutdown recovery | |
| US20090006500A1 (en) | Namespace replication program, namespace replication device, and namespace replication method | |
| US20050262170A1 (en) | Real-time apply mechanism in standby database environments | |
| US12210505B2 (en) | Operation request processing method, apparatus, device, readable storage medium, and system | |
| CN111930850A (zh) | 数据校验方法、装置、计算机设备和存储介质 | |
| CN111506592A (zh) | 一种数据库的升级方法和装置 | |
| CN109902127B (zh) | 历史态数据处理方法、装置、计算机设备及存储介质 | |
| CN111737331A (zh) | 一种数据库与对象存储的事务一致性处理方法及系统 | |
| CN110196788B (zh) | 一种数据读取方法、装置、系统及存储介质 | |
| CN109947742A (zh) | 面向二阶段锁的多版本数据库并发控制方法和系统 | |
| WO2024174516A1 (zh) | 数据同步方法和装置 | |
| JP2004318288A (ja) | データ処理方法および装置ならびにその処理プログラム | |
| JP2004164401A (ja) | データベースシステム、センタサーバ及びデータベースのアクセス方法 | |
| CN116775609A (zh) | Oracle到GaussDB的数据库迁移方法及装置 | |
| CN114356508B (zh) | 事务回放方法以及装置 | |
| Zhou et al. | FoundationDB: a distributed key value store | |
| US12547611B2 (en) | Control method, and adapter | |
| CN106407385B (zh) | 数据管理方法、设备和系统 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 25790214 Country of ref document: EP Kind code of ref document: A1 |