WO2018014650A1 - Procédé de synchronisation de données pour base de données répartie, appareil et système associés - Google Patents

Procédé de synchronisation de données pour base de données répartie, appareil et système associés Download PDF

Info

Publication number
WO2018014650A1
WO2018014650A1 PCT/CN2017/085486 CN2017085486W WO2018014650A1 WO 2018014650 A1 WO2018014650 A1 WO 2018014650A1 CN 2017085486 W CN2017085486 W CN 2017085486W WO 2018014650 A1 WO2018014650 A1 WO 2018014650A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
data node
node
version
log
Prior art date
Application number
PCT/CN2017/085486
Other languages
English (en)
Chinese (zh)
Inventor
陶维忠
吴刚
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2018014650A1 publication Critical patent/WO2018014650A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the present invention relates to the field of database technologies, and in particular, to a distributed database data synchronization method, related device and system.
  • Distributed database specifically refers to the use of high-speed computer networks to connect physically dispersed multiple data storage nodes to form a logically unified database.
  • the basic idea of a distributed database is to distribute the data in the original centralized database to multiple data storage nodes connected through the network to obtain larger storage capacity and higher concurrent access.
  • the prior art provides a distributed database. As shown in FIG. 1, the same data is stored on the data node 1 and the data node 2, and both the data node 1 and the data node 2 are in an active state.
  • the data operation instruction may be sent to the data node 1 or the data node 2 for execution. After the data node executes the data operation instruction, the execution result is synchronized to another data node by the form of the replication log, and the other data node performs data replication according to the replication log. , thereby achieving data synchronization between data nodes 1 and 2.
  • the prior art provides a data node-based data synchronization method.
  • the method is described below with a data node 2 upgrade. The method mainly includes the following steps:
  • Node 1 suspends processing of data operation commands sent by the application client and suspension of data replication log transmission between node 1 and node 2.
  • node 2 receives the upgrade instruction, performs a version upgrade operation, and then node 2 receives the data operation command sent by the application client and executes, and records the data replication log.
  • the node 2 sends the upgrade command and the data replication log to the node 1, and the node 1 receives the data replication log and the upgrade command sent by the node 2.
  • S4 and node 1 first upgrade the version according to the upgrade instruction, and then perform data replication according to the replication log. After the replication is completed, the data operation instruction sent by the application client is normally received, and the normal data between the node 1 and the node 2 is restored. Synchronize.
  • the data node 1 cannot process the data operation command sent by the application client, and only the data node 2 can process the data operation command sent by the application client, thereby reducing the reliability of the distributed database. .
  • Embodiments of the present invention provide a data synchronization method and apparatus for improving reliability of a distributed database.
  • an embodiment of the present invention provides a distributed database data synchronization method, which is applied to a destination data node in the distributed database, where the destination data node further includes a data operation instruction sent by an application client.
  • the upgrade is performed according to the upgrade instruction, and after the upgrade is completed, data replication is performed according to the cached replication log.
  • the upgrade instruction carries a field for modifying a data model of the destination data node or an instruction for adding a field of the data model.
  • the source data node can normally process the data operation instruction sent by the application client, and generate a data replication log to be sent to the destination data node.
  • the destination data node When the destination data node is upgraded, it can receive the data replication log sent by the source data node.
  • the data replication log is cached first, and after the version upgrade is completed, the data replication is performed according to the cached replication log, and the synchronization between the source data node and the destination data node is completed.
  • the upgrade method provided by the embodiment of the present invention enables the source data node and the destination data node to process data operation commands sent by the application client while upgrading, thereby improving the reliability of the distributed database.
  • the destination data node obtains the version of the source data node.
  • the method further includes:
  • the destination data node generates a data replication log when the data operation instruction is executed, where the data replication log carries data that needs to be synchronized to a data node corresponding to the destination data node;
  • the data replication log is sent to the source data node to enable data synchronization between the destination data node and the source data node.
  • the destination data node when the destination data node further determines that the version of the source data node is lower than its own version, it indicates that it can process the replication log sent by the source data node, and the data.
  • the node performs data copying according to the copy log, thereby implementing data synchronization from a low version to a high version.
  • the sending, by the destination data node, the data replication log to the source data node includes:
  • the replication log is placed in a local transmit queue and sent to the source data node.
  • the replication log can be passed to improve the availability of the database system.
  • an embodiment of the present invention provides a destination data node of a distributed database, including:
  • a data operation instruction processing unit configured to execute a data operation instruction sent by the application client
  • a receiving unit configured to receive a data replication log sent by the source data node, where the replication log carries data that needs to be synchronized to the destination node;
  • a log cache unit when the version of the source data node is obtained, and the version of the source data node is determined to be higher than its own version, the replication log is cached;
  • the receiving unit is further configured to receive an upgrade instruction sent by a metadata server, where the upgrade instruction is used to indicate the item Data node for version upgrade;
  • An upgrade unit configured to perform an upgrade according to the upgrade instruction
  • the log synchronization unit is configured to perform data replication according to the cached replication log after the upgrade is completed.
  • the obtaining, by the log cache unit, the version of the source data node includes:
  • the log cache unit acquires a version of the source data node according to the data replication log, where the data replication log carries a version of the source data node;
  • the log buffering unit receives the notification message sent by the source data node, where the notification message carries the version of the source data node, and obtains the version of the source data node according to the notification message.
  • the destination data node further includes:
  • a log generating unit configured to: when the data operation instruction is executed, generate a data replication log, where the data replication log carries data that needs to be synchronized to a data node corresponding to the destination data node;
  • a sending unit configured to send the data replication log to the source data node.
  • the log synchronization unit of the destination data node is further configured to perform data replication according to the replication log when determining that the version of the source data node is lower than its own version.
  • the sending, by the sending unit of the destination data node, the data replication log to the source data node includes:
  • the sending unit sends the replication log to a local sending queue to the source data node.
  • an embodiment of the present invention provides a distributed database data synchronization method, which is applied to a destination data node in the distributed database, where the destination data node further includes a data operation instruction sent by the application client.
  • the data corresponding to the entry to be deleted is filtered when the data is copied according to the replication log, and the data is already in the destination data node.
  • the existing entries are processed according to default values to achieve data synchronization between the source data node and the destination data node.
  • the method further includes:
  • the destination data node can know in advance which data corresponding to the entry needs to be deleted, thereby receiving the replication log. Or when the replication log is sent to the source data node, the data is synchronized in advance.
  • the method further includes:
  • the data replication log When the data replication log is generated according to the execution result of the data operation instruction, the data corresponding to the entry to be deleted is filtered, and the data replication log is sent to the source data node.
  • the embodiment of the present invention provides a distributed database destination data node, where the destination data node specifically includes:
  • a data operation instruction processing unit configured to execute a data operation instruction sent by the application client
  • a receiving unit configured to receive an upgrade pre-notification instruction sent by the metadata server, where the upgrade pre-notification instruction is Carry the entry that needs to be deleted;
  • the receiving unit is further configured to receive a data replication log sent by the source data node, where the data replication log carries data that needs to be synchronized to the destination node;
  • a log processing unit configured to filter data corresponding to the entry to be deleted when the data is copied according to the copy log, when the data corresponding to the entry to be deleted is included in the copy log,
  • the entries already existing in the destination data node are processed according to default values, thereby implementing data synchronization between the source data node and the destination data node.
  • the destination data node further includes:
  • an upgrade unit configured to receive an upgrade command sent by the metadata server, where the upgrade command is used to delete an entry of the data node, and delete an entry of the data node according to the upgrade instruction.
  • the destination data node can know in advance which data corresponding to the entry needs to be deleted, thereby receiving the replication log. Or when the replication log is sent to the source data node, the data is synchronized in advance.
  • the destination data node further includes:
  • a log sending unit configured to: when the data replication log is generated according to the execution result of the data operation instruction, filter the data corresponding to the entry to be deleted, and send the data replication log to the source data node.
  • the sending, by the destination data node, the data replication log to the source data node includes:
  • the replication log is placed in the local send queue and sent to the source data node.
  • an embodiment of the present invention provides a distributed database, including the destination data node according to the second aspect or the fourth aspect, and a source data node corresponding to the destination data node.
  • the distributed data provided by the embodiment of the present invention can complete data synchronization between the source data node and the destination data node while upgrading the source data node and the destination data node, thereby further improving the reliability of the distributed database.
  • FIG. 1 is a schematic structural diagram of a distributed database provided by the prior art
  • FIG. 2 is a schematic structural diagram of a distributed database according to Embodiment 1 of the present invention.
  • FIG. 3 is a schematic diagram of data storage of a data node according to an embodiment of the present invention.
  • FIG. 4 is a flowchart of a data synchronization method according to Embodiment 2 of the present invention.
  • FIG. 5 is a flowchart of a data synchronization method according to Embodiment 3 of the present invention.
  • FIG. 6 is a schematic diagram of data storage of a data node according to an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a destination data node according to Embodiment 5 of the present invention.
  • FIG. 9 is a schematic structural diagram of a destination data node according to Embodiment 6 of the present invention.
  • FIG. 10 is a schematic structural diagram of a destination data node in a distributed database according to Embodiment 7 of the present invention.
  • FIG. 2 is a schematic structural diagram of a database system according to Embodiment 1 of the present invention.
  • the data processing system of the present invention includes an application client, a metadata server, and a data node server from top to bottom.
  • the application client is an application that uses a database, such as a billing application.
  • the application client can access the data stored in the data node server.
  • the metadata server is responsible for the distributed management capabilities of the database system.
  • the metadata server can be deployed independently or in conjunction with the data node server.
  • the metadata server can be deployed using a minicomputer, an X86 computer, or a personal computer server PC server.
  • the application client communicates with the metadata server and the data node server respectively through the IP network, wherein the communication interface between the client and the metadata server or the data node server may be a Transmission Control Protocol (TCP) interface or User Datagram Protocol (UDP) interface.
  • TCP Transmission Control Protocol
  • UDP User Datagram Protocol
  • the data node server includes a plurality of data nodes (also referred to as physical nodes), and the data nodes may be minicomputers, X86 computers, or personal computer servers, PC servers, etc., and data in the data nodes may be stored in a storage medium located in the storage network.
  • a plurality of data nodes and a storage network are read and written by a block IO (block IO), that is, a storage medium is read and written by a block.
  • the storage medium may be a hard disk drive (HDD) or a solid state. Hard State Drives (SSD), etc.
  • the client also deploys a driver driver, which caches routing information. In this way, when the client sends a data operation instruction to the physical node, the client can complete the route judgment through the cached routing information and access the corresponding physical node.
  • Figure 3 shows a schematic diagram of data storage on a data node.
  • Data node 1 and data node 2 are backups of each other, a slice located on data node 1 (which may also be referred to as a virtual node) 1, and a slice 2 located on data node 2 as a primary slice, and a segment on data node 1 Slice 2 and slice 1 located on data node 2 are spare slices.
  • Data node 1 and data node 2 may be referred to as a source data node and a destination data node, respectively. The names of the source data node and the destination data node are relative.
  • the metadata server stores the mapping relationship between the virtual node and the physical node, and the definition of the primary fragment and the backup slice of the virtual node.
  • the metadata server is also used to control data synchronization between data nodes.
  • FIG. 4 is a flowchart of a data synchronization method according to Embodiment 2 of the present invention.
  • the data node 1 is a source data node
  • the data node 2 is a destination data node.
  • the data synchronization method of the database provided by this embodiment can be applied to the scenario of database upgrade, and the method mainly includes the following steps:
  • Step 301 The data node 1 receives an upgrade instruction sent by the metadata server, where the upgrade instruction is used to indicate that the source data node performs version upgrade.
  • the data nodes 1 and 2 successively receive the upgrade instruction sent by the metadata server to notify the upgrade from the version V1 to the V2, and the upgrade instruction may modify the field of the data model (for example, a table structure) or add a field of the data model. Due to the existence of the time when the upgrade command is received, the data models of data nodes 1 and 2 are inconsistent.
  • the data node 1 first receives an upgrade instruction as an example.
  • Step 302 The data node 1 performs an upgrade according to the upgrade instruction.
  • Step 303 After the upgrade succeeds, the data node 1 receives the data operation command sent by the application client, performs a data operation according to the data operation command, and generates a replication log.
  • the data node 1 can receive data operation instructions sent by the application client, such as querying, modifying, and the like of the data of the primary fragment 1 while performing the upgrade. After performing the data operation, the data node 1 generates a replication log, which carries data that needs to be synchronized to the data node corresponding to the backup slice 1 (ie, the data node 2).
  • Step 304 the data node 1 sends the generated replication log to the data node 2.
  • Data Node 1 can also receive and process the data operation instructions sent by the client and generate a replication log. At this time, the versions of the data node 1 and the data node 2 are the same, and the data node 1 directly transmits the copy log to the data node 2 to realize data synchronization between the data node 1 and the data node 2.
  • the data node 1 can also receive and process the data operation instructions sent by the client and generate a replication log.
  • the version of the data node 1 is higher than the version of the data node 2, that is, the data model is out of synchronization between the data node 1 and the data node 2, and the data node 2 in the prior art cannot process the copy received from the data node 1.
  • the log is described in the embodiment of the present invention.
  • the replication log can be transmitted asynchronously between the data node 1 and the data node 2, that is, the data node 1 can put the replication log into the local transmission queue, and the local processing ends. Passing the replication log relative to the synchronous mode (the destination data node handles the replication log successfully) can improve the availability and partition fault tolerance of the database system.
  • Step 305 The data node 2 receives the data replication log sent by the data node 1, and the replication log carries data that needs to be synchronized to the data node 1.
  • Step 306 The data node 2 acquires the version of the data node 1, and determines that the version of the data node 1 is higher than its own version, and caches the replication log.
  • the copy log may carry the version of the data node 1, and the data node 2 may obtain the version of the data node 1 from the copy log, and compare the version of the data node 1 with its own version.
  • the data node 2 can also receive the notification message sent by the data node 1, the notification message carrying the version of the source data node, and the data node 2 acquiring the version of the data node 1 according to the notification message.
  • the version of the data node 1 is higher than its own version, it indicates that the copy log is generated according to the new data model, and the data node 2 temporarily does not process the copy log and cache the copy log.
  • the data node 2 can process the replication log sent by the data node 1, that is, according to the replication log. data synchronization. If the version of data node 1 is the same as its own version, data node 2 can process the replication log sent by data node 1, ie, synchronize data according to the replication log.
  • Step 307 The data node 2 receives an upgrade instruction sent by the metadata server, where the upgrade instruction is used to instruct the data node 2 to perform a version upgrade.
  • the data nodes 1, 2 receive the upgrade instructions sent by the source data server in succession.
  • Step 308 the data node 2 is upgraded according to the upgrade instruction.
  • Step 309 After the upgrade is completed, the data node 2 performs data replication according to the cached replication log.
  • the version of the data node 2 is the same as the version of the data node 1, and the data node 2 starts to process the replication log, that is, performs data replication according to the replication log cached in step 306 (also referred to as data replication). Data redo) to ensure data synchronization between Data Node 1 and Data Node 2.
  • the source data node (data node 1) can normally process the data operation instruction sent by the application client, and generate a data replication log to be sent to the destination data node (data node 2).
  • the destination data node When the destination data node is upgraded, it can receive the data replication log sent by the source data node.
  • the data replication log is cached first, and after the version upgrade is completed, the data replication is performed according to the cached replication log, and the synchronization between the source data node and the destination data node is completed.
  • the upgrade method provided by the embodiment of the present invention enables the source data node and the destination data node to process data operation commands sent by the application client while upgrading, thereby improving the reliability of the distributed database.
  • the data node 2 may perform the data operation instruction sent by the application client while performing the foregoing upgrade, and the data node 2 may further perform the following steps:
  • Step 310 When executing the data operation instruction, the data node 2 generates a data replication log, where the data replication log carries data that needs to be synchronized to the data node corresponding to the data node 2.
  • the data node corresponding to the data node 2 is the data node 1.
  • the modified data carries the modified data, and the data needs to be synchronized to the backup slice 2 of the data node 1.
  • Step 311 The data node 2 sends the data replication log to the data node 1.
  • the replication log can be transmitted asynchronously between the data node 2 and the data node 1, that is, the data node 2 can put the replication log into the local transmission queue, and the local processing ends. Passing replication logs synchronously can improve database system availability and partition fault tolerance.
  • steps 310-311 and the previous steps 305-309 are timing-independent, that is, the data node 1 and the data node 2 receive the data operation instruction is timing-independent.
  • FIG. 5 is a flowchart of a data synchronization method according to Embodiment 3 of the present invention.
  • the data node 1 is a source data node
  • the data node 2 is a destination data node.
  • the database data synchronization method provided in this embodiment is mainly described by taking the data node 2 as an example.
  • the method further includes the following steps:
  • Step 401 The data node 2 receives an upgrade pre-notification command sent by the metadata server, where the upgrade pre-notification command carries an entry that needs to be deleted.
  • the metadata server sends an upgrade pre-notification instruction to the data nodes 1 and 2 in advance before sending the upgrade instruction, where the upgrade pre-notification instruction is used to notify the data node that the data model needs to be modified in advance.
  • the notification instruction carries an entry (also referred to as a field) that needs to be deleted in the data model (for example, a table structure), indicating that the entry in the data model needs to be deleted.
  • the data node 2 (destination data node) is taken as an example for description.
  • Step 402 The data node 2 receives a data replication log sent by the data node 1, where the data replication log carries data that needs to be synchronized to the data node 2.
  • the data synchronization between the data node 1 and the data node 2 does not stop.
  • the data node 1 After receiving the data operation instruction sent by the application client, the data node 1 performs a corresponding data operation, generates a data replication log, and synchronizes to the data node. 1.
  • the data replication log carries data that needs to be synchronized to data node 2.
  • Step 403 When it is determined that the data corresponding to the entry to be deleted is included in the replication log, the data node 2 filters the data corresponding to the entry to be deleted when performing data replication according to the replication log, for the purpose.
  • the existing entries in the data node are processed according to the default values.
  • the upgrade pre-notification instruction carries the need to delete
  • the data node 2 may determine whether the data corresponding to the entry to be deleted is included in the replication log, and if so, the data node 2 filters the entry to be deleted when performing data replication according to the received replication log.
  • Corresponding data is processed according to the default value of the existing entries in the data node 2, that is, the data corresponding to the entry to be deleted in the data node 2 is not modified (avoiding modification first, then deleting when upgrading), thereby realizing Data synchronization between data nodes 1 and 2 improves the efficiency of data synchronization.
  • Step 404 The data node 2 receives the upgrade command sent by the metadata server, where the upgrade command is used to delete the entry of the data node, and delete the entry of the data node according to the upgrade instruction.
  • the data node 2 can execute the upgrade command sent by the metadata server while processing the copy log, that is, the step 404 and the steps 401-403 are time-independent, and the step 404 can also be performed before the step 401. .
  • the metadata server sends an upgrade instruction for deleting the entry of the data node to the data nodes 1 and 2.
  • the data node 1, 2 deletes the entry in the data node 1, 2 according to the upgrade instruction.
  • Step 405 The data node 2, when generating the data replication log according to the execution result of the data operation instruction, filters the data corresponding to the entry to be deleted, and sends the data replication log to the source data node.
  • the data node 2 after executing the data operation instruction sent by the application client, the data node 2 itself generates a data replication log according to the execution result of the data operation instruction.
  • the data node 2 can filter the data corresponding to the entry to be deleted when the data replication log is generated, so that the data corresponding to the entry to be deleted is not carried in the replication log, and the transmission efficiency of the data replication log is improved.
  • step 405 and steps 402-404 are time-independent, for example, step 405 can be performed before step 402.
  • the replication log can be transmitted asynchronously between the data node 1 and the data node 2 in this embodiment, that is, the data node 2 can put the replication log into the local transmission queue, and the processing ends even if the local processing is completed. Passing replication logs synchronously can improve database system availability and partition fault tolerance.
  • FIG. 6 is a schematic diagram of data storage of a data node according to an embodiment of the present invention.
  • the distributed database consists of four physical nodes 1, 2, 3 and 4, each node has 3 main shards (the part filled with slashes in the figure) and 3 spare shards (unfilled parts in the figure) .
  • slice 3 is copied from node 1 to node 2
  • slice 6 is copied from node 2 to node 1.
  • Each shard in the distributed database consists of two table structures, Table_A and Table_B, which are currently in V1 version.
  • the table structure is as follows:
  • Table_A uses cust_id as the primary key.
  • Table_B uses product_id as the primary key.
  • FIG. 7 is a data synchronization method of a distributed database according to Embodiment 4 of the present invention.
  • the above synchronization methods mainly include:
  • Step 601 The metadata server receives a table structure upgrade instruction sent by the application client.
  • the table structure upgrade instruction includes a Table_A delete field and a Table_B add field.
  • Step 602 The metadata server sends an upgrade pre-notification command to all the data nodes, where the upgrade pre-notification command carries the entry to be deleted.
  • the upgrade pre-notification command is used to inform the data node to upgrade from version V1 to V2, and inform the table_A in the data node that the field to be deleted (also referred to as a heterogeneous field) cust_bank.
  • the data node 1 and the data node 2 are taken as an example for illustration.
  • step 603 the data nodes 1, 2 demote the synchronous replication to the asynchronous replication mode.
  • the steps of the data nodes 1 and 2 processing the upgrade pre-notification command include demoting the synchronous replication to the asynchronous replication mode.
  • the replication is degraded to ensure that the log replication process changes during the upgrade process, and does not affect or block online services, ensuring high availability of the system.
  • each data node also identifies the field Table_A.cust_bank that needs to be deleted during the upgrade.
  • Step 604 The data node 1, 2 receives the data operation instruction sent by the application client, performs a corresponding data operation, and generates a replication log.
  • the online service generates a copy log for the write operation of the data nodes 1, 2.
  • the data node can directly filter out the field Table_A.cust_bank according to the identified field that needs to be deleted, and the manner of generating the replication log can be heterogeneous replication.
  • Step 605 The data node 1 asynchronously transmits the generated replication log to the data node 2.
  • the data of the primary fragment 3 of the data node 1 is modified, and the replication log generated by the data node 1 carries the data on the backup slice 3 that needs to be synchronized to the data node 2.
  • Step 606 The data node 2 performs data synchronization according to the replication log.
  • the heterogeneous field Table_A.cust_bank exists in the replication log, and the data node 2 performs filtering when performing data synchronization (also referred to as data redo processing), and the heterogeneous field Table_A.cust_bank is also processed according to the default value (nul l). .
  • the data node 1 also receives the replication log sent by the data node 2, and performs data synchronization according to the replication log.
  • Step 607 The metadata server sends an upgrade instruction to the data nodes 1 and 2 to instruct to perform a version upgrade.
  • the metadata server sends an upgrade instruction through a Data Definition Language (DDL) operation, and the upgrade content includes a Table_A table deletion field cust_bank, and a Table_B table adds a field.
  • Product_discount int is a Data Definition Language (DDL) operation, and the upgrade content includes a Table_A table deletion field cust_bank, and a Table_B table adds a field.
  • the default data node 1 in the embodiment first receives the upgrade command as an example.
  • Step 608 the data node 1 first receives and processes the DDL upgrade instruction.
  • Step 609 the data node 1 sends the generated replication log to the data node 2.
  • the replication log generated by the online service to the primary fragment 3 write operation of the data node 1 is synchronously synchronized to the number node 2, and the replication log is matched with the new version V2 (the field is added in the replication log Table_B.Product_discount, and the field Table_A is deleted. Cust_bank).
  • Step 610 The data node 2 performs data synchronization according to the replication log.
  • Data node 2 obtains version V2 of data node 1 from the replication log. Since version V1 of data node 2 is low (no field Table_B.Product_discount), the replication log of V2 version cannot be processed (the log information of the added field Table_B.Product_discount cannot be recognized). ). At this point, data node 2 first caches this portion of the replication log.
  • Step 611 the data node 2 sends the generated replication log to the data node 1.
  • the replication log Since the data node 2 writes the primary fragment 6 of the data node 2 before the DDL upgrade command is received, the replication log is generated and synchronized to the data node 1 in an asynchronous manner. The replication log is matched with the old version V1.
  • Step 612 The data node 1 performs data synchronization according to the replication log.
  • the copy log of the V1 version fragment 6 can be processed.
  • the data node 1 normally processes the fragment 6 copy log transmitted from the data node 2, and the new field Table_B.Product_discount that is not involved in the V1 version of the data node 2, according to the default value nul when performing the copy processing. l processing.
  • processing is performed according to the processing logic of step 606.
  • Step 613 the data node 2 processes the DDL upgrade instruction.
  • the data node 2 receives the DDL upgrade instruction sent by the metadata server with respect to the data node 1.
  • the data node 2 executes the DDL upgrade instruction to complete the version upgrade.
  • the Table_A table deletes the field cust_bank, and the Table_B table adds the field Product_discount int.
  • Step 614 The data node 2 performs data synchronization according to the cached replication log.
  • the upgraded data node has the version V2, and can process the previously cached replication logs to complete data synchronization between the data nodes 1 and 2.
  • Step 615 After the upgrade is completed, the data nodes 1, 2 send a notification message of successful upgrade to the metadata server.
  • the metadata server After identifying that all nodes have completed the upgrade, the metadata server enters the post-upgrade processing, for example, notifying each data node to restore the replication level to synchronous replication. At the same time, heterogeneous replication is also cancelled, and the replication log is generated normally.
  • FIG. 8 is a schematic structural diagram of a destination data node according to Embodiment 5 of the present invention.
  • the destination data node may be the data node 2 shown in Figure 3-5.
  • the destination data node employs general purpose computer hardware including a processor 101, a memory 102, a bus 103, an input device 104, an output device 105, and a network interface 106.
  • memory 102 can include computer storage media in the form of volatile and/or nonvolatile memory, such as read only memory and/or random access memory.
  • the memory 102 can store an operating system, an application, and other programs Modules, executable code, and program data.
  • the input device 104 can be used to input commands and information to a destination data node, such as a keyboard or pointing device, such as a mouse, trackball, touchpad, microphone, joystick, game pad, round dish satellite television antenna, scanner Or similar equipment. These input devices can be connected to the processor 101 via a bus 103.
  • a destination data node such as a keyboard or pointing device, such as a mouse, trackball, touchpad, microphone, joystick, game pad, round dish satellite television antenna, scanner Or similar equipment.
  • the output device 105 can be used for the destination data node to output information.
  • the output device 105 can also be configured for other peripheral outputs, such as speakers and/or printing devices, which can also be connected to the processor via the bus 103. 101.
  • the destination data node can be connected to the network through the network interface 106, for example to a local area network (LAN).
  • LAN local area network
  • computer-executed instructions stored in a destination data node may be stored in a remote storage device, and are not limited to being stored locally.
  • the destination data node may perform the method steps on the destination data node side in the second, third, and fourth embodiments above, for example, performing steps 305-311, 401-405, 603, 606, 610, and the like.
  • steps 305-311, 401-405, 603, 606, 610, and the like for details, refer to the second, third, and fourth embodiments, and details are not described herein again.
  • FIG. 9 is a schematic structural diagram of a destination data node according to Embodiment 6 of the present invention.
  • the destination data node provided by the embodiment of the present invention includes:
  • the data operation instruction processing unit 710 is configured to execute a data operation instruction sent by the application client;
  • the receiving unit 720 is configured to receive a data replication log sent by the source data node, where the replication log carries data that needs to be synchronized to the destination node.
  • the log buffering unit 730 is configured to obtain a version of the source data node, and determine that the version of the source data node is higher than its own version, and cache the replication log.
  • the receiving unit 720 is further configured to receive an upgrade instruction sent by the metadata server, where the upgrade instruction is used to indicate that the destination data node performs a version upgrade.
  • the upgrading unit 740 is configured to perform an upgrade according to the upgrade instruction
  • the log synchronization unit 750 is configured to perform data replication according to the cached replication log after the upgrade is completed.
  • the destination data node provided by the embodiment of the present invention may be used in the foregoing method embodiments 2 and 4, which is between the data operation instruction unit 710, the receiving unit 720, the log buffer unit 730, the upgrading unit 740, and the log synchronization unit 750.
  • the cooperation steps are performed to complete the method steps on the data node side of the second and fourth embodiments.
  • the destination data node provided by this embodiment has the same beneficial effects as the foregoing method embodiment when performing data synchronization.
  • the version of the source data node obtained by the log cache unit 730 in the destination data node includes:
  • the log cache unit 730 obtains a version of the source data node according to the data replication log, where the data replication log carries a version of the source data node;
  • the log buffering unit 730 receives the notification message sent by the source data node, where the notification message carries the version of the source data node, and obtains the version of the source data node according to the notification message.
  • the destination data node described in FIG. 5 further includes:
  • the log generating unit 760 is configured to generate a data replication log, where the data replication log carries data that needs to be synchronized to a data node corresponding to the destination data node, when the data operation instruction is executed;
  • the sending unit 770 is configured to send the data replication log to the source data node. Wherein, the sending unit 770 can The copy log is sent to the local data node by placing it in a local transmit queue.
  • the specific process of the log generation unit 760 generating the replication log and the sending unit 770 sending the replication log to the source data node may refer to the description of steps 310-311 in the foregoing method embodiment.
  • the log synchronization unit 750 of the destination data node is further configured to perform data replication according to the replication log when determining that the version of the source data node is lower than its own version.
  • FIG. 10 is a schematic structural diagram of a destination data node in a distributed database according to Embodiment 7 of the present invention.
  • the destination data node specifically includes:
  • the data operation instruction processing unit 810 is configured to execute a data operation instruction sent by the application client;
  • the receiving unit 820 is configured to receive an upgrade pre-notification command sent by the metadata server, where the upgrade pre-notification command carries an entry that needs to be deleted;
  • the receiving unit 820 is further configured to receive a data replication log sent by the source data node, where the data replication log carries data that needs to be synchronized to the destination node;
  • the log processing unit 830 is configured to: when the data corresponding to the entry to be deleted is included in the replication log, filter the data corresponding to the entry to be deleted when performing data replication according to the replication log, The table items already existing in the destination data node are processed according to default values, thereby implementing data synchronization between the source data node and the destination data node.
  • the destination data node provided by the embodiment of the present invention may be used in the foregoing method embodiments 3 and 4, and the third embodiment is implemented by the cooperation between the data operation instruction unit 810, the receiving unit 820, and the log processing unit 830. Method steps on the data node side of the fourth. Compared with the destination data node in the prior art, the destination data node provided by this embodiment has the same beneficial effects as the foregoing method embodiment when performing data synchronization.
  • the destination data node provided by the embodiment of the present invention further includes:
  • the upgrading unit 840 is configured to receive an upgrade command sent by the metadata server, where the upgrade command is used to delete an entry of the data node, and delete an entry of the data node according to the upgrade instruction.
  • the destination data node can know in advance which data corresponding to the entry needs to be deleted, thereby receiving the replication log. Or when the replication log is sent to the source data node, the data is synchronized in advance.
  • the log sending unit 850 is configured to filter data corresponding to the entry to be deleted when the data replication log is generated according to the execution result of the data operation instruction, and send the data replication log to the source data node.
  • the log sending unit 850 can send the replication log to the local sending queue to the source data node.
  • the destination data node is presented in the form of a functional unit.
  • a "unit” herein may refer to an application-specific integrated circuit (ASIC), circuitry, a processor and memory that executes one or more software or firmware programs, integrated logic circuitry, and/or other functions that provide the functionality described above. Device.
  • ASIC application-specific integrated circuit
  • the destination data node can also take the form shown in FIG.
  • the functions implemented by the data operation instruction unit 710, the reception unit 720, the log buffer unit 730, the upgrade unit 740, and the log synchronization unit 750 can all be implemented by the processor 101 and the memory 102 in FIG.
  • the data operation instruction processing unit 710 executing the data operation instruction sent by the application client may be executed by the processor 101.
  • the code stored in the line memory 102 is implemented.
  • the processor for implementing the above-mentioned data node of the present invention may be a central processing unit (CPU), a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA). Or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the present disclosure.
  • the processor may also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like.
  • aspects of the present invention, or possible implementations of various aspects may be embodied as a system, method, or computer program product.
  • aspects of the invention, or possible implementations of various aspects may be in the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, etc.), or a combination of software and hardware aspects, They are collectively referred to herein as "circuits," “modules,” or “systems.”
  • aspects of the invention, or possible implementations of various aspects may take the form of a computer program product, which is a computer readable program code stored in a computer readable medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

La présente invention concerne un procédé de synchronisation de données dans lequel un nœud de données source traite normalement les consignes d'opérations sur les données envoyées par un client d'application et génère un journal de réplication de données à envoyer à un nœud de données de destination. Lors d'une mise à jour, le nœud de données de destination reçoit le journal de réplication de données envoyé par le nœud de données source. Lorsque la version du nœud de données source est supérieure à la version du nœud de données de destination, le nœud de données de destination place d'abord en antémémoire le journal de réplication et, après avoir réalisé une mise à jour de version de la version du nœud de données de destination, réalise une réplication de données en se basant sur le journal de réplication placé en antémémoire, réalisant ainsi une synchronisation entre le nœud de données source et le nœud de données de destination. Comparé à l'état antérieur de la technique, le procédé de mise à niveau selon les modes de réalisation de la présente invention permet à un nœud de données source et à un nœud de données de destination de traiter des consignes d'opérations sur les données envoyées par un client d'application pendant la mise à niveau, améliorant la fiabilité de la base de données répartie.
PCT/CN2017/085486 2016-07-20 2017-05-23 Procédé de synchronisation de données pour base de données répartie, appareil et système associés WO2018014650A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610578730.9A CN107644030B (zh) 2016-07-20 2016-07-20 分布式数据库数据同步方法、相关装置及系统
CN201610578730.9 2016-07-20

Publications (1)

Publication Number Publication Date
WO2018014650A1 true WO2018014650A1 (fr) 2018-01-25

Family

ID=60991877

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/085486 WO2018014650A1 (fr) 2016-07-20 2017-05-23 Procédé de synchronisation de données pour base de données répartie, appareil et système associés

Country Status (2)

Country Link
CN (1) CN107644030B (fr)
WO (1) WO2018014650A1 (fr)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11281484B2 (en) 2016-12-06 2022-03-22 Nutanix, Inc. Virtualized server systems and methods including scaling of file system virtual machines
CN114356347A (zh) * 2022-01-05 2022-04-15 中国建设银行股份有限公司 一种创建分片式数据库集群的方法及装置
US11537384B2 (en) 2016-02-12 2022-12-27 Nutanix, Inc. Virtualized file server distribution across clusters
US11562034B2 (en) 2016-12-02 2023-01-24 Nutanix, Inc. Transparent referrals for distributed file servers
US11568073B2 (en) 2016-12-02 2023-01-31 Nutanix, Inc. Handling permissions for virtualized file servers
US11768809B2 (en) 2020-05-08 2023-09-26 Nutanix, Inc. Managing incremental snapshots for fast leader node bring-up
US11770447B2 (en) 2018-10-31 2023-09-26 Nutanix, Inc. Managing high-availability file servers
US11775397B2 (en) 2016-12-05 2023-10-03 Nutanix, Inc. Disaster recovery for distributed file servers, including metadata fixers
CN117149905A (zh) * 2023-08-16 2023-12-01 上海沄熹科技有限公司 一种时序数据复制方法及装置
US11888599B2 (en) 2016-05-20 2024-01-30 Nutanix, Inc. Scalable leadership election in a multi-processing computing environment
US11954078B2 (en) 2016-12-06 2024-04-09 Nutanix, Inc. Cloning virtualized file servers
US12072770B2 (en) 2021-08-19 2024-08-27 Nutanix, Inc. Share-based file server replication for disaster recovery

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109033391B (zh) * 2018-07-31 2019-12-31 北京嘀嘀无限科技发展有限公司 一种多数据中心之间数据同步方法、系统及计算机可读存储介质
CN109491599A (zh) * 2018-10-24 2019-03-19 山东超越数控电子股份有限公司 一种分布式存储系统及其异构加速方法
CN111666331A (zh) * 2019-03-06 2020-09-15 华为技术有限公司 复制数据的方法、装置及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101388759A (zh) * 2007-09-10 2009-03-18 中兴通讯股份有限公司 实现数据的异步复制到同步复制的转换方法和系统
CN104346373A (zh) * 2013-07-31 2015-02-11 华为技术有限公司 分区日志队列同步管理方法及设备
US20150347250A1 (en) * 2014-05-30 2015-12-03 Altibase Corp. Database management system for providing partial re-synchronization and partial re-synchronization method of using the same
CN105512266A (zh) * 2015-12-03 2016-04-20 曙光信息产业(北京)有限公司 一种实现分布式数据库操作一致性的方法及装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101753349B (zh) * 2008-12-09 2012-08-15 中国移动通信集团公司 数据节点的升级方法、升级调度节点及升级系统
CN101908064A (zh) * 2010-07-20 2010-12-08 中兴通讯股份有限公司 数据库备份恢复方法和装置
US8924384B2 (en) * 2010-08-04 2014-12-30 Sap Ag Upgrading column-based databases
CN102088489B (zh) * 2010-12-31 2013-05-22 北京理工大学 一种分布式数据同步系统及方法
US9792321B2 (en) * 2013-07-09 2017-10-17 Oracle International Corporation Online database migration
CN105550288B (zh) * 2015-12-10 2019-07-02 百度在线网络技术（北京）有限公司 数据库系统的更新方法和管理系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101388759A (zh) * 2007-09-10 2009-03-18 中兴通讯股份有限公司 实现数据的异步复制到同步复制的转换方法和系统
CN104346373A (zh) * 2013-07-31 2015-02-11 华为技术有限公司 分区日志队列同步管理方法及设备
US20150347250A1 (en) * 2014-05-30 2015-12-03 Altibase Corp. Database management system for providing partial re-synchronization and partial re-synchronization method of using the same
CN105512266A (zh) * 2015-12-03 2016-04-20 曙光信息产业(北京)有限公司 一种实现分布式数据库操作一致性的方法及装置

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11922157B2 (en) 2016-02-12 2024-03-05 Nutanix, Inc. Virtualized file server
US11947952B2 (en) 2016-02-12 2024-04-02 Nutanix, Inc. Virtualized file server disaster recovery
US12014166B2 (en) 2016-02-12 2024-06-18 Nutanix, Inc. Virtualized file server user views
US11966729B2 (en) 2016-02-12 2024-04-23 Nutanix, Inc. Virtualized file server
US11550559B2 (en) 2016-02-12 2023-01-10 Nutanix, Inc. Virtualized file server rolling upgrade
US11550557B2 (en) 2016-02-12 2023-01-10 Nutanix, Inc. Virtualized file server
US11550558B2 (en) 2016-02-12 2023-01-10 Nutanix, Inc. Virtualized file server deployment
US11966730B2 (en) 2016-02-12 2024-04-23 Nutanix, Inc. Virtualized file server smart data ingestion
US11544049B2 (en) 2016-02-12 2023-01-03 Nutanix, Inc. Virtualized file server disaster recovery
US11579861B2 (en) 2016-02-12 2023-02-14 Nutanix, Inc. Virtualized file server smart data ingestion
US11645065B2 (en) 2016-02-12 2023-05-09 Nutanix, Inc. Virtualized file server user views
US11669320B2 (en) 2016-02-12 2023-06-06 Nutanix, Inc. Self-healing virtualized file server
US11537384B2 (en) 2016-02-12 2022-12-27 Nutanix, Inc. Virtualized file server distribution across clusters
US11888599B2 (en) 2016-05-20 2024-01-30 Nutanix, Inc. Scalable leadership election in a multi-processing computing environment
US11568073B2 (en) 2016-12-02 2023-01-31 Nutanix, Inc. Handling permissions for virtualized file servers
US11562034B2 (en) 2016-12-02 2023-01-24 Nutanix, Inc. Transparent referrals for distributed file servers
US11775397B2 (en) 2016-12-05 2023-10-03 Nutanix, Inc. Disaster recovery for distributed file servers, including metadata fixers
US11281484B2 (en) 2016-12-06 2022-03-22 Nutanix, Inc. Virtualized server systems and methods including scaling of file system virtual machines
US11922203B2 (en) 2016-12-06 2024-03-05 Nutanix, Inc. Virtualized server systems and methods including scaling of file system virtual machines
US11954078B2 (en) 2016-12-06 2024-04-09 Nutanix, Inc. Cloning virtualized file servers
US11770447B2 (en) 2018-10-31 2023-09-26 Nutanix, Inc. Managing high-availability file servers
US11768809B2 (en) 2020-05-08 2023-09-26 Nutanix, Inc. Managing incremental snapshots for fast leader node bring-up
US12072770B2 (en) 2021-08-19 2024-08-27 Nutanix, Inc. Share-based file server replication for disaster recovery
CN114356347A (zh) * 2022-01-05 2022-04-15 中国建设银行股份有限公司 一种创建分片式数据库集群的方法及装置
CN117149905A (zh) * 2023-08-16 2023-12-01 上海沄熹科技有限公司 一种时序数据复制方法及装置
CN117149905B (zh) * 2023-08-16 2024-05-24 上海沄熹科技有限公司 一种时序数据复制方法及装置

Also Published As

Publication number Publication date
CN107644030A (zh) 2018-01-30
CN107644030B (zh) 2021-05-18

Similar Documents

Publication Publication Date Title
WO2018014650A1 (fr) Procédé de synchronisation de données pour base de données répartie, appareil et système associés
US10929428B1 (en) Adaptive database replication for database copies
EP3811230B1 (fr) Renvoi automatique de requêtes à une base de donnees stand-by
US12063270B2 (en) Commissioning and decommissioning metadata nodes in a running distributed data storage system
AU2016405587B2 (en) Splitting and moving ranges in a distributed system
US9031910B2 (en) System and method for maintaining a cluster setup
US11893264B1 (en) Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity
US8229893B2 (en) Metadata management for fixed content distributed data storage
US10922303B1 (en) Early detection of corrupt data partition exports
JP2023546249A (ja) トランザクション処理方法、装置、コンピュータ機器及びコンピュータプログラム
JP2020514902A (ja) データセット及び他の管理オブジェクトをクラウドベースのストレージシステムに同期複製すること
US9218405B2 (en) Batch processing and data synchronization in cloud-based systems
US9515878B2 (en) Method, medium, and system for configuring a new node in a distributed memory network
CN105493474B (zh) 用于支持用于同步分布式数据网格中的数据的分区级别日志的系统及方法
EP2619695A2 (fr) Système et procédé permettant de gérer l'intégrité dans une base de données répartie
KR101922044B1 (ko) 데이터베이스 이중화 무중단 정합성 복구 기법
WO2012045245A1 (fr) Procédé et système de maintien de cohérence de données
US11514000B2 (en) Data mesh parallel file system replication
JP6196389B2 (ja) 分散型ディザスタリカバリファイル同期サーバシステム
US20230409559A1 (en) Techniques for maintaining file consistency during file system cross-region replication
WO2024109253A1 (fr) Procédé et système de sauvegarde de données et dispositif
US11275657B2 (en) Method and system for minimizing rolling database recovery downtime
US10346299B1 (en) Reference tracking garbage collection for geographically distributed storage system
US11880385B1 (en) Ordering updates to secondary indexes using conditional operations
AU2011265370B2 (en) Metadata management for fixed content distributed data storage

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17830287

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17830287

Country of ref document: EP

Kind code of ref document: A1