CN114756408A

CN114756408A - Metadata backup recovery method and device, electronic equipment and storage medium

Info

Publication number: CN114756408A
Application number: CN202210461427.6A
Authority: CN
Inventors: 赵伟; 张金冬; 吴夏; 王潮杰; 何革新; 盛欢; 刘良城; 张世伟; 黄远芳; 刘永松
Original assignee: Zetuo Technology Shenzhen Co ltd
Current assignee: Zetuo Technology Shenzhen Co ltd
Priority date: 2022-04-28
Filing date: 2022-04-28
Publication date: 2022-07-15

Abstract

The embodiment of the invention discloses a metadata backup recovery method, a device, electronic equipment and a storage medium, wherein the method is applied to a computing node in a distributed database management system and comprises the following steps: starting the computing node, wherein the computing node stores a metadata table and is used for recording the replication position of the computing node in a DDL log through coordinate data; and pulling the DDL log from a metadata node according to the coordinate data, executing a DDL command in the DDL log to update the local metadata of the computing node and update the coordinate data of the metadata table, and conveniently and accurately realizing the recovery and reconstruction of the computing node of the distributed database by using the local transaction processing capacity of the computing node.

Description

Metadata backup recovery method and device, electronic equipment and storage medium

Technical Field

The invention relates to the technical field of distributed database management systems, in particular to a metadata backup recovery method and device, electronic equipment and a storage medium.

Background

A Distributed DataBase Management System (DDBMS) typically includes three types of components, a compute node, a storage node, and a metadata node. The metadata node is responsible for storing metadata of the cluster, including the topology of the cluster, and metadata of user data for cluster management, such as user table definition statements, etc. The computing node is mainly responsible for the following work: receiving a client connection request, verifying the request and establishing a connection state; processing a query request sent by a client connection, and returning a result to the client; data reading and writing are completed by interaction with the storage nodes; a Distributed Transaction Coordinator (DTC) is implemented to complete the reliable commit of the distributed transaction.

When one or more computing nodes are abnormal and data of the computing nodes needs to be recovered, the processing data volume is large, the recovery efficiency is low or the operation is complex.

Disclosure of Invention

The invention mainly aims to provide a metadata backup recovery method, a metadata backup recovery device, electronic equipment and a storage medium, which can solve the problem of recovery and reconstruction of computing nodes of a distributed database.

To achieve the above object, a first aspect of the present application provides a metadata backup and recovery method, including:

starting the computing node, wherein the computing node stores a metadata table and is used for recording the replication position of the computing node in a DDL log through coordinate data;

and pulling the DDL log from a metadata node according to the coordinate data, executing DDL commands in the DDL log to update local metadata of the computing node, and updating the coordinate data of the metadata table.

Optionally, the coordinate data comprises a first coordinate, the first coordinate identifying a version number of local metadata of the compute node and a starting location for performing a recovery operation from the DDL log;

the pulling the DDL log from a metadata node according to the coordinate data comprises:

pulling, from the metadata node, a DDL log starting from the first coordinate.

Optionally, the executing the DDL command in the DDL log to update the local metadata of the compute node includes:

and updating the local metadata of the computing node according to the command type of each DDL command in the DDL log.

Optionally, the updating the coordinate data of the metadata table includes:

updating the coordinate data of the metadata table based on the replication location of the compute node in the DDL log in the compute node local transaction executing the DDL command such that the coordinate data in the metadata table always corresponds to the metadata stored in the compute node.

Optionally, the method further comprises:

and stopping data updating after the DDL log corresponding to the target time point is executed, and finishing the reconstruction recovery work.

Optionally, before starting the computing node, the method further comprises:

acquiring a full backup closest to a target time point;

and restoring the computing node by using the full backup.

Optionally, before the obtaining the full backup closest to the target time point, the method further includes:

and according to preset backup logic, performing full-scale logical backup or physical backup and incremental backup on the computing nodes, the storage clusters and the metadata clusters in the distributed database management system.

To achieve the above object, a second aspect of the present application provides a metadata backup restoring apparatus, including:

the system comprises a starting module, a storage module and a processing module, wherein the starting module is used for starting a computing node, and the computing node stores a metadata table and is used for recording the replication position of the computing node in a DDL log through coordinate data;

and the updating module is used for pulling the DDL log from a metadata node according to the coordinate data, executing DDL commands in the DDL log to update the local metadata of the computing node, and updating the coordinate data of the metadata table.

Optionally, the coordinate data includes a first coordinate, where the first coordinate is used to identify a version number of local metadata of the compute node and a starting location of a recovery operation performed from the DDL log;

the update module is specifically configured to:

pulling, from the metadata node, a DDL log starting from the first coordinate.

Optionally, the update module is further specifically configured to:

Optionally, the update module is specifically further configured to:

Optionally, the update module is further configured to:

Optionally, the metadata backup and restore apparatus further includes a restore module, configured to:

before starting the computing node, acquiring a full backup closest to a target time point;

and restoring the computing node by using the full backup.

A third aspect of the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps as described in the first aspect and any of its possible implementations.

To achieve the above object, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method according to the first aspect.

The application provides a metadata backup recovery method, which is applied to a computing node in a distributed database management system, wherein by starting the computing node, the computing node stores a metadata table and is used for recording the replication position of the computing node in a DDL log through coordinate data; and pulling the DDL log from a metadata node according to the coordinate data, executing a DDL command in the DDL log to update the local metadata of the computing node and update the coordinate data of the metadata table, and conveniently and accurately realizing the recovery and reconstruction of the computing node of the distributed database by using the local transaction processing capacity of the computing node.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Wherein:

fig. 1 is a schematic flowchart illustrating a metadata backup and recovery method according to an embodiment of the present disclosure;

fig. 2 is a schematic flowchart of another metadata backup and recovery method according to an embodiment of the present disclosure;

fig. 3 is a schematic structural diagram of a metadata backup and recovery apparatus according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

The embodiments of the present application will be described below with reference to the drawings.

Please refer to fig. 1, which is a flowchart illustrating a metadata backup and recovery method in an embodiment of the present application, where the method is applicable to a compute node in a distributed database management system. As shown in fig. 1, the method includes:

101. starting a computing node, wherein the computing node stores a metadata table and is used for recording the copying position of the computing node in a DDL log through coordinate data;

102. and pulling the DDL log from a metadata node according to the coordinate data, executing a DDL command in the DDL log to update the local metadata of the computing node, and updating the coordinate data of the metadata table.

The execution subject in this embodiment may be a metadata backup and recovery apparatus, and may be an electronic device, and in specific implementation, the electronic device may be a computing node terminal.

First, a brief description will be made of the DDBMS. A DDBMS generally includes compute nodes, storage nodes, and metadata nodes. Background it has been mentioned that a compute node is primarily responsible for the following:

receiving a client connection request, verifying the request and establishing a connection state;

processing a query request sent by a client connection, and returning a result to the client;

data reading and writing are completed by interaction with the storage nodes;

a Distributed Transaction Coordinator (DTC) is implemented to complete the reliable commit of the distributed transaction.

The metadata node is responsible for storing metadata of the cluster, including the topology of the cluster, and metadata of user data for cluster management, such as user table definition statements, etc.

And the storage nodes are responsible for storing data and executing and committing the distributed transaction branches under coordination of the compute nodes DTC. And forming a cluster based on the data change replication mechanism to realize a high availability mechanism.

User requests executed by the DDBMS are mainly classified into two categories, one is called DDL, that is, a statement defining a storage method (such as a table, a container, etc.) of data by a user; one type is called DML, that is, a statement for reading and writing data in an established storage unit (such as a table, a container, etc.).

In the embodiment of the application, the computing node stores the user metadata, so that a backup (backup) is needed. These user metadata are generated by executing DDL statements at the compute nodes. Meanwhile, when executing the DDL command, the computing node stores a DDL operation log in the metadata node and completes the transaction submission. When restoring a rebuild (restore) distributed database cluster, it is also necessary to restore several compute nodes using the backed up compute node data.

Specifically, when a DDBMS cluster needs to be restored, a metadata cluster and each storage cluster need to be restored in sequence, and finally each computing node needs to be restored. In the embodiment of the application, data recovery can be performed on the computing nodes based on the preset backup, and then the computing nodes are started.

Optionally, before starting the computing node, the method further includes:

according to preset backup logic, performing full-scale logic backup or physical backup and incremental backup on the computing nodes, the storage clusters and the metadata clusters in the distributed database management system;

acquiring a full backup closest to a target time point;

and restoring the computing node by using the full backup.

In the embodiment of the application, full-volume logical backup or physical backup and incremental backup (i.e. incremental continuous logical backup) can be performed on the computing nodes, the storage clusters and the metadata clusters of the DDBMS periodically, and the backup files are stored in a suitable storage service or storage device for standby.

Specifically, when a computing node is restored, a full backup (the backup time point of which is denoted as T0) closest to the target time point T1 may be found, and several computing nodes are restored by using the backup.

In an alternative embodiment, said coordinate data includes a first coordinate, said first coordinate is used for identifying a version number of local metadata of said compute node, and a starting position for performing a recovery operation from said DDL log;

the pulling the DDL log from the metadata node according to the coordinate data includes:

and pulling the DDL log starting from the first coordinate from the metadata node.

In an optional implementation manner, the executing the DDL command in the DDL log to update the local metadata of the computing node includes:

Specifically, in order to restore a computing node to a certain target time point T1, it is necessary to reconstruct a computing node using the full backup at the time point T0(T0 ═ T1 and (there is no other full backup of computing nodes in T0, T1)), and then sequentially execute the DDL logs stored in the metadata cluster in the section (T0, T1), so as to restore the computing node to the time point T1.

In order to obtain the DDL log at time T1, it is necessary to periodically perform full backup and continuous streaming incremental backup on the metadata cluster, so that the full metadata node backup data at time T0 and the incremental backup of [ T0, T1] can be used to obtain metadata such as the DDL log at time T1.

In the embodiment of the application, the computing node can pull the DDL log from the metadata node and sequentially execute each obtained DDL command to update the metadata of the computing node. The compute node may record its copy location in the DDL log, such as the command location to copy to at a time, via the coordinate data.

Specifically, in the DDL log (data table) of the metadata cluster, one column is a monotonically increasing integer, such as 1, 2, 3, which may be referred to as logID. Each DDL log has a unique identification logID. When a computing node executes a DDL command, a DDL log is appended (i.e., written to the end of the DDL log) to the DDL log table, and this data has a logID to uniquely identify the DDL command.

In the embodiment of the present application, the location or coordinate stored in DDL _ coord inside the computing node may be understood as logID, which means to which log in the DDL log table the computing node executes. When the computing node replays the DDL log, one DDL log is executed in each transaction and coordinates in DDL _ coord are updated by using the logID of the DDL log so as to reliably record the position of replay, and each time replay is continued, the position of the log ID +1 is started.

It is therefore possible to pull the DDL log from the metadata node according to the above-mentioned coordinate data (i.e., logID), execute the DDL command therein to update the local metadata of the compute node, and simultaneously update the coordinate data of the metadata table.

After the recovery computing node is started, the first coordinate DDL0 recorded in the metadata table DDL _ coord of the computing node is the version number (which can be understood as a time point) of the local metadata of the computing node, and is the position where the computing node should continue to replay the log from the DDL log. Further, the compute node may pull a DDL log starting from DDL0 from the metadata node, execute each obtained DDL command in turn, and execute the command by updating the compute node metadata according to each command type. There is no need to send DDL commands to the storage cluster, nor to send DDL logs to the metadata cluster again.

In an optional embodiment, the updating the coordinate data of the metadata table includes:

in the local transaction of the computing node executing the DDL command, updating the coordinate data of the metadata table based on the copy position of the computing node in the DDL log, so that the coordinate data in the metadata table always corresponds to the metadata stored in the computing node.

Specifically, in the local transaction of the computing node executing the DDL command, the coordinate data of the metadata table DDL _ coord is updated, so that after each DDL transaction is submitted, the coordinates in the metadata table DDL _ coord always completely correspond to the metadata actually stored in the computing node. If the recovery operation is interrupted for any reason, recovery can then continue from the interrupted position.

In an alternative embodiment, after step 102, the method further comprises:

Specifically, after the DDL journal corresponding to the target time point T1 is executed, the coordinates recorded in the metadata table DDL _ coord also reach DDL 1. Each DDL log is time stamped, recording the exact point in time to perform the DDL operation. This point in time corresponds to the cluster data version number recognized by all computing nodes. The stopping point of the recovery process can be found in the DDL log by the T1 time point. The recovery to this stopping point completes the recovery reconstruction work. At this point the metadata of the compute node is aligned to the same version as the user data of the storage node.

The metadata backup recovery method in the embodiment of the application is applied to a computing node in a distributed database management system, and by starting the computing node, the computing node stores a metadata table and is used for recording the replication position of the computing node in a DDL log through coordinate data; and pulling the DDL log from a metadata node according to the coordinate data, executing a DDL command in the DDL log to update the local metadata of the computing node and update the coordinate data of the metadata table, and conveniently and accurately realizing the recovery and reconstruction of the computing node of the distributed database by using the local transaction processing capacity of the computing node.

Please refer to fig. 2, which is a flowchart illustrating another metadata backup and recovery method according to an embodiment of the present application. As shown in FIG. 2, by way of example in recovery cluster to target time point T1, its metadata cluster and each storage cluster are recovered in turn, and finally each compute node is recovered.

Full backup at time T0(T0< ═ T1) and [ T0, T1] incremental data change backup are needed to restore their metadata clusters to time T1;

using the full backup of the computing node to generate the computing node at the time of T0 (at this time, the coordinate DDL0 is recorded in the DDL _ coord of the computing node as the version number (time point) of the local metadata of the computing node, and the version number is also the position for continuously replaying the log from the DDL log);

when the coordinate time point DDL0 recorded in the DDL _ coord is smaller than the time point T1, acquiring a DDL log started from DDL0 from the DDL log, sequentially executing each acquired DDL command, simultaneously starting a local transaction, and updating the coordinate data of the DDL _ coord so that the coordinate in the DDL _ coord is completely corresponding to the metadata actually stored in the computing node all the time; until the execution is stopped until the DDL log corresponding to the time point of T1 (at this time, the coordinate time point DDL1 recorded in DDL _ coord is the time point of T1).

In addition to kunlun databases, other distributed database systems in the industry currently do not implement local transactions in compute nodes, nor do they support storing metadata at compute nodes, nor metadata nodes, or storing DDL operation logs at metadata nodes for replaying (replay) DDLs. The method in the embodiment of the present application is to restore a reconstruction (restore) node to the latest state by using the DDL operation log. A metadata table DDL _ coord is maintained in the computing node to record the copied position of the computing node in the DDL log, namely which statement in the DDL log is executed to, and the position is also the starting point of the recovery process for starting to execute the DDL operation log.

In the embodiment of the application, the local transaction processing capability of the computing node is used, the fault-tolerant capability of a data recovery (restore) process is realized, and the data can be continuously recovered from an interrupt position no matter the recovery process is interrupted for any reason. Moreover, the operation log of the computing node metadata corresponding to the DDL is recorded by using the metadata table DDL _ coord, so that the one-to-one correspondence relationship between the computing node metadata and the DDL operation log is ensured, the recovery can be started from an accurate starting point in the recovery process, and the data consistency of the whole cluster is ensured. In addition, the DDL operation log is used for incremental recovery, and the local transaction log of the computing node is not required to be used for incremental recovery. The DDL operation log is used to accurately determine the cluster data version corresponding to the time point to be recovered, so that the recovery process can be accurately stopped to an accurate position and data version.

Based on the description of the above metadata backup recovery method embodiment, an embodiment of the present application further discloses a metadata backup recovery apparatus, please refer to fig. 3, where the metadata backup recovery apparatus 300 includes:

a starting module 310, configured to start a compute node, where the compute node stores a metadata table, and is configured to record, through coordinate data, a replication location of the compute node in a DDL log;

an update module 320, configured to pull the DDL log from a metadata node according to the coordinate data, execute a DDL command in the DDL log to update local metadata of the compute node, and update the coordinate data of the metadata table.

the update module 320 is specifically configured to:

pulling, from the metadata node, a DDL log starting from the first coordinate.

Optionally, the updating module 320 is further specifically configured to:

updating the coordinate data of the metadata table based on the replication location of the compute node in the DDL log in the compute node local transaction executing the DDL command so that the coordinate data in the metadata table always corresponds to the metadata stored in the compute node.

Optionally, the update module 320 is further configured to:

Optionally, before starting the computing node, the system may perform full-volume logical backup or physical backup, and incremental backup on the computing node, the storage cluster, and the metadata cluster in the distributed database management system according to a preset backup logic;

the metadata backup and restore apparatus 300 further includes a restore module 330 configured to:

acquiring a full backup closest to a target time point;

and restoring the computing node by using the full backup.

It can be understood that, relevant contents related to each module in fig. 3 have been described in detail in the foregoing method embodiment, and specific reference may be made to the contents in the method embodiment, which is not described herein again.

In the metadata backup and recovery apparatus 300 according to the embodiment of the present application, the metadata backup and recovery apparatus 300 may start the computing node, where the computing node stores a metadata table for recording a copy position of the computing node in the DDL log through coordinate data; and pulling the DDL log from a metadata node according to the coordinate data, executing a DDL command in the DDL log to update the local metadata of the computing node and update the coordinate data of the metadata table, and conveniently and accurately realizing the recovery and reconstruction of the computing node of the distributed database by using the local transaction processing capacity of the computing node.

In one embodiment of the application, an electronic device is also provided. Referring to fig. 4, fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 4, the electronic device 400 comprises a processor 401 and a memory 402, said memory 402 storing a computer program which, when executed by said processor 401, performs any of the steps of the above-described method embodiments as shown in fig. 1-2.

In an embodiment, a computer-readable storage medium is also proposed, which stores a computer program, which, when executed by a processor 401, causes the processor 401 to perform any of the steps of the above-described method embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

All possible combinations of the technical features in the above embodiments may not be described for the sake of brevity, but should be considered as being within the scope of the present disclosure as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A metadata backup recovery method applied to a computing node in a distributed database management system, the method comprising:

2. The metadata backup restoration method according to claim 1, wherein the coordinate data comprises a first coordinate, the first coordinate is used for identifying a version number of the local metadata of the computing node and a starting position for executing a restoration operation from the DDL log;

pulling, from the metadata node, a DDL log starting from the first coordinate.

3. The metadata backup restoration method according to claim 1, wherein the executing DDL commands in the DDL log to update the local metadata of the compute node comprises:

4. The metadata backup restoration method according to claim 1, wherein the updating the coordinate data of the metadata table includes:

5. The metadata backup restoration method according to claim 1, wherein the method further comprises:

and after the DDL log corresponding to the target time point is executed, stopping data updating and finishing the reconstruction recovery work.

6. The metadata backup restoration method according to claim 1, wherein prior to launching the compute node, the method further comprises:

acquiring a full backup closest to a target time point;

and restoring the computing node by using the full backup.

7. The metadata backup restoration method of claim 6, wherein prior to the obtaining of the full backup closest to the target point in time, the method further comprises:

8. A metadata backup restoration apparatus, comprising:

9. An electronic device, comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1-7.