CN115617580B

CN115617580B - Incremental backup and recovery method and system based on Shared SST (SST) file

Info

Publication number: CN115617580B
Application number: CN202211612061.4A
Authority: CN
Inventors: 宋鹏威; 王玉珏; 叶小萌
Original assignee: Hangzhou Ouruozhi Technology Co ltd
Current assignee: Hangzhou Ouruozhi Technology Co ltd
Priority date: 2022-12-15
Filing date: 2022-12-15
Publication date: 2023-03-28
Anticipated expiration: 2042-12-15
Also published as: CN115617580A

Abstract

The application relates to an incremental backup recovery method based on a shared SST file, wherein the method comprises the following steps: acquiring local files to be executed in a database, determining a backup group corresponding to a backup task, acquiring an SST file set generated by previous backup in the backup group, judging whether each SST file in the local files is stored in the SST file set according to the unique identification name of the SST file, marking the SST file as an existing SST file if the SST file set is generated, marking the SST file as an newly added SST file if the SST file set is not generated, uploading a WAL file and the newly added SST file to the backup group, creating corresponding link files for each existing SST file, and uploading the link files to the backup group; and acquiring the network address of the backup group, and downloading all SST files and WAL files in the backup group according to the network address. By the method and the device, the characteristics of the LSM-Tree are fully utilized, the SST sharing mode is used, the playback process during recovery is omitted, and the problem of poor efficiency of incremental backup and recovery technology is solved, so that the recovery speed is increased.

Description

Incremental backup and recovery method and system based on Shared SST (SST) file

Technical Field

The present application relates to the field of database management, and in particular, to a method and a system for incremental backup and recovery based on a shared SST file.

Background

To ensure high availability of data, backup and restore functions are one of the most important, commonly used functions of a database. Databases are typically divided in implementation into a query layer and a storage layer, with backup and restore functions typically performed on the storage layer.

In the related art, a common incremental backup scheme is to perform incremental backup based on WAL (Write-Ahead Logging), that is, after data reaches a certain magnitude, only newly added data is backed up in order to increase backup speed.

However, with the above method, if the number of incremental backups is large, it takes a long time to restore data, and the pre-write log generated in the previous incremental backup is put back into the full backup data.

At present, no effective solution is provided for the problem of poor efficiency of incremental backup and recovery techniques in the related art.

Disclosure of Invention

The embodiment of the application provides a method, a system, computer equipment and a computer readable storage medium for putting in increment recovery based on a shared SST file, so as to at least solve the problem of poor efficiency of increment backup and recovery technology in the related technology.

In a first aspect, an embodiment of the present application provides a method for incremental backup recovery based on shared SST files, where the method includes:

the backup process comprises the following steps: obtaining a local file to be executed in a database, wherein the database adopts an LSM tree structure, the local file comprises a log before writing and a plurality of SST files,

determining a backup group corresponding to a backup task, acquiring an SST file set generated by backup of each time in the backup group, and

judging whether each SST file in the local files is stored in the SST file set or not according to the unique identification name of the SST file, if so, marking the SST file as an existing SST file, and if not, marking the SST file as a newly added SST file, and

uploading the pre-write log and the newly added SST file to the backup group, creating corresponding link files for each existing SST file, and uploading the link files to the backup group;

the recovery process comprises the following steps: and acquiring the network address of the backup group, and downloading all SST files and pre-write logs in the backup group according to the network address.

In some embodiments, downloading all SST files and pre-write logs in the backup group according to the network address comprises:

judging whether each SST file stored in the backup group is the newly added SST file or not according to the link file;

if yes, the newly added SST file is directly downloaded according to the network address,

and if not, acquiring a link file corresponding to the existing SST file, and downloading the existing SST file according to the link address recorded in the link file.

In some embodiments, after downloading all the SST files and the pre-write log in the backup group according to the link address, the method further comprises:

starting a storage engine based on the log before writing, the existing SST file and the newly added SST file;

after the storage engine is started, the database provides services to the outside.

In some of these embodiments, the method further comprises:

before the backup process is started, receiving a user-defined operation instruction of a user, and deleting redundant data in the log before writing according to the user-defined operation instruction;

and uploading the log before writing after the redundant data is deleted to the backup group in the backup process.

In some embodiments, after the booting the storage engine, the method further comprises:

and pulling the log before writing after the redundant file is deleted when the database is connected by the multi-copy application.

In some embodiments, the backup group includes a full backup and at least one incremental backup, and at any time t, the backup group includes a full backup and t incremental backups.

In a second aspect, an embodiment of the present application provides an incremental backup and restore system, where the system includes: a backup module and a restore module, wherein,

the backup module is used for acquiring a local file to be executed in a database, wherein the database adopts an LSM tree structure, the local file comprises a log before writing and a plurality of SST files,

and the recovery module is used for acquiring the network address of the backup group and downloading all SST files and pre-write logs in the backup group according to the network address.

In some embodiments, the downloading, by the recovery module, all SST files and pre-write logs in the backup group according to the network address includes:

In a third aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the method according to the first aspect is implemented.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the method according to the first aspect.

Compared with the related art, the incremental backup recovery method based on the shared SST file provided by the embodiment of the application makes full use of the characteristics of the LSM-Tree, adopts the mode of sharing the SST, omits the playback process during recovery, solves the problem of poor efficiency of incremental backup and recovery technology, and accordingly improves the recovery speed.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a schematic application environment diagram of an incremental backup restoration method based on a multiplexed SST file according to an embodiment of the present application;

FIG. 2 is a flow chart of a method of incremental backup according to an embodiment of the present application;

FIG. 3 is a schematic diagram of an LSM tree architecture according to an embodiment of the present application;

FIG. 4 is a block diagram of a shared SST file based incremental backup restoration system according to an embodiment of the present application;

fig. 5 is an internal structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.

It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. The use of the terms "a" and "an" and "the" and similar referents in the context of describing the invention (including a single reference) are to be construed in a non-limiting sense as indicating either the singular or the plural. The use of the terms "including," "comprising," "having," and any variations thereof herein, is meant to cover a non-exclusive inclusion; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to only those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

In this document, it is to be understood that the terminology referred to may be the technical means used to implement a part of the invention or other conclusive terminology. For example, the term may include:

incremental backup: only all data changes of the system after the last backup are backed up, so that the backup quantity is reduced;

WAL: write Ahead Log, for recording all change events;

LSM tree: log-Structured Merge Tree, a common storage architecture for distributed database inventory reservoirs;

SST: sorted String Table, the main data storage file in the LSM tree;

backup group: the method comprises one-time full backup and multiple incremental backups, wherein the incremental backup comprises an incremental backup 1 and an incremental backup 2 … … incremental backup n;

backup 0~t: the first t +1 backup sequences in the backup group comprise a full backup and t incremental backups, and the cluster data can be restored to the time point t according to the sequence.

Those skilled in the art know that: WAL is a common component in databases that is used to record all system changes. The incremental backup method of the existing database is mostly realized based on WAL, and the process of each incremental backup is as follows:

a backup process:

1. calculating the newly added WAL for the last backup;

2. uploading the WAL and the newly added file in the step 1 to a value backup group;

and (3) recovering:

1, downloading the WAL of the incremental backup of the current time and previous times;

2. downloading corresponding full-amount backup data Checkpoint;

3. the WALs in the past incremental backup are played back into Checkpoint one by one.

According to the above, it can be understood that, in the prior art, if the number of times of incremental backup is large, the "playback" time in step 3 is lengthened, and the database cannot provide services in the process, and can only operate normally after the playback is completed. Therefore, the method in the prior art has poor data recovery efficiency and influences the normal use of the database in the recovery process.

Based on the above problems in the prior art, the present application provides an incremental backup and recovery method based on a multiplexing SST file.

Fig. 1 is a schematic application environment of an incremental backup and recovery method based on a multiplexed SST file according to an embodiment of the present application, and as shown in fig. 1, a user may operate a database 11 through a client 10 to send a backup request or a recovery request to the database. After receiving the request, the database 11 responds to the request to perform the backup and restore process.

In the embodiment, the SST files are multiplexed among different backups, so that the WAL files are prevented from being played back in the recovery stage, and the recovery efficiency is improved. It should be noted that, in this embodiment, the client 10 may be a smart phone, a PC, a tablet computer, and the like, and the database 11 may be deployed on a cloud platform or in a cluster formed by multiple servers, and may be a distributed database formed by multiple service nodes.

Fig. 2 is a flowchart of an incremental backup method according to an embodiment of the present application, where the flowchart includes the following steps, as shown in fig. 2:

s201, local files to be executed in a database are obtained, wherein the database adopts an LSM tree structure, and the local files comprise a WAL file and a plurality of SST files;

the local file is a file to be backed up, and may be stored in one or more nodes of the database.

Further, it should be noted that the file is stored in the LSM tree structure in the form of SST, and in particular, fig. 3 is a schematic diagram of an architecture of an LSM tree according to an embodiment of the present application, as shown in fig. 3:

the LSM tree architecture mainly includes three data files: WAL, memTable, and SST.

MemTable is a data structure in memory that holds the most recently updated data. Data is temporarily stored in the memory, but the memory is not reliably stored, and the data is lost if power is off, so the reliability of the data is usually ensured by way of a Write-ahead log (WAL).

SST (ordered String Table), which is a data structure of LSM tree groups in disk, is stored hierarchically in memory. The technical scheme mainly relates to two files stored in a file system: WAL and SST.

S202, determining a backup group corresponding to a backup task, acquiring an SST file set which is backed up in the backup group for the previous time, and judging whether each SST file in the local files is located in the SST file set according to the unique identification name of the SST file, if so, marking the SST file as an existing SST file, and if not, marking the SST file as a newly added SST file;

if the local file needs to be backed up, a backup group corresponding to the file needs to be determined first. A database may have multiple backup sets simultaneously, each backup set being used to store different types of data, such as: the operation and maintenance data are stored in a backup group A, and the monitoring data are stored in a backup group B.

It can be understood that, because incremental backup is adopted, the data stored in the backup group is obtained through multiple backups, and before starting the backup task, the backup group needs to be traversed first to obtain the SST files stored in the backup group to form a set S;

further, the local file is compared with the backup group to judge the SST file in the local file relative to the newly added SST file combined with the S, specifically:

since in the LSM tree, the file name of each SST file cannot be changed once it is generated, and the file name of each SST file exists uniquely in the global. Therefore, the file name can be used as the unique identification name of each SST file, and then the SST files in the local file and the SST files in the set S are compared one by one according to the unique identification name,

if a certain SST file is in the set S, the SST file is already stored in the backup group, and in this embodiment, the SST file is marked as an existing file;

if a certain SST file is not in the set S, the SST file is not stored in the backup group, and the SST file is a file that needs to be continuously incrementally backed up.

S203, uploading the WAL file and the newly added SST file to a backup group, creating corresponding link files for each existing SST file, and uploading the link files to the backup group;

in this embodiment, for a newly added SST file, the file itself is directly transferred to a backup group; for an existing SST file in a backup group, a link file is created for it, where the link file may be, but is not limited to, various types of soft-connect files.

A real link address is recorded in the link file, and a real SST file can be downloaded according to the link address in the subsequent recovery process.

S204, acquiring the network address of the backup group, and downloading all SST files and WAL files in the backup group according to the link address.

The above steps S201 to S203 reflect the data backup process, and the step S204 reflects the data recovery process, including the following steps:

judging whether each SST file stored in the backup group is a newly added SST file;

specifically, since in the backup link (step S203), a link file is created for an existing SST file, and only the link file is saved in the backup group. Therefore, in this step, whether the "SST file" in the backup group is a link file can be determined according to whether the "SST file" in the backup group is a new SST file, and it can be understood that,

if the file is not the link file, the new SST file is directly downloaded according to the network address,

if yes, determining a link file corresponding to the existing SST file, and downloading the existing SST file according to the link connection address recorded in the link file.

Through the steps S201 to S204, compared with the method for performing incremental backup and recovery based on the WAL file in the prior art, in the present application, the file backup and recovery are performed by multiplexing the SST file among the past backups by using the unique and unalterable characteristic of the SST file name in the LSM tree. The backup and recovery processes can be completed without the 'playback' of the WAL file, so that when the backup times are more, the backup and recovery processes can be realized more quickly, and the interference to normal service is avoided.

It should be understood that, when the schemes of steps S201 to S204 are adopted, the existing SST file and the newly added SST file are dynamically changed according to the number of times of incremental backups. In the process of accumulative incremental backup, the 'newly added SST file' can be further changed into the 'existing SST file', and the SST file can be reused through the continuous dynamic transformation. However, regardless of the number of times of incremental backups, the technical solution of the present application can be flexibly adapted. In the incremental backup process, the incremental backup and recovery process can be realized without replaying WAL and uploading all SST files every time, so that the backup and recovery time is shortened, and the overall efficiency is improved.

In some embodiments, after all SST files and WAL files are downloaded, a storage engine is started based on the files, and a database provides services to the outside.

In some embodiments, because the WAL files generated by past backups need to be played back to the full backup data in the prior art, editing of the WAL data is prohibited, and the WAL cannot be deleted periodically before incremental backup;

in the embodiment of the present application, because the backup and recovery processes are not implemented based on the WAL file, part of the WAL file can be deleted according to the user requirement during the incremental backup.

Therefore, compared with the traditional mode, the uploaded WAL file is smaller, so that after the storage engine is restarted, when the WAL file is pulled up by multiple copies of application, the speed is higher, and the service effect can be further improved.

In some embodiments, fig. 4 is a block diagram of a system for incremental backup and restore based on shared SST files according to an embodiment of the present application, as shown in fig. 4, the system including: a backup module and a restore module, wherein,

the backup module is used for acquiring a local file to be executed in a database, wherein the database adopts an LSM tree structure, the local file comprises a WAL file and a plurality of SST files, a backup group corresponding to a backup task is determined, an SST file set generated by backup in a past time is acquired in the backup group,

and judging whether each SST file in the local files is stored in an SST file set or not according to the unique identification name of the SST file, if so, marking the SST file as the existing SST file, and if not, marking the SST file as the newly added SST file, and

uploading the WAL file and the newly added SST file to a backup group, creating corresponding link files for each existing SST file, and uploading the link files to the backup group;

and the recovery module is used for acquiring the network address of the backup group and downloading all SST files and WAL files in the backup group according to the network address.

In some embodiments, the downloading, by the recovery module, all SST files and WAL files in the backup group according to the network address includes:

judging whether each SST file in the backup group is a newly added SST file one by one;

if yes, the new SST file is directly downloaded according to the network address,

if not, determining a link file corresponding to the existing SST file, acquiring a connection address recorded in the link file, and downloading the existing SST file according to the connection address.

In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program when executed by a processor implements a method for incremental backup recovery based on shared SST files. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.

In an embodiment, fig. 5 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application, and as shown in fig. 5, there is provided an electronic device, which may be a server, and an internal structure diagram of which may be as shown in fig. 5. The electronic device comprises a processor, a network interface, an internal memory and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used for providing calculation and control capability, the network interface is used for communicating with an external terminal through network connection, the internal memory is used for providing an environment for an operating system and the running of a computer program, the computer program is executed by the processor to realize an incremental backup and recovery method based on shared SST files, and the database is used for storing data.

Those skilled in the art will appreciate that the configuration shown in fig. 5 is a block diagram of only a portion of the configuration associated with the present application, and does not constitute a limitation on the electronic device to which the present application is applied, and a particular electronic device may include more or less components than those shown in the drawings, or may combine certain components, or have a different arrangement of components.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by hardware instructions of a computer program, which may be stored in a non-volatile computer-readable storage medium, and when executed, the computer program may include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly and implicitly understood by one of ordinary skill in the art that the embodiments described herein may be combined with other embodiments without conflict.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not to be construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent application shall be subject to the appended claims.

Claims

1. An incremental backup restoration method based on a shared SST file, the method comprising:

determining a backup group corresponding to a backup task, and in the backup group, acquiring a set of SST files generated by the saved past backups, wherein the SST files are main data storage files in an LSM tree and are in the form of an ordered key value pair set, and

the recovery process comprises the following steps: acquiring a network address of the backup group, and downloading all SST files and pre-write logs in the backup group according to the network address;

after downloading all SST files and pre-write logs in the backup group according to the network address, the method further comprises: starting a storage engine based on the log before writing, the existing SST file and the newly added SST file, and after the storage engine is started, providing services for the database;

before the backup process is started, receiving a user-defined operation instruction of a user, deleting redundant data in the log before writing according to the user-defined operation instruction, uploading the log before writing after the redundant data is deleted to the backup group in the backup process, and pulling the log before writing after the redundant file is deleted when the plurality of copies of application are connected with the database after the storage engine is restarted.

2. The method of claim 1, wherein downloading all SST files and pre-write logs in the backup group based on the network address comprises:

3. The method of claim 1, wherein the backup group comprises a full backup and at least one incremental backup, and wherein at any time t, the full backup and t incremental backups are comprised in the backup group.

4. An incremental backup-restore system based on shared SST files, the system comprising: a backup module and a restore module, wherein,

the recovery module is used for acquiring the network address of the backup group and downloading all SST files and pre-write logs in the backup group according to the network address;

after downloading all SST files and pre-write logs in the backup group according to the network address, starting a storage engine based on the pre-write logs, the existing SST files and the newly added SST files, and after starting the storage engine, providing services for the outside by the database;

before a backup process is started, a user-defined operation instruction of a user is received, redundant data in the log before writing is deleted according to the user-defined operation instruction, the log before writing after the redundant data is deleted is uploaded to the backup group in the backup process, and the log before writing after the redundant file is deleted is pulled when a plurality of copies of the application are connected with the database after the storage engine is restarted.

5. The system of claim 4, wherein the recovery module downloads all SST files and pre-write logs in the backup group according to the network address, comprising:

6. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 3 when executing the computer program.

7. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 3.