CN112463447A - Optimization method for realizing physical backup based on distributed database - Google Patents
Optimization method for realizing physical backup based on distributed database Download PDFInfo
- Publication number
- CN112463447A CN112463447A CN202011336509.5A CN202011336509A CN112463447A CN 112463447 A CN112463447 A CN 112463447A CN 202011336509 A CN202011336509 A CN 202011336509A CN 112463447 A CN112463447 A CN 112463447A
- Authority
- CN
- China
- Prior art keywords
- backup
- snapshot
- file
- database
- scanning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1456—Hardware arrangements for backup
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2452—Query translation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/211—Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/253—Grammatical analysis; Style critique
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
Abstract
The invention relates to the technical field of database backup, and particularly provides an optimization method for realizing physical backup based on a distributed database, wherein S1, a user executes backup operation to backup the appointed database to appointed nodes, partitions and disk-dropping positions, the database service analyzes SQL statements to generate a syntax tree, and then an execution plan is generated through the syntax tree; s2, storing a snapshot corresponding to a given time stamp based on an execution plan, creating a backup process in the background, and scanning the snapshot; s3, scanning metadata in the snapshot to generate file description information, and generating a DUMP file according to the file description information; s4, scanning the data part in the snapshot, and generating an SST file by the start key, the end key and the data part; and S5, deleting the snapshot file after the backup is finished. Compared with the prior art, the method and the device well realize data backup and have good popularization value.
Description
Technical Field
The invention relates to the technical field of database backup, and particularly provides an optimization method for realizing physical backup based on a distributed database.
Background
According to the essential division of database backup, the method can be divided into logical backup and physical backup:
logical backup: refers to backing up data through logical exports. Exporting the user object in the database to a binary file, wherein the logical backup is to backup the sql statement, and the sql statement which is backed up is executed to realize the reproduction of the database data during the recovery.
Physical backup: the physical backup of the database is the backup of the physical objects of the database, and the physical objects of the database comprise physical data files, log files, configuration files and the like of the database. The database is backed up in a physical file copying mode, and physical backup can be divided into cold backup and hot backup.
Wherein, the hot backup is:
the hot backup is to backup the sql statements of the database operation under the condition that the database is running, and when the database has a problem, the sql statements of the backup can be executed again. The backup can be carried out in tablespace or at a data file level, the backup time is short, the database can still be used during the backup, the recovery is quick, and the recovery is carried out when the database still works under most conditions.
The prior art can not simultaneously realize hot backup, physical backup and backup in transactions for backup in a distributed system.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides an optimization method for realizing physical backup of a distributed database with strong practicability.
The technical scheme adopted by the invention for solving the technical problems is as follows:
an optimization method for realizing physical backup based on a distributed database comprises the following steps:
s1, the user executes backup operation, the designated database is backed up to the designated node, partition and disk-dropping position, the database service analyzes SQL statements to generate a syntax tree, and then an execution plan is generated through the syntax tree;
s2, storing a snapshot corresponding to a given time stamp based on an execution plan, creating a backup process in the background, and scanning the snapshot;
s3, scanning metadata in the snapshot to generate file description information, and generating a DUMP file according to the file description information;
s4, scanning the data part in the snapshot, and generating an SST file by the start key, the end key and the data part;
and S5, deleting the snapshot file after the backup is finished.
Further, in step S1, the method further includes:
s101, after a user inputs a backup statement, a parser generates a syntax tree from the statement through lexical analysis, syntactic analysis and semantic analysis;
s102, the optimizer generates a logic plan from the syntax tree, and then generates a physical plan for specific execution from the logic plan.
Further, in step S2, the method further includes:
s201, a Checkpoint file generated by a storage engine based on RocksDB (an embeddable and persistent key-value pair storage);
s202, after the snapshot file is successfully created, a background process for executing backup operation is created, and backup is carried out based on the current snapshot.
Preferably, in step S3, the metadata includes a database structure, schema, and view.
Further, in step S3, the method further includes:
s301, executing a plan according to the context information to generate primary export file description information;
s302, after generating the primary export file description information, forming complete export file description information together with the start time, the end time and the cluster parameter node information;
s303, generating the DUMP file according to the generated complete file description information, the file name and the file path.
Further, in step S4, the method further includes:
s401, scanning a snapshot, and acquiring all keys between a start key and an end key in a prefix to acquire all revisions between start time and end time;
s402, when scanning data, if an uncommitted transaction exists at present, backing up the previous version of the data related to the transaction;
and S403, generating the SST file according to the data obtained by scanning the snapshot.
Further, in step S5, the user sets the snapshot retention time by setting, and if not, it is a gc time of the database.
Furthermore, when the database is greatly modified, the snapshot is set to be preserved or stored permanently in a month.
Compared with the prior art, the optimization method for realizing physical backup based on the distributed database has the following outstanding beneficial effects:
(1) the invention can well realize the backup of data, perfectly integrates the advantages of hot backup, has short recovery time, good data integrity and consistency, minimum possibility of data loss and support the backup in transactions. The defects and shortcomings of large equipment investment, high communication cost, high communication environment requirement and complex operation management at ordinary times are greatly solved and optimized.
(2) Compared with a cold backup mode adopted by some databases, the technology provided by the invention backups based on a snapshot of a timestamp during backup, and processes run in the background, so that the transaction operation currently executed by the database is not influenced.
(3) Compared with a hot backup mode adopted by some databases, when the data is backed up by the technology, the last submitted version of the data is backed up for the data related to the transaction which is not submitted at the current timestamp, so that all the backed up data are submitted complete data, and the consistency of the data is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart diagram (I) of an optimization method for realizing physical backup based on a distributed database;
FIG. 2 is a flow chart diagram (II) of an optimization method for implementing physical backup based on a distributed database;
fig. 3 is a flow chart (iii) of an optimization method for implementing physical backup based on a distributed database.
Detailed Description
The present invention will be described in further detail with reference to specific embodiments in order to better understand the technical solutions of the present invention. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A preferred embodiment is given below:
an optimization method for realizing physical backup based on a distributed database comprises the following steps:
s1, the user executes the backup operation to backup the designated database (mode, table) to the designated node, partition and disk drop position. The database service analyzes the SQL statement to generate a syntax tree, and then generates an execution plan through the syntax tree.
The method comprises the following specific steps:
s101, after a user inputs a backup statement, a parser generates a syntax tree from the statement through lexical parsing, syntactic parsing and semantic parsing.
S102, the optimizer generates a logic plan from the syntax tree, and then generates a physical plan for specific execution from the logic plan.
As shown in fig. 1:
s2, storing the snapshot corresponding to the timestamp given by the execution plan, creating a backup process in the background, and scanning the snapshot.
The method comprises the following specific steps:
s201, a Checkpoint file generated by a storage engine based on RocksDB (an embeddable and persistent key-value pair storage).
S202, after the snapshot file is successfully created, a background process for executing backup operation is created, and backup is carried out based on the current snapshot. The process does not affect the execution of the current database transaction, and simultaneously ensures the consistency of data.
As shown in fig. 2:
s3, scanning the metadata in the snapshot to generate file description information, and generating the DUMP file according to the file description information.
The method comprises the following specific steps:
s301, executing the plan according to the context information, and generating preliminary export file description information.
S302, after generating the primary export file description information, forming complete export file description information together with the start time, the end time and the cluster parameter node information; specific data formats, such as:
and S303, finally generating the DUMP file according to the generated complete file description information, the file name and the file path.
As shown in fig. 3:
s4, scanning the data part in the snapshot, and generating the SST file by the start key, the end key and the data part.
The method comprises the following specific steps:
s401, scanning the snapshot, and acquiring all keys between the start key and the end key in the prefix for acquiring all revisions between the start time and the end time.
S402, when scanning data, if an uncommitted transaction exists currently, backing up the last version of the data related to the transaction. Specific data formats, such as:
and S403, generating an SST (ordered key value pair data) file according to the data obtained by scanning the snapshot.
And S5, deleting the snapshot file after the backup is finished.
The user can set the snapshot retention time through setting, and if not, the snapshot retention time is a gc time of the database; when the database is greatly modified, such as modifying the table structure, a longer retention time can be set or the snapshot can be permanently stored, so that the rollback operation can be conveniently carried out in the future.
The above embodiments are only specific cases of the present invention, and the protection scope of the present invention includes but is not limited to the above embodiments, and any suitable changes or substitutions that are made by a person of ordinary skill in the art and are in accordance with the claims of an optimization method for implementing physical backup based on distributed database according to the present invention shall fall within the protection scope of the present invention.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (8)
1. An optimization method for realizing physical backup based on a distributed database is characterized by comprising the following steps:
s1, the user executes backup operation, the designated database is backed up to the designated node, partition and disk-dropping position, the database service analyzes SQL statements to generate a syntax tree, and then an execution plan is generated through the syntax tree;
s2, storing a snapshot corresponding to a given time stamp based on an execution plan, creating a backup process in the background, and scanning the snapshot;
s3, scanning metadata in the snapshot to generate file description information, and generating a DUMP file according to the file description information;
s4, scanning the data part in the snapshot, and generating an SST file by the start key, the end key and the data part;
and S5, deleting the snapshot file after the backup is finished.
2. The method of claim 1, wherein in step S1, the method further comprises:
s101, after a user inputs a backup statement, a parser generates a syntax tree from the statement through lexical analysis, syntactic analysis and semantic analysis;
s102, the optimizer generates a logic plan from the syntax tree, and then generates a physical plan for specific execution from the logic plan.
3. The method for optimizing physical backup based on distributed database according to claim 1, wherein in step S2, the method further comprises:
s201, a Checkpoint file generated by a storage engine based on RocksDB;
s202, after the snapshot file is successfully created, a background process for executing backup operation is created, and backup is carried out based on the current snapshot.
4. The method of claim 1, wherein in step S3, the metadata includes database structure, schema and view.
5. The method of claim 4, wherein in step S3, the method further comprises:
s301, executing a plan according to the context information to generate primary export file description information;
s302, after generating the primary export file description information, forming complete export file description information together with the start time, the end time and the cluster parameter node information;
s303, generating the DUMP file according to the generated complete file description information, the file name and the file path.
6. The method for optimizing physical backup based on distributed database according to claim 1, wherein in step S4, the method further comprises:
s401, scanning a snapshot, and acquiring all keys between a start key and an end key in a prefix to acquire all revisions between start time and end time;
s402, when scanning data, if an uncommitted transaction exists at present, backing up the previous version of the data related to the transaction;
and S403, generating the SST file according to the data obtained by scanning the snapshot.
7. The method of claim 1, wherein in step S5, the user sets the snapshot retention time by setting, and if not, the snapshot retention time is a gc time of the database.
8. The method of claim 7, wherein the snapshot is set to be preserved monthly or permanently when the database is extensively modified.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011336509.5A CN112463447B (en) | 2020-11-25 | 2020-11-25 | Optimization method for realizing physical backup based on distributed database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011336509.5A CN112463447B (en) | 2020-11-25 | 2020-11-25 | Optimization method for realizing physical backup based on distributed database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112463447A true CN112463447A (en) | 2021-03-09 |
| CN112463447B CN112463447B (en) | 2022-12-23 |
Family
ID=74798353
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011336509.5A Active CN112463447B (en) | 2020-11-25 | 2020-11-25 | Optimization method for realizing physical backup based on distributed database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463447B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115599595A (en) * | 2022-08-26 | 2023-01-13 | 北京万里开源软件有限公司（Cn） | Physical backup method based on distributed database |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110082832A1 (en) * | 2009-10-05 | 2011-04-07 | Ramkumar Vadali | Parallelized backup and restore process and system |
| CN102012947A (en) * | 2010-12-16 | 2011-04-13 | 创新科存储技术有限公司 | Method and system for online backup of database |
| CN102722536A (en) * | 2012-05-22 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for realizing snapshot based on distributed file system |
| CN104239166A (en) * | 2014-09-11 | 2014-12-24 | 武汉噢易云计算有限公司 | Method for realizing file backup for running virtual machine |
| WO2016180160A1 (en) * | 2015-10-23 | 2016-11-17 | 中兴通讯股份有限公司 | Data snapshot recovery method and apparatus |
| CN107229540A (en) * | 2017-06-06 | 2017-10-03 | 贵州易鲸捷信息技术有限公司 | A kind of database restoring method and system based on time point |
| CN109086388A (en) * | 2018-07-26 | 2018-12-25 | 百度在线网络技术（北京）有限公司 | Block chain date storage method, device, equipment and medium |
| CN111782671A (en) * | 2020-07-01 | 2020-10-16 | 浪潮云信息技术股份公司 | Optimization method for realizing CDC (performance control) based on distributed database |
-
2020
- 2020-11-25 CN CN202011336509.5A patent/CN112463447B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110082832A1 (en) * | 2009-10-05 | 2011-04-07 | Ramkumar Vadali | Parallelized backup and restore process and system |
| CN102012947A (en) * | 2010-12-16 | 2011-04-13 | 创新科存储技术有限公司 | Method and system for online backup of database |
| CN102722536A (en) * | 2012-05-22 | 2012-10-10 | 中兴通讯股份有限公司 | Method and device for realizing snapshot based on distributed file system |
| CN104239166A (en) * | 2014-09-11 | 2014-12-24 | 武汉噢易云计算有限公司 | Method for realizing file backup for running virtual machine |
| WO2016180160A1 (en) * | 2015-10-23 | 2016-11-17 | 中兴通讯股份有限公司 | Data snapshot recovery method and apparatus |
| CN106610876A (en) * | 2015-10-23 | 2017-05-03 | 中兴通讯股份有限公司 | Method and device for recovering data snapshot |
| CN107229540A (en) * | 2017-06-06 | 2017-10-03 | 贵州易鲸捷信息技术有限公司 | A kind of database restoring method and system based on time point |
| CN109086388A (en) * | 2018-07-26 | 2018-12-25 | 百度在线网络技术（北京）有限公司 | Block chain date storage method, device, equipment and medium |
| CN111782671A (en) * | 2020-07-01 | 2020-10-16 | 浪潮云信息技术股份公司 | Optimization method for realizing CDC (performance control) based on distributed database |
Non-Patent Citations (1)
| Title |
|---|
| 温正湖: "MyRocks及其使用场景分析", 《微信公众号"老叶茶馆"》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115599595A (en) * | 2022-08-26 | 2023-01-13 | 北京万里开源软件有限公司（Cn） | Physical backup method based on distributed database |
| CN115599595B (en) * | 2022-08-26 | 2023-04-11 | 北京万里开源软件有限公司 | Physical backup method based on distributed database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112463447B (en) | 2022-12-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11429641B2 (en) | Copying data changes to a target database | |
| US10678808B2 (en) | Eager replication of uncommitted transactions | |
| US7133884B1 (en) | Unobtrusive point-in-time consistent copies | |
| EP3234780B1 (en) | Detecting lost writes | |
| CN109891402B (en) | Revocable and online mode switching | |
| US6453325B1 (en) | Method and means for backup and restoration of a database system linked to a system for filing data | |
| US9569514B2 (en) | Statement-level and procedural-level replication | |
| US8676752B2 (en) | Techniques for the log-based replication of high-level procedures | |
| CA2167790A1 (en) | Relational database system and method with high data availability during table data restructuring | |
| US8874519B1 (en) | Method and apparatus for restoring a table in a database | |
| WO2021139376A1 (en) | Method for indexing data in storage engines, and related device | |
| US10866865B1 (en) | Storage system journal entry redaction | |
| WO2022002103A1 (en) | Method for playing back log on data node, data node, and system | |
| CN109902127B (en) | Historical state data processing method and device, computer equipment and storage medium | |
| CN110597891A (en) | Device, system, method and storage medium for aggregating MySQL into PostgreSQL database | |
| CN112463447B (en) | Optimization method for realizing physical backup based on distributed database | |
| CN114925073A (en) | Distributed database architecture supporting flexible dynamic fragmentation and implementation method thereof | |
| WO2023015809A1 (en) | Method and device for optimizing distributed memory data query | |
| CN115658391A (en) | Backup recovery method of WAL mechanism based on QianBase MPP database | |
| CN112685230A (en) | Method for realizing backup and restoration of assigned time point by distributed database | |
| CN112612647A (en) | Log parallel replay method, device, equipment and storage medium | |
| CN113190379B (en) | Method, system and medium for reducing backup set of database backup | |
| CN111221801A (en) | Database migration method, system and related device | |
| CN115544173B (en) | Linearly expandable distributed database | |
| Zhou et al. | FoundationDB: A Distributed Key Value Store |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |