CN113672679A - Method and system for restoring historical table structure - Google Patents

Method and system for restoring historical table structure Download PDF

Info

Publication number
CN113672679A
CN113672679A CN202110752372.XA CN202110752372A CN113672679A CN 113672679 A CN113672679 A CN 113672679A CN 202110752372 A CN202110752372 A CN 202110752372A CN 113672679 A CN113672679 A CN 113672679A
Authority
CN
China
Prior art keywords
database table
information
database
change operation
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110752372.XA
Other languages
Chinese (zh)
Inventor
李建辉
张文件
陈栋
罗春
李春
魏兴华
吴炎
臧冰凌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Woqu Technology Co ltd
Original Assignee
Hangzhou Woqu Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Woqu Technology Co ltd filed Critical Hangzhou Woqu Technology Co ltd
Priority to CN202110752372.XA priority Critical patent/CN113672679A/en
Publication of CN113672679A publication Critical patent/CN113672679A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2358Change logging, detection, and notification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/242Query formulation
    • G06F16/2433Query languages

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention relates to a database data replication technology, and discloses a method and a system for restoring a history table structure, which are applied to a data synchronization system, wherein the method comprises the following steps of; recording the structure information of a reference database table, and acquiring the information of all table structures in the database, wherein the structure information is the structure information of the reference database table; recording all database table change operation information; and restoring the structure information of the point database table at any time, and restoring the structure information of the point database table at any time according to the structure information of the reference database table and the change operation information of the database table. By the method, all the table change history operations DDL (SQL) before the moment and the data table structure information can be applied (applied) at any moment, and the complete table structure information at the specified moment can be restored.

Description

Method and system for restoring historical table structure
Technical Field
The invention relates to a database data replication technology, in particular to a method and a system for restoring a history table structure.
Background
In a database synchronization middleware (CDC) technology based on log parsing, a table structure Event (TableMap Event) in a log stream is obtained by receiving and parsing database log stream data, where the TableMap Event object only includes basic description information of a table, such as the number of columns, column precision, and storage format of subsequent data, but does not include key information such as column name, column code, and column type, so that for a subsequent complete parsing of real data, complete table structure information needs to be obtained in this step, and in an existing solution, technical means such as a reverse checking database are used to obtain complete table structure information, but in real service use, such a processing means cannot solve the following problems:
during the time from T0 to T1, the DDL operation of adding columns occurs in the table structure A, and when the binlog of the table in the T0 time period A is processed, the taken table structure is the mirror image of T1, and the situation of column mismatch occurs;
c1 columns are added and C2 columns are deleted from T0-T1, the total number of the columns taken at the time is consistent with that of T0, but the corresponding columns are staggered;
at T0 to T1, DDL of drop table occurs, and at this time, an abnormality occurs in which a table cannot be found when a table structure is held, and the processing of the entire log analysis is always blocked.
For example, patent names, methods, apparatus, systems, and media for tracking database DDL change operations; patent application No.: CN 201910485604.2; the application date is: 2019-06-03; in the patent: after receiving an SQL statement through a preset Structured Query Language (SQL) execution interface, detecting whether the SQL statement meets a preset execution condition; if the SQL statement meets the execution condition, acquiring a first DDL change statement corresponding to the SQL statement after the SQL statement is successfully executed; and acquiring user information of a user corresponding to the SQL statement, and storing the user information and the first DDL change statement into a preset database.
In the prior art, a binlog analysis technology is adopted to identify that a user executes DDL operation on a database, the execution is compared with an original (first DDL snapshot) table structure, and the situation that SQL change initiated by the user and the original table structure are different and used for database health monitoring and the like is tracked; the prior art can not quickly calculate the current table structure information and backtrack historical data or synchronize data with a large amount of time delay.
Disclosure of Invention
Aiming at the situation that in the prior art, a user executes DDL operation on a database through a binlog analysis technology, the execution is compared with an original (first DDL snapshot) table structure, and SQL change initiated by the user is tracked to be different from the original table structure and used for database health monitoring and other scenes; the prior art can not quickly calculate the current table structure information, has the defects of backtracking historical data or synchronizing data with a large amount of time delay, and provides a method and a system for restoring a historical table structure.
In order to solve the technical problem, the invention is solved by the following technical scheme:
a method for restoring a history table structure is applied to a data synchronization system, and the method comprises the following steps;
recording the structure information of a reference database table, and acquiring the information of all table structures in the database, wherein the information of all table structures is the structure information of the reference database table;
recording all database table change operation information;
and restoring the structure information of the point database table at any time, and restoring the structure information of the point database table at any time according to the structure information of the reference database table and the change operation information of the database table.
By the method, all the table change history operations DDL (SQL) before the moment and the data table structure information can be applied (applied) at any moment, and the complete table structure information at the specified moment can be restored
Preferably, the database table change operation information comprises an operation statement for changing the database table and a position pos of an operation in a database log.
Preferably, the method for recording all database table change operation information comprises the steps of capturing the database table change operation information and recording the information as D1; analyzing the database table change operation information, and analyzing the database table change operation information through an SQL analyzer; applying database table change operation information, and executing DDL operation on the database table change operation information analyzed by the SQL analyzer and the reference database table structure information; and outputting the analysis database table.
Preferably, the method for restoring the structure information of the point database table at any time comprises the steps of snapshot saving, wherein the output data analysis table is subjected to snapshot saving, and the stored time point is checkpoint; and determining the structure of the data table at any time point, and constructing the data table structure at any time point according to the database table structure information saved in the snapshot and an incremental DDL (distributed data storage) mode to obtain the data table structure at any time point.
Preferably, the data table structure at any time point is determined to be a data table structure at any time point constructed in a manner of adding increment DDL to database table structure information saved in a snapshot; the trace back point is to any time point Tx, and the corresponding table structure is Tx ═ Cm(checkpoint)+ Dn, where Tx is a table structure at any time; cm is the reference table structure generated by periodic checkpoint, Dn, all DDL records generated by the database table since the most recent Cm.
A system for restoring a historical table structure is applied to a data synchronization system and comprises a module for recording the structure information of a reference database table, a module for recording the change operation information of all database tables and a module for restoring the structure information of the database table of any time point; the recording reference database table structure information module acquires information of all table structures in a database, wherein the information of all table structures is the reference database table structure information;
the database table change operation information recording module is used for recording database table change operation information;
the system comprises a database table structure information restoring module, a database table structure information restoring module and a database table change operation information processing module, wherein the database table structure information restoring module restores the database table structure information of the point at any time according to the reference database table structure information and the database table change operation information.
Preferably, the database table change operation information module comprises an operation statement for changing the database table and a position pos of an operation in the database log.
Preferably, the module for recording all database table change operation information comprises a capture module, an analysis module and a change operation information module;
the capture module is used for acquiring database table change operation information and recording the information as D1; the analysis module is used for analyzing the database table change operation information and analyzing the database table change operation information through an SQL analyzer;
the change operation information module is used for applying database table change operation information and executing DDL operation on the database table change operation information analyzed by the SQL analyzer and the structure information of the reference database table; and outputting the analysis database table.
Preferably, the module for restoring the structure information of the point database table at any time comprises a snapshot storage module and a structure determination module;
the snapshot saving module is used for performing snapshot saving on the output data analysis table, and the time point of the snapshot saving is checkpoint;
the structure determining module is used for determining the structure of the data table at any time point, and constructing the data table structure at any time point according to the database table structure information stored in the snapshot and the incremental DDL mode to obtain the data table structure at any time point.
Preferably, the arbitrary time point data table structure determining module constructs a data table structure of an arbitrary time point in a manner of adding increment DDL to the database table structure information saved in the snapshot; the trace back point is to any time point Tx, and the corresponding table structure is Tx ═ Cm(checkpoint)+ Dn, where Tx is a table structure at any time; cm is the reference table structure generated by periodic checkpoint, Dn, all DDL records generated by the database table since the most recent Cm.
Due to the adoption of the technical scheme, the invention has the remarkable technical effects that:
the invention can restore the structure information module of the point database table at any time, and the structure information module of the point database table at any time restores the structure information of the point database table at any time according to the structure information of the reference database table and the change operation information of the database table.
The invention provides a method for solving the problem that when a table generates DDL at any time, the current table structure information can be quickly calculated, particularly when historical data is backtracked or binlog data is synchronized to have a large time delay, namely, the current table structure analyzed by the binlog cannot be restored by acquiring the current table structure, the SQL procedure is executed by depending on a latest historical snapshot record (checkpoint) and successive DDL statements and an SQL module (SQL analyzer) to simulate a database, and the application is performed in a memory to restore the table structure information at the specified time.
Drawings
FIG. 1 is a flow chart of example 1 of the present invention.
Fig. 2 is a system diagram of embodiment 2 of the present invention.
FIG. 3 is a schematic diagram of an embodiment 3 of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
Example 1
A method for restoring a history table structure is applied to a data synchronization system, and the method comprises the following steps;
recording the structure information of a reference database table, and acquiring the information of all table structures in the database, wherein the information of all table structures is the structure information of the reference database table;
recording all database table change operation information;
and restoring the structure information of the point database table at any time, and restoring the structure information of the point database table at any time according to the structure information of the reference database table and the change operation information of the database table.
By the method, all table change history operations DDL (SQL) before the moment and the data table structure information can be applied (applied) at any moment, and the complete table structure information database table change operation information at the specified moment is restored to include operation statements for changing the database table and the position pos of the operation in the database log.
The method for recording all database table change operation information comprises the steps of capturing the database table change operation information and recording the information as D1; analyzing the database table change operation information, and analyzing the database table change operation information through an SQL analyzer; applying database table change operation information, and executing DDL operation on the database table change operation information analyzed by the SQL analyzer and the reference database table structure information; and outputting the analysis database table.
The method for restoring the structure information of the point database table at any moment comprises the steps of snapshot saving, wherein the output data analysis table is subjected to snapshot saving, and the stored time point is checkpoint; and determining the structure of the data table at any time point, and constructing the data table structure at any time point according to the database table structure information saved in the snapshot and an incremental DDL (distributed data storage) mode to obtain the data table structure at any time point.
The data table structure at any time point is determined to be a data table structure at any time point in a mode of adding the incremental DDL to the database table structure information saved in the snapshot; the trace back point is to any time point Tx, and the corresponding table structure is Tx ═ Cm(checkpoint)+ Dn, where Tx is a table structure at any time; cm is the reference table structure generated by periodic checkpoint, Dn, the database since the most recent CmTable generated all DDL records.
Example 2
On the basis of the embodiment 1, the system for restoring the historical table structure based on the method for restoring the historical table structure is applied to a data synchronization system and comprises a module for recording the structure information of a reference database table, a module for recording the change operation information of all database tables and a module for restoring the structure information of a point database table at any time; the recording reference database table structure information module acquires information of all table structures in a database, wherein the information of all table structures is reference database table structure information;
the database table change operation information recording module is used for recording database table change operation information;
the system comprises a database table structure information restoring module, a database table structure information restoring module and a database table change operation information processing module, wherein the database table structure information restoring module restores the database table structure information of the point at any time according to the reference database table structure information and the database table change operation information.
The database table change operation information module comprises operation statements for changing the database table and a position pos where the operation is in the database log.
The database table change operation information recording module comprises a capturing module, an analyzing module and a change operation information module;
the capture module is used for acquiring database table change operation information and recording the information as D1; the analysis module is used for analyzing the database table change operation information and analyzing the database table change operation information through an SQL analyzer;
the change operation information module is used for applying database table change operation information and executing DDL operation on the database table change operation information analyzed by the SQL analyzer and the structure information of the reference database table; and outputting the analysis database table.
The module for restoring the structure information of the point database table at any time comprises a snapshot storage module and a structure determination module;
the snapshot saving module is used for performing snapshot saving on the output data analysis table, and the time point of the snapshot saving is checkpoint;
and the structure determining module is used for determining the structure of the data table at any time point, and constructing the data table structure at any time point according to the database table structure information stored in the snapshot and the incremental DDL mode to obtain the data table structure at any time point.
The arbitrary time point data table structure determining module constructs a data table structure of an arbitrary time point in a mode of adding the increment DDL to the database table structure information saved by the snapshot; the trace back point is to any time point Tx, and the corresponding table structure is Tx ═ Cm(checkpoint)+ Dn, where Tx is a table structure at any time; cm is the reference table structure generated by periodic checkpoint, Dn, all DDL records generated by the database table since the most recent Cm.
Example 3
On the basis of the above embodiment, it can be seen from the attached drawings that C0 is initialized checkpoint, and acquires all table structure information; d1 is a DDL in a database log stream, and has a tag with a timestamp T and is used for recording the precedence relationship between different D1/D2; generating checkpoint cm at regular time and saving corresponding checkpoint time stamps; if the user traces back the site to any time point Tx, the corresponding table structure is the combination of checkpoint + ddl increments.

Claims (10)

1. A method for restoring a history table structure is applied to a data synchronization system, and the method comprises the following steps;
recording the structure information of a reference database table, and acquiring all table structure information in the database, wherein the table structure information is the structure information of the reference database table;
recording all database table change operation information;
and restoring the structure information of the point database table at any time, and restoring the structure information of the point database table at any time according to the structure information of the reference database table and the change operation information of the database table.
2. The method of claim 1, wherein the database table change operation information includes an operation statement for changing the database table and a location pos where the operation is in the database log.
3. The method for restoring the historical table structure according to claim 1, wherein the method for recording all database table change operation information comprises;
capturing database table change operation information and recording the information as D1;
analyzing the database table change operation information, and analyzing the database table change operation information through an SQL analyzer;
applying database table change operation information, and executing DDL operation on the database table change operation information analyzed by the SQL analyzer and the reference database table structure information;
and outputting the analysis database table.
4. The method for restoring the historical table structure according to claim 1, wherein the method for restoring the table structure information of the point database at any time comprises the following steps;
snapshot saving, namely performing snapshot saving on the output data analysis table, wherein the stored time point is checkpoint;
and determining the structure of the data table at any time point, and constructing the data table structure at any time point according to the database table structure information saved in the snapshot and an incremental DDL (distributed data storage) mode to obtain the data table structure at any time point.
5. The method for restoring the historical table structure according to claim 4, wherein the data table structure at any time point is determined to construct the data table structure at any time point in a manner of adding the incremental DDL to the database table structure information saved in the snapshot; the trace back point is to any time point Tx, and the corresponding table structure is Tx ═ Cm(checkpoint)+ Dn, where Tx is a table structure at any time; cm is the reference table structure generated by periodic checkpoint, Dn, all DDL records generated by the database table since the most recent Cm.
6. A system for restoring a historical table structure is applied to a data synchronization system and is characterized by comprising a module for recording the structure information of a reference database table, a module for recording the change operation information of all database tables and a module for restoring the structure information of a data base table at any time;
the recording reference database table structure information module acquires information of all table structures in a database, wherein the information of all table structures is reference database table structure information;
the database table change operation information recording module is used for recording database table change operation information;
and the structure information module of the point database table at any time restores the structure information of the point database table at any time according to the structure information of the reference database table and the change operation information of the database table.
7. The system for restoring the historical table structure of claim 6, wherein the database table change operation information module comprises an operation statement for changing the database table and a location pos where the operation is in the database log.
8. The system for restoring a history table structure according to claim 1, wherein the module for recording all database table change operation information comprises a capture module, a parsing module and a change operation information module;
the capture module is used for acquiring database table change operation information and recording the information as D1;
the analysis module is used for analyzing the database table change operation information and analyzing the database table change operation information through an SQL analyzer;
the change operation information module is used for applying the database table change operation information and executing DDL operation on the database table change operation information analyzed by the SQL analyzer and the structure information of the reference database table; and outputting the analysis database table.
9. The system for restoring the historical table structure according to claim 1, wherein the module for restoring the database table structure information of the point data at any time comprises a snapshot saving module and a structure determining module;
the snapshot saving module is configured to perform snapshot saving on the output data analysis table, where a storage time point is checkpoint;
the structure determining module is used for determining the structure of the data table at any time point, and constructing the data table structure at any time point according to the database table structure information stored in the snapshot and the incremental DDL mode to obtain the data table structure at any time point.
10. The system for restoring a history table structure according to claim 9, wherein the arbitrary time point data table structure determining module constructs a data table structure at an arbitrary time point in a manner of adding an incremental DDL to database table structure information saved in a snapshot; the trace back point is to any time point Tx, and the corresponding table structure is Tx ═ Cm(checkpoint)+ Dn, where Tx is a table structure at any time; cm is the reference table structure generated by periodic checkpoint, Dn, all DDL records generated by the database table since the most recent Cm.
CN202110752372.XA 2021-07-02 2021-07-02 Method and system for restoring historical table structure Pending CN113672679A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110752372.XA CN113672679A (en) 2021-07-02 2021-07-02 Method and system for restoring historical table structure

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110752372.XA CN113672679A (en) 2021-07-02 2021-07-02 Method and system for restoring historical table structure

Publications (1)

Publication Number Publication Date
CN113672679A true CN113672679A (en) 2021-11-19

Family

ID=78538515

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110752372.XA Pending CN113672679A (en) 2021-07-02 2021-07-02 Method and system for restoring historical table structure

Country Status (1)

Country Link
CN (1) CN113672679A (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050086263A1 (en) * 2003-09-04 2005-04-21 Oracle International Corporation Self-managing performance statistics repository for databases
CN101183387A (en) * 2007-12-14 2008-05-21 沈阳东软软件股份有限公司 Increment data capturing method and system
US20080162590A1 (en) * 2007-01-03 2008-07-03 Oracle International Corporation Method and apparatus for data rollback
CN106326376A (en) * 2016-08-15 2017-01-11 东软集团股份有限公司 Method and device for copying information after table structure modification
CN109298978A (en) * 2018-11-14 2019-02-01 武汉烽火信息集成技术有限公司 A kind of restoration methods and system of the data-base cluster of designated position
CN109918386A (en) * 2019-01-31 2019-06-21 北京明略软件系统有限公司 A kind of data reconstruction method and device, computer readable storage medium
CN111611108A (en) * 2020-05-21 2020-09-01 云和恩墨(北京)信息技术有限公司 Method and device for restoring virtual database
CN111752929A (en) * 2020-07-01 2020-10-09 浪潮云信息技术股份公司 Method for realizing CockroachDB database snapshot
CN112966025A (en) * 2021-03-17 2021-06-15 焦点科技股份有限公司 Binlog log mining dictionary implementation method

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050086263A1 (en) * 2003-09-04 2005-04-21 Oracle International Corporation Self-managing performance statistics repository for databases
US20080162590A1 (en) * 2007-01-03 2008-07-03 Oracle International Corporation Method and apparatus for data rollback
CN101183387A (en) * 2007-12-14 2008-05-21 沈阳东软软件股份有限公司 Increment data capturing method and system
CN106326376A (en) * 2016-08-15 2017-01-11 东软集团股份有限公司 Method and device for copying information after table structure modification
CN109298978A (en) * 2018-11-14 2019-02-01 武汉烽火信息集成技术有限公司 A kind of restoration methods and system of the data-base cluster of designated position
CN109918386A (en) * 2019-01-31 2019-06-21 北京明略软件系统有限公司 A kind of data reconstruction method and device, computer readable storage medium
CN111611108A (en) * 2020-05-21 2020-09-01 云和恩墨(北京)信息技术有限公司 Method and device for restoring virtual database
CN111752929A (en) * 2020-07-01 2020-10-09 浪潮云信息技术股份公司 Method for realizing CockroachDB database snapshot
CN112966025A (en) * 2021-03-17 2021-06-15 焦点科技股份有限公司 Binlog log mining dictionary implementation method

Similar Documents

Publication Publication Date Title
CN109656934B (en) Source Oracle database DDL synchronization method and device based on log analysis
CN106656536B (en) Method and equipment for processing service calling information
CN100574225C (en) The automatic test approach of daily record and Auto-Test System
CN102184138B (en) Method and system for automatically reproducing and positioning software error
CN109634970B (en) Table data synchronization method, apparatus, storage medium and device
CN112559475B (en) Data real-time capturing and transmitting method and system
CN113360322B (en) Method and equipment for recovering data based on backup system
CN110795287A (en) Data recovery method, system, electronic device and computer storage medium
CN109240854B (en) Method and system for recovering data of change caused by misoperation
WO2019074687A1 (en) Method for replicating production behaviours in a development environment
KR20080084332A (en) Database migration apparatus and method
CN112966025B (en) Binlog log mining dictionary implementation method
CN113672679A (en) Method and system for restoring historical table structure
EP2037365A2 (en) Process flow analysis based on processing artifacts
CN107451056B (en) Method and device for monitoring interface test result
CN108241543B (en) Method, service server and system for executing service operation breakpoint
CN111198920B (en) Method and device for determining comparison table snapshot based on database synchronization
CN108900497B (en) Data synchronization method and system between heterogeneous systems
CN112363873A (en) Distributed consistent backup and recovery system and backup method thereof
CN112181992A (en) Method for realizing acquisition of oracle database incremental data
CN112286729B (en) Method for restoring specified time
US20130290385A1 (en) Durably recording events for performing file system operations
CN113392006A (en) Method and equipment for monitoring automatic test logs by using capsules
CN115408200A (en) Data backup method and device for multiple storage engines, electronic equipment and storage medium
US10922301B1 (en) Apparatus, computer program, and method for trigger-based tracking of database modifications

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination