CN118093273A - Transaction processing method, system and CDP replay and recovery method in redo record - Google Patents
Transaction processing method, system and CDP replay and recovery method in redo record Download PDFInfo
- Publication number
- CN118093273A CN118093273A CN202410487119.XA CN202410487119A CN118093273A CN 118093273 A CN118093273 A CN 118093273A CN 202410487119 A CN202410487119 A CN 202410487119A CN 118093273 A CN118093273 A CN 118093273A
- Authority
- CN
- China
- Prior art keywords
- transaction
- redo
- record
- data
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000011084 recovery Methods 0.000 title claims abstract description 29
- 238000003672 processing method Methods 0.000 title claims abstract description 7
- 239000004744 fabric Substances 0.000 claims abstract description 4
- 238000004806 packaging method and process Methods 0.000 claims description 26
- 230000008569 process Effects 0.000 claims description 9
- 238000002955 isolation Methods 0.000 abstract description 5
- 238000005538 encapsulation Methods 0.000 abstract description 3
- 230000004048 modification Effects 0.000 description 12
- 238000012986 modification Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a transaction processing method and system in a redo record and a CDP replay and recovery method, belonging to the technical field of disaster recovery backup. The method is used for the data backup end and comprises the following steps: acquiring identification parameters of the redo records; judging whether the redo record is a transaction ending point or not; several redo records belonging to the same transaction are encapsulated into one transaction structure. The system is used in a data backup terminal and comprises: a redo record identification parameter module is obtained; judging a transaction ending point module; the transaction fabric module is encapsulated. The invention realizes the display distinction and encapsulation of the transaction by acquiring the identification parameters of the redo records, ensures the consistency, the durability, the isolation and the atomicity of the data, lays a foundation for realizing the recovery of any transaction, and is safe and effective.
Description
Technical Field
The invention belongs to the technical field of disaster recovery backup, and relates to a transaction processing method and system in a redo record and a CDP replay and recovery method.
Background
MySQL is a widely used relational database management system used by many applications and websites to store and process large amounts of data. The MySQL database employs the concept of transactions to ensure consistency, persistence, isolation, and atomicity of data. The atomicity of transactions is one of the important features that ensures that data modifications generated by a transaction are either all dropped or all rolled back.
In MySQL, a Redo Log (redox Log) is a critical component used to record change operations in a database. The redo log is a persistent log file that records each modification made to the database. Its function is to recover incomplete transactions after a database crash or restart, and to ensure consistency and persistence of data. Each Redo log entry is referred to as a Redo Record (Redo Record) that contains the modification operations to the data. By utilizing the characteristic, the redo log records of the database can be captured at the production end in real time, the log records are analyzed at the backup end and data is replayed, and the data is recovered, so that Continuous Data Protection (CDP) of the database is realized, but in order to ensure the integrity and consistency of the database data, the atomicity of the transaction must be maintained in the process.
In MySQL transactions, atomicity means that during execution of the transaction, either all modification operations are successfully applied to the database (drop) or all modifications in the transaction are rolled back, and the database is not affected by it. Maintaining the atomicity of the transaction ensures the integrity and consistency of the database.
However, current MySQL redox Log does not explicitly distinguish between transactions, and thus cannot directly identify a particular transaction. This technical problem presents difficulties in implementing CDP backup restoration.
There is currently no prior art to address this problem. In order to implement CDP replay and recovery, a method is needed to explicitly identify and distinguish each transaction in MySQL redox Log, so that any transaction can be recovered. Meanwhile, any modification and expansion to MySQL Redo Log are required to be ensured under the premise of ensuring the consistency, durability, isolation and atomicity of data so as to avoid introducing new risks.
Disclosure of Invention
The present invention is directed to solving the above-mentioned technical problems in the background art, and provides a method and a system for processing transactions in a redo record and a method for playing back and recovering a CDP.
The technical scheme for solving the technical problems is as follows:
In a first aspect, a method for processing a transaction in a redo record is provided, which is used in a data backup terminal, and includes the steps of:
The method comprises the step of acquiring identification parameters, wherein the identification parameters of the redo records are acquired, and the identification parameters comprise: the type of the redo record, the table space number of the redo record, the real data that the redo record will write to the data page;
Judging a transaction end point, namely judging whether the redo records simultaneously meet the following three conditions: (1) the type of redo record is MLOG _2BYTE; (2) The tablespace number of the redo record points to the tablespace of the transaction retraction log; (3) The intra-page offset of the real data points to the state field in the transaction retraction page, the real data marks the state field as released, cleared or cached, if yes, the transaction end point is judged, and the next step is executed; if not, judging that the transaction is not the transaction ending point, and carrying out the next judgment;
and a step of packaging the transaction structure body, wherein a plurality of redo records belonging to the same transaction are packaged into one transaction structure body according to the transaction ending point.
In one embodiment, the encapsulating transaction structure step includes:
a step of creating a transaction structure body, wherein a redo record queue is arranged in the transaction structure body and used for storing the redo records;
Writing a redo record step, namely writing a redo record into a current redo record queue;
determining a redo record END, namely determining the redo record END to be written according to the transaction ending point;
Packaging the transaction structure body, namely packaging the transaction structure body after writing the redo record END, and recording a transaction number, a transaction starting block number and a transaction starting position offset in the transaction structure body;
And repeating the executing step, repeating the step of creating the transaction structure body to the step of packaging the transaction structure body until all the transaction structure bodies are packaged.
In one embodiment, before the step of obtaining the identification parameter, the method further includes:
a step of obtaining a redo block, which is to analyze a redo log and obtain the redo block;
and a step of obtaining the redo records, wherein the redo records are obtained from the redo blocks.
In a second aspect, a transaction processing system in a redo record is provided, which is used in a data backup end, and includes:
the redo record acquisition identification parameter module is used for acquiring identification parameters of the redo record, wherein the identification parameters comprise: the type of the redo record, the table space number of the redo record, the real data that the redo record will write to the data page;
The transaction ending point judging module is used for judging whether the redo records simultaneously meet the following three conditions: (1) the type of redo record is MLOG _2BYTE; (2) The tablespace number of the redo record points to the tablespace of the transaction retraction log; (3) The intra-page offset of the real data points to the state field in the transaction retraction page, the real data marks the state field as released, cleared or cached, if yes, the transaction end point is judged, and the next step is executed; if not, judging that the transaction is not the transaction ending point, and carrying out the next judgment;
And the packaging transaction structure body module is used for packaging a plurality of redo records belonging to the same transaction into one transaction structure body according to the transaction ending point.
In one embodiment, the encapsulated transaction fabric module includes:
The method comprises the steps of creating a transaction structure unit, wherein a redo record queue is arranged in the transaction structure unit and used for storing the redo record structure;
the writing and redoing recording unit is used for writing a redo record into the current redo record queue;
a unit for determining a redo record END, which is used for determining the redo record END to be written according to the transaction ending point;
A packaging transaction structure unit for packaging the transaction structure after writing the redo record END, and recording the transaction number, the transaction start block number and the transaction start position offset in the transaction structure;
And the repeated execution unit is used for repeatedly creating the transaction structure body unit to the packaging transaction structure body unit until the whole transaction structure body is packaged.
In one embodiment, before the obtaining the redo record identifying parameter module, the method further includes:
The redo block obtaining module is used for analyzing the redo log and obtaining a redo block;
and the redo record acquisition module is used for acquiring the redo record from the redo block.
In a third aspect, a CDP replay and recovery method is provided, and the transaction processing method in the redo record is used in the data backup end, and further includes the steps of:
A new data replay step of writing new data T 1 into ibd file M 0 of the backup side database through each transaction structure, and continuously updating ibd file M 0 to complete new data replay;
Generating CDP logs, namely generating a plurality of CDP logs in the new data replay process, wherein the CDP logs are in one-to-one correspondence with the changed data pages in the ibd file M 0, each CDP log consists of a plurality of historical data records R, and the structure of each historical data record R comprises: the recording head, the table space number, the page number and the historical data T 0, wherein the recording head records the transaction number, the first historical data record identifier, the historical data record length and the offset position of the last historical data record; wherein, the history data T 0 is generated in the process of writing the new data T 1 into the ibd file M 0 of the backup database;
Recovering the target ibd file according to the CDP log;
And recovering the target redo log file, wherein the target redo log file is recovered according to the CDP log.
In one embodiment, the new data playback step includes:
A step of obtaining a redo record, in which each redo record in the target transaction structure is obtained;
a step of locating the playback target data pages, wherein each playback target data page is located in the ibd file M 0 in sequence by the table space number and the page number of each redo record;
A data writing step of sequentially writing the history data T 0 in each playback-target data page into the CDP log, and writing the new data T 1 in each redo record into each playback-target data page;
and a second repeated execution step, namely repeatedly obtaining the redo recording step to the data writing step, continuously updating the ibd file M 0, and completing new data playback.
In one embodiment, the step of recovering the target ibd file includes:
Copying, namely copying ibd file M 0 to generate ibd file M 1;
A first pre-determining step, namely determining a target transaction number to be recovered, and determining a target historical data record R in a CDP log through the target transaction number;
Reading, namely starting reading from the tail of the CDP log one by one until a target historical data record R is read;
Positioning and recovering the target data page, namely positioning and recovering the target data page in the ibd file M 1 according to the table space number and the page number of the target historical data record R;
And a step of recovering the target ibd file, wherein the data of the recovery target data page is read into the memory, and then the historical data T 0 in the target historical data record R is written into the recovery target data page.
In one embodiment, the recovering the target redo log file step includes:
creating a redo Log1, namely creating a blank redo Log1;
A second pre-determining step of determining a target transaction number to be recovered;
a step of acquiring positioning information, namely acquiring a transaction start block number and a transaction start position offset of a target transaction in a CDP log through a target transaction number;
A step of locating a recovery target transaction, wherein the recovery target transaction position is determined on an original redo Log Log0 through a transaction starting block number and a transaction starting position offset;
And recovering the target redo Log file, namely writing the data before recovering the target transaction position on the redo Log Log0 into the redo Log Log1.
The beneficial effects of the invention are as follows:
(1) The invention acquires the identification parameters of the redo records and further confirms the mark of the transaction end, thereby laying a foundation for realizing the display and distinction of the transaction;
(2) According to the transaction ending mark, a plurality of redo records belonging to the same transaction are packaged, and then data replay and data recovery are operated by taking the transaction as a unit, so that a foundation is laid for realizing the recovery of any transaction;
(3) The invention identifies and encapsulates the transaction, and operates under the principle of ensuring the consistency, durability, isolation and atomicity of the data, thereby being safe and effective.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for processing transactions in a redo log in accordance with embodiment 1 of the present invention.
FIG. 2 is a diagram illustrating a system for processing transactions in a redo log in accordance with embodiment 2 of the present invention.
Fig. 3 is a schematic structural diagram of a packaging transaction structure module in embodiment 2 of the present invention.
Fig. 4 is a flowchart of a CDP playback and recovery method in embodiment 3 of the present invention.
FIG. 5 is a schematic diagram of the structure of CDP log in embodiment 3 of the present invention.
In the drawings, the list of components represented by the various numbers is as follows:
2001. Obtaining a redo block module; 2002. obtaining a redo recording module; 2003. a redo record identification parameter module is obtained; 2004. judging a transaction ending point module; 2005 encapsulates a transaction fabric module; 20051. creating a transaction structure unit; 20052. writing in a redo recording unit; 20053. determining a redo record END unit; 20054. packaging the transaction structure unit; 20055. the execution unit is repeated.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Example 1
As shown in fig. 1, a method for processing a transaction in a redo record is provided, which is used in a data backup end, and the method includes:
S101, analyzing the redo log to obtain a redo block.
It will be appreciated that the inside of the redo log is composed of fixed size blocks, and the first block in the redo log is the first file block of the log header for recording the version of the log and other information. The second and fourth blocks are checkpoint blocks for storing information about database checkpoints. Starting from the fifth block, is a redo block for recording the redo information of the database, including the redo record.
S102, obtaining the redo record from the redo block.
S103, acquiring identification parameters of the redo records, wherein the identification parameters comprise: the type of the redo record, the table space number of the redo record, and the real data that the redo record will write to the data page.
It will be appreciated that the redo record records a modification to a data page of a data file.
The structure of the redo record comprises: the type of the redo record, the table space number of the redo record, the page number of the redo record, the real data that the redo record will write to the data page.
It should be noted that, the tablespace number of the redo record indicates the tablespace number to which the data page to be modified in the current redo log belongs, and each tablespace number corresponds to a unique data file, i.e. the ibd file.
The page number of the redo record indicates the page number of the data page to be modified in the current redo log in the table space to which the data page belongs, and the offset position of the data page in the ibd file can be calculated according to the page number.
The format and length of the real data is not fixed and depends on the specific type of current redo log.
S104, judging whether the redo records simultaneously meet the following three conditions: (1) the type of redo record is MLOG _2BYTE; (2) The tablespace number of the redo record points to the tablespace of the transaction retraction log; (3) The intra-page offset of the real data points to the state field in the transaction retraction page, the real data marks the state field as released, cleared or cached, if yes, the transaction end point is judged, and the next step is executed; if not, judging that the transaction is not the transaction ending point, and carrying out the next judgment.
It is worth noting that a transaction is atomic, i.e., data modifications generated by a transaction are either all committed or all rolled back. During execution of a transaction, an undo record (undo record) is continually generated for rollback of the transaction.
Specifically, innoDB (storage engine) has a key component in the Mysql database, undo log (undo log). The role of the undo log is to provide the ability for the transaction to rollback and recover.
When a transaction is started, the revocation log applies for a revocation Segment (Undo Segment), and at least one revocation Page (Undo Page) is held in a revocation Segment, and a state field in the revocation Segment indicates a state of a current revocation Segment, such as being used, or waiting for reclamation.
During the execution of the transaction, the generated revocation record is written into the revocation page of the revocation segment, and the generation of the revocation record is accompanied by the generation of the corresponding redo record.
The modification of the state field marks the end of the transaction, and the modification of the state field also generates a corresponding redo record, so that we can determine the end point of the transaction by identifying the type of the redo record.
S105, according to the transaction end point, packaging a plurality of redo records belonging to the same transaction into a transaction structure body.
Further, the step S105 further includes:
S1051, creating a transaction structure body, wherein a redo record queue is arranged in the transaction structure body and used for storing the redo record structure body;
S1052, writing a piece of redo record into a current redo record queue;
s1053, determining a redo record END to be written according to the transaction END point;
S1054, after writing the redo record END, packaging the transaction structure body, and recording a transaction number, a transaction starting block number and a transaction starting position offset in the transaction structure body;
S1055, repeating the steps S1051 to S1054 until the whole transaction structure is packaged.
It should be noted that writing a redo record into a redo record structure until a redo record of the type mlo_multi_rec_endo is identified, writing of a redo record structure is completed, and then adding the redo record structure to a redo record queue and then starting the next redo record structure.
According to the technical scheme of the embodiment, the identification parameters of the redo records are obtained, the mark of the transaction end is further confirmed, and then a plurality of redo records belonging to the same transaction are packaged according to the transaction end mark, so that the display distinction of the transaction is realized, and a foundation is laid for the subsequent recovery of any transaction.
Example 2
As shown in fig. 2, in one embodiment, there is provided a system for in-record transaction processing for use in a data backup, the system comprising:
the redo block obtaining module 2001 is configured to parse the redo log and obtain a redo block;
the redo record acquisition module 2002 is used for acquiring a redo record from the redo block;
The redo record acquisition identification parameter module 2003 is configured to acquire identification parameters of the redo record, where the identification parameters include: the type of the redo record, the table space number of the redo record, the real data that the redo record will write to the data page;
The determine transaction end point module 2004 is configured to determine whether the redo records satisfy the following three conditions simultaneously: (1) the type of redo record is MLOG _2BYTE; (2) The tablespace number of the redo record points to the tablespace of the transaction retraction log; (3) The intra-page offset of the real data points to the state field in the transaction retraction page, and the real data marks the state field as released, cleared or cached, if yes, the transaction end point is determined, and the encapsulated transaction structure module 2005 is executed; if not, judging that the transaction is not the transaction ending point, and carrying out the next judgment;
The package transaction structure module 2005 is configured to package a plurality of redo records belonging to the same transaction into one transaction structure according to the transaction end point.
Further, as shown in fig. 3, the packaging transaction structure module 2005 includes:
Creating a transaction structure unit 20051, wherein the transaction structure unit is internally provided with a redo record queue, and the redo record queue is used for storing the redo record structure;
a write redo record unit 20052, configured to write a redo record into a current redo record queue;
a redo record determining END unit 20053, configured to determine, according to the transaction END point, a redo record END to be written;
A packaging transaction structure unit 20054 for packaging the transaction structure after writing the redo record END, and recording the transaction number, the transaction start block number and the transaction start position offset in the transaction structure;
The repeated execution unit 20055 is configured to repeatedly execute the creation transaction structure unit 20051 to the encapsulation transaction structure unit 20054 until the entire transaction structure is encapsulated.
In the technical solution of this embodiment, a redo block obtaining module 2001 is configured to parse a redo log and obtain a redo block; the redo record acquisition module 2002 is used for acquiring a redo record from the redo block; the redo record acquisition identification parameter module 2003 is configured to acquire identification parameters of the redo record, where the identification parameters include: the type of the redo record, the table space number of the redo record, the real data that the redo record will write to the data page; a determine transaction end point module 2004 for determining whether the redo record is a transaction end point; the package transaction structure module 2005 is configured to package a plurality of redo records belonging to the same transaction into one transaction structure according to the transaction end point. The transaction identification and encapsulation in the embodiment are operated under the principle of ensuring the consistency, the durability, the isolation and the atomicity of the data, and are safe and effective.
Example 3
As shown in fig. 4, a CDP replay and recovery method is provided, and the transaction processing method in the redo record described in embodiment 1 is used in the data backup side, and further includes the steps of:
S301, analyzing the redo log to obtain a redo block.
S302, obtaining the redo record from the redo block.
S303, acquiring identification parameters of the redo records, wherein the identification parameters comprise: the type of the redo record, the table space number of the redo record, and the real data that the redo record will write to the data page.
S304, judging whether the redo records simultaneously meet the following three conditions: (1) the type of redo record is MLOG _2BYTE; (2) The tablespace number of the redo record points to the tablespace of the transaction retraction log; (3) The intra-page offset of the real data points to the state field in the transaction retraction page, the real data marks the state field as released, cleared or cached, and if the intra-page offset is simultaneously satisfied, the transaction is judged to be an end point of the transaction, and the next step is executed; if the transaction end point is not satisfied, judging that the transaction end point is not the transaction end point, and carrying out the next judgment.
S305, according to the transaction end point, packaging a plurality of redo records belonging to the same transaction into a transaction structure body.
S306, writing new data T 1 into an ibd file M 0 of the backup side database through each transaction structure body, and continuously updating the ibd file M 0 to finish new data replay.
Further, the step S306 includes:
s3061, obtaining each redo record in a target transaction structure body;
s3062, sequentially positioning each replay target data page in the ibd file M 0 according to the table space number and the page number of each redo record;
S3063, sequentially writing historical data T 0 in each replay target data page into a CDP log, and then writing new data T 1 in each replay record into each replay target data page;
S3064 repeating the steps from S3061 to S3063, continuously updating the ibd file M 0, and completing new data playback.
It should be noted that, in step S3063, the history data T 0 is first read into the memory, and then the history data T 0 is written into the CDP log.
It should be further noted that, in the CDP implementation of MySql, the technical solution may involve ibd files in addition to the redo log. The ibd file is a binary file for storing table data and indexes of MySql. It contains the table's real data and the table's index structure, the table's index is organized using a b+ tree structure, and the table's data records are stored in leaf nodes.
In the leaf nodes of the b+ tree index, tables are stored that record the real data of the individual fields. Each leaf node maintains one or more table records, each record containing the value of each field in the table.
Through the table space number and the page number, corresponding leaf nodes can be positioned in the ibd file M 0 index tree, and then table record data in the leaf nodes can be read.
S307, generating a plurality of CDP logs in the new data replay process, wherein the CDP logs are in one-to-one correspondence with the changed data pages in the ibd file M 0, as shown in FIG. 5, each CDP log is composed of a plurality of historical data records R, and the structure of each historical data record R comprises: the recording head, the table space number, the page number and the historical data T 0, wherein the recording head records the transaction number, the first historical data record identifier, the historical data record length and the offset position of the last historical data record; the history data T 0 is generated during the process of writing the new data T 1 into the ibd file M 0 of the backup database.
Illustratively, the ibd file is organized to store data in pages (pages). The size of each page is typically 16KB. The modification of ibd file data pages is not performed in sequence, and thus, the CDP log needs to be in one-to-one correspondence with the changed data pages in ibd file M 0.
S308, recovering the target ibd file according to the CDP log.
Further, the step S308 includes:
S3081, copying ibd files M 0 to generate ibd files M 1;
S3082, determining a target transaction number to be recovered, and determining a target historical data record R in a CDP log through the target transaction number;
S3083, starting to read from the end of the CDP log one by one until a target historical data record R is read;
S3084, positioning and recovering a target data page in the ibd file M 1 according to the table space number and the page number of the target historical data record R;
S3085, reading the data of the recovery target data page into a memory, and writing the historical data T 0 in the target historical data record R into the recovery target data page.
In step S3081, ibd file M 0 and ibd file M 1 can be understood to be identical.
In addition, the technical scheme identifies each transaction by the transaction number when the transaction structure body is packaged, and each record of the CDP log also marks the transaction number to which the record belongs. Therefore, we can confirm the target historical data record R by the transaction number, whether the transaction number corresponding to the current data version or the transaction number corresponding to the previous data version.
S309, recovering the target redo log file according to the CDP log.
Further, the step S309 includes:
s3091, creating a blank redo Log Log1;
s3092, determining a target transaction number to be recovered;
S3093, acquiring a transaction start block number and a transaction start position offset of a target transaction from a CDP log through a target transaction number;
S3094, determining a recovery target transaction position on an original redo Log Log0 through a transaction start block number and a transaction start position offset;
s3095, writing data before recovering the target transaction position on the redo Log Log0 into the redo Log Log1.
According to the technical scheme, in the process of analyzing the redo log records, the ending point of each transaction is marked, the marked point is restored when the data is restored, the atomicity of the transaction is maintained, meanwhile, after the CDP log records the first synchronization, all historical data on the ibd file data page and the corresponding transaction numbers are recorded, then the newly generated ibd file and the redo log file are sent to the production end, the damaged file is replaced by the new file of the production end, and the data restoration can be effectively completed.
The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
The computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++, ruby, go and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (10)
1. A method for transaction processing in a redo log, for use in a data backup, comprising the steps of:
The method comprises the step of acquiring identification parameters, wherein the identification parameters of the redo records are acquired, and the identification parameters comprise: the type of the redo record, the table space number of the redo record, the real data that the redo record will write to the data page;
Judging a transaction end point, namely judging whether the redo records simultaneously meet the following three conditions: (1) the type of redo record is MLOG _2BYTE; (2) The tablespace number of the redo record points to the tablespace of the transaction retraction log; (3) The intra-page offset of the real data points to the state field in the transaction retraction page, the real data marks the state field as released, cleared or cached, if yes, the transaction end point is judged, and the next step is executed; if not, judging that the transaction is not the transaction ending point, and carrying out the next judgment;
and a step of packaging the transaction structure body, wherein a plurality of redo records belonging to the same transaction are packaged into one transaction structure body according to the transaction ending point.
2. The method of in-redo recording transaction processing according to claim 1, wherein said encapsulating transaction structure step comprises:
a step of creating a transaction structure body, wherein a redo record queue is arranged in the transaction structure body and used for storing the redo records;
Writing a redo record step, namely writing a redo record into a current redo record queue;
determining a redo record END, namely determining the redo record END to be written according to the transaction ending point;
Packaging the transaction structure body, namely packaging the transaction structure body after writing the redo record END, and recording a transaction number, a transaction starting block number and a transaction starting position offset in the transaction structure body;
And repeating the executing step, repeating the step of creating the transaction structure body to the step of packaging the transaction structure body until all the transaction structure bodies are packaged.
3. The method of claim 1, further comprising, prior to the step of obtaining the identification parameter:
a step of obtaining a redo block, which is to analyze a redo log and obtain the redo block;
and a step of obtaining the redo records, wherein the redo records are obtained from the redo blocks.
4. A transaction processing system in a redo log, for use in a data backup, comprising:
the redo record acquisition identification parameter module is used for acquiring identification parameters of the redo record, wherein the identification parameters comprise: the type of the redo record, the table space number of the redo record, the real data that the redo record will write to the data page;
The transaction ending point judging module is used for judging whether the redo records simultaneously meet the following three conditions: (1) the type of redo record is MLOG _2BYTE; (2) The tablespace number of the redo record points to the tablespace of the transaction retraction log; (3) The intra-page offset of the real data points to the state field in the transaction retraction page, the real data marks the state field as released, cleared or cached, if yes, the transaction end point is judged, and the next step is executed; if not, judging that the transaction is not the transaction ending point, and carrying out the next judgment;
And the packaging transaction structure body module is used for packaging a plurality of redo records belonging to the same transaction into one transaction structure body according to the transaction ending point.
5. The in-redo log transaction processing system of claim 4, wherein said encapsulated transaction fabric module comprises:
The method comprises the steps of creating a transaction structure unit, wherein a redo record queue is arranged in the transaction structure unit and used for storing the redo record structure;
the writing and redoing recording unit is used for writing a redo record into the current redo record queue;
a unit for determining a redo record END, which is used for determining the redo record END to be written according to the transaction ending point;
A packaging transaction structure unit for packaging the transaction structure after writing the redo record END, and recording the transaction number, the transaction start block number and the transaction start position offset in the transaction structure;
And the repeated execution unit is used for repeatedly creating the transaction structure body unit to the packaging transaction structure body unit until the whole transaction structure body is packaged.
6. The in-redo log transaction processing system of claim 4, further comprising, prior to said acquiring redo log identification parameters module:
The redo block obtaining module is used for analyzing the redo log and obtaining a redo block;
and the redo record acquisition module is used for acquiring the redo record from the redo block.
7. A CDP replay and recovery method, wherein the transaction processing method in the redo records according to any of claims 1 to 3 is used in the data backup side, and further comprising the steps of:
A new data replay step of writing new data T 1 into ibd file M 0 of the backup side database through each transaction structure, and continuously updating ibd file M 0 to complete new data replay;
Generating CDP logs, namely generating a plurality of CDP logs in the new data replay process, wherein the CDP logs are in one-to-one correspondence with the changed data pages in the ibd file M 0, each CDP log consists of a plurality of historical data records R, and the structure of each historical data record R comprises: the recording head, the table space number, the page number and the historical data T 0, wherein the recording head records the transaction number, the first historical data record identifier, the historical data record length and the offset position of the last historical data record; wherein, the history data T 0 is generated in the process of writing the new data T 1 into the ibd file M 0 of the backup database;
Recovering the target ibd file according to the CDP log;
And recovering the target redo log file, wherein the target redo log file is recovered according to the CDP log.
8. The CDP playback and recovery method of claim 7, wherein the new data playback step includes:
A step of obtaining a redo record, in which each redo record in the target transaction structure is obtained;
a step of locating the playback target data pages, wherein each playback target data page is located in the ibd file M 0 in sequence by the table space number and the page number of each redo record;
A data writing step of sequentially writing the history data T 0 in each playback-target data page into the CDP log, and writing the new data T 1 in each redo record into each playback-target data page;
and a second repeated execution step, namely repeatedly obtaining the redo recording step to the data writing step, continuously updating the ibd file M 0, and completing new data playback.
9. The CDP replay and restore method of claim 7, wherein said restoring a target ibd file step comprises:
Copying, namely copying ibd file M 0 to generate ibd file M 1;
A first pre-determining step, namely determining a target transaction number to be recovered, and determining a target historical data record R in a CDP log through the target transaction number;
Reading, namely starting reading from the tail of the CDP log one by one until a target historical data record R is read;
Positioning and recovering the target data page, namely positioning and recovering the target data page in the ibd file M 1 according to the table space number and the page number of the target historical data record R;
And a step of recovering the target ibd file, wherein the data of the recovery target data page is read into the memory, and then the historical data T 0 in the target historical data record R is written into the recovery target data page.
10. The CDP replay and recovery method of claim 7, wherein the recovery target redo log file step comprises:
creating a redo Log1, namely creating a blank redo Log1;
A second pre-determining step of determining a target transaction number to be recovered;
a step of acquiring positioning information, namely acquiring a transaction start block number and a transaction start position offset of a target transaction in a CDP log through a target transaction number;
A step of locating a recovery target transaction, wherein the recovery target transaction position is determined on an original redo Log Log0 through a transaction starting block number and a transaction starting position offset;
And recovering the target redo Log file, namely writing the data before recovering the target transaction position on the redo Log Log0 into the redo Log Log1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410487119.XA CN118093273B (en) | 2024-04-23 | 2024-04-23 | Transaction processing method, system and CDP replay and recovery method in redo record |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410487119.XA CN118093273B (en) | 2024-04-23 | 2024-04-23 | Transaction processing method, system and CDP replay and recovery method in redo record |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118093273A true CN118093273A (en) | 2024-05-28 |
| CN118093273B CN118093273B (en) | 2024-07-02 |
Family
ID=91160157
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410487119.XA Active CN118093273B (en) | 2024-04-23 | 2024-04-23 | Transaction processing method, system and CDP replay and recovery method in redo record |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118093273B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1784677A (en) * | 2004-03-31 | 2006-06-07 | 微软公司 | System and method for a consistency check of a database backup |
| CN108664359A (en) * | 2018-05-23 | 2018-10-16 | 上海达梦数据库有限公司 | A kind of database restoring method, device, equipment and storage medium |
| US20190102421A1 (en) * | 2017-09-29 | 2019-04-04 | Oracle International Corporation | Method and system for supporting data consistency on an active standby database after dml redirection to a primary database |
| CN113760846A (en) * | 2021-01-08 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
-
2024
- 2024-04-23 CN CN202410487119.XA patent/CN118093273B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1784677A (en) * | 2004-03-31 | 2006-06-07 | 微软公司 | System and method for a consistency check of a database backup |
| US20190102421A1 (en) * | 2017-09-29 | 2019-04-04 | Oracle International Corporation | Method and system for supporting data consistency on an active standby database after dml redirection to a primary database |
| CN108664359A (en) * | 2018-05-23 | 2018-10-16 | 上海达梦数据库有限公司 | A kind of database restoring method, device, equipment and storage medium |
| CN113760846A (en) * | 2021-01-08 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
Non-Patent Citations (2)
| Title |
|---|
| HU WAN 等: "Empirical study of redo and undo logging in persistent memory", 《2016 5TH NON-VOLATILE MEMORY SYSTEMS AND APPLICATIONS SYMPOSIUM (NVMSA)》, 18 August 2016 (2016-08-18), pages 1 - 6, XP032947906, DOI: 10.1109/NVMSA.2016.7547178 * |
| 张申勇 等: "Oracle数据库性能优化分析与配置", 《电脑知识与技术》, vol. 9, no. 18, 25 June 2013 (2013-06-25), pages 4146 - 4149 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118093273B (en) | 2024-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108664359B (en) | Database recovery method, device, equipment and storage medium | |
| US6665815B1 (en) | Physical incremental backup using snapshots | |
| CN104239443B (en) | A kind of storage method of serialized data operation log | |
| CN100498796C (en) | Logic log generation method, database backup/ restoration method and system | |
| CN110795287B (en) | Data recovery method, system, electronic equipment and computer storage medium | |
| CN110008129B (en) | Reliability test method, device and equipment for storage timing snapshot | |
| KR20110032343A (en) | An apparatus and method for logging optimization using non-volatile memory | |
| CN112463724B (en) | Data processing method and system for lightweight file system | |
| CN104937556A (en) | Recovering pages of database | |
| CN104199888A (en) | Data recovery method and device for resilient file system | |
| CN102024021A (en) | Method for logging metadata in logical file system | |
| CN103838645B (en) | Remote difference synthesis backup method based on Hash | |
| CN112925676A (en) | Method for realizing recovery of distributed database cluster at any time point based on WAL | |
| CN110222035A (en) | A kind of efficient fault-tolerance approach of database page based on exclusive or check and journal recovery | |
| CN118093273B (en) | Transaction processing method, system and CDP replay and recovery method in redo record | |
| CN102024052B (en) | Method and device for realizing transaction of data storage system | |
| KR101670473B1 (en) | Method for recovering deleted data by mysql innodb database | |
| CN104978241A (en) | Data recovery method and apparatus for COW type file system | |
| CN112000623A (en) | Metadata access method and device and computer readable storage medium | |
| KR100501414B1 (en) | Method of and apparatus for logging and restoring the meta data in file system | |
| CN109325005A (en) | A kind of data processing method and electronic equipment | |
| CN106599006A (en) | Data recovery method and device | |
| CN111984472B (en) | Data snapshot method, device and related equipment | |
| CN110209530B (en) | Method and system for recovering IO data of CDP system | |
| JPS62245348A (en) | Method and device for updating data base |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |