CN110442560B - Log replay method, device, server and storage medium - Google Patents
Log replay method, device, server and storage medium Download PDFInfo
- Publication number
- CN110442560B CN110442560B CN201910748717.7A CN201910748717A CN110442560B CN 110442560 B CN110442560 B CN 110442560B CN 201910748717 A CN201910748717 A CN 201910748717A CN 110442560 B CN110442560 B CN 110442560B
- Authority
- CN
- China
- Prior art keywords
- log
- redo
- replay
- page
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The embodiment of the invention discloses a log replay method, a device, a server and a storage medium, wherein the method is applied to a main database and comprises the following steps: if the current database instance is determined to modify the target data page for the first time, generating a redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log; and sending the redo log to the standby database so that the standby database performs log replay according to the log type of the redo log. According to the technical scheme of the embodiment of the invention, when the redo log is generated by the main database, the redo log type of the new page synchronous redo log is added, so that the backup database does not need extra logic judgment, the redo logs except the page synchronous redo log can be processed in parallel according to the log type, and the replay efficiency and performance of the backup database are greatly improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of databases, in particular to a log replay method, a log replay device, a server and a storage medium.
Background
Relational database management systems typically use REDO logs to ensure the persistence of transactions, any data modification operation generates an REDO log inside the system, and ensures that modified data pages are written to disk only after the REDO log is flushed. The Log Sequence Number (LSN) is a big type value maintained inside the relational database management system to identify the REDO Log records, and each LSN value represents a set of a series of data modification operations inside the relational database management system.
In a single-node database (i.e. a database comprising one database instance), the LSN value has the characteristics of continuous increment and global uniqueness, when the REDO log is generated by data modification, the system allocates an LSN value, and the size of the LSN value represents the data modification sequence. When the system fault is restarted, the REDO logs are redone in sequence from small to large according to the LSN value, and the data can be restored to the state before the system fault according to the modification sequence of the data strictly. The shared storage database cluster can comprise a plurality of database instances, a user can be connected with any database instance to start a transaction, modify data and generate an REDO log, each database instance independently maintains log information, and LSN values in the database instances also have continuous increment characteristics and unique characteristics. In order to distinguish the sequence of modification, it is ensured that the LSN values generated when different database instances modify the same data page are increased progressively. Thus, the LSN value for each data instance in a shared storage database cluster is not necessarily continuous nor unique.
Data daemons are currently a relatively common solution for high availability of databases, and usually consist of one main database (i.e. the host of the database) and one or several backup databases (i.e. the backup of the database). And the secondary database maintains data synchronization with the primary database through the REDO log generated by the replay of the primary database. In the data daemon system, if the main database is a shared storage database cluster, the backup database receives REDO logs sent by a plurality of database instances, in order to ensure data consistency, the REDO logs must be redone from small to large according to the LSN value, and otherwise, the correctness of the data is damaged. In the prior art, a feasible method is to merge and sort multiple paths of REDO logs and replay the REDO logs sequentially after receiving the REDO logs of a main database, but the method has the following problems: because the REDO logs of multiple database instances in the main database are independent, after the REDO log generated by one data instance is received by the standby database, whether the REDO logs are generated by other database instances needs to be known definitely, otherwise, merging and sorting cannot be completed, and the standby database needs to merge and sort multiple paths of REDO logs and replay the REDO logs, so that the processing efficiency cannot meet the requirement, and the replay performance cannot be guaranteed.
Disclosure of Invention
The embodiment of the invention provides a log replay method, a log replay device, a server and a storage medium, which are used for optimizing a log replay scheme of a standby machine in a database and improving log replay efficiency.
In a first aspect, an embodiment of the present invention provides a log replaying method, applied to a master database, including:
if it is determined that the current database instance modifies the target data page for the first time, generating a redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log;
and sending the redo log to a standby database so that the standby database performs log replay according to the log type of the redo log.
In a second aspect, an embodiment of the present invention further provides a log replaying method, applied to a backup database, including:
acquiring a redo log sent by a main database;
and performing log replay on the redo log according to the log type of the redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log.
In a third aspect, an embodiment of the present invention further provides a log replaying apparatus, configured in a master database, and including:
the log generation module is used for generating a redo log if the current database instance is determined to modify the target data page for the first time, wherein the redo log comprises a page synchronization redo log and a conventional redo log;
and the log sending module is used for sending the redo log to a standby database so as to enable the standby database to replay the log according to the log type of the redo log.
Further, the log generation module includes a first modification determination unit, and the first modification determination unit is specifically configured to:
and if the target data page is determined to be disqualified, or the target data page is not disqualified, and the database instance corresponding to the last modification record in the global cache is different from the current database instance, determining that the current database instance modifies the target data page for the first time.
Further, the log generation module includes a page synchronization redo log unit, and the page synchronization redo log unit is specifically configured to:
determining a page log sequence number of the target data page before the first modification as a synchronization log sequence number, wherein the synchronization log sequence number is incremented and unique;
and generating the page synchronous redo log based on the synchronous log serial number and the database instance corresponding to the last modification record of the target data page before the first modification.
Further, the apparatus further comprises:
and the conventional log module is used for generating a conventional redo log if the current database instance is determined to modify the target data page for the non-first time.
In a fourth aspect, an embodiment of the present invention further provides a log replaying apparatus, configured in a backup database, including:
the log acquisition module is used for acquiring a redo log sent by the main database;
and the log replay module is used for replaying the redo logs according to the log types of the redo logs, wherein the redo logs comprise page synchronous redo logs and conventional redo logs.
Further, the log replay module comprises:
and the first replay unit is used for replaying the redo log according to a comparison result of the synchronous log serial number of the redo log and the replay completion log serial number if the redo log is the page synchronous redo log.
Further, the first replay unit is specifically configured to:
and if the replay completion log serial number is less than the synchronous log serial number of the replay log, executing a waiting operation until the replay completion log serial number is greater than or equal to the synchronous log serial number of the replay log.
Further, the log replay module comprises:
and the second replay unit is used for replaying the logs of the conventional redo logs in parallel if the redo logs are the conventional redo logs, wherein the number of the conventional redo logs is at least one.
In a fifth aspect, an embodiment of the present invention further provides a server, where the server includes:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the log replay method of the first aspect or the log replay method of the second aspect.
In a sixth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the log replay method according to the first aspect, or the log replay method according to the second aspect.
According to the embodiment of the invention, when the primary database determines that the current database instance modifies the target data page for the first time, the redo log is generated and sent to the standby database, so that the standby database performs log replay according to the log type of the redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log. According to the technical scheme of the embodiment of the invention, when the redo log is generated by the main database, the redo log type of the new page synchronous redo log is added, so that the backup database does not need extra logic judgment, the redo logs except the page synchronous redo log can be processed in parallel according to the log type, and the replay efficiency and performance of the backup database are greatly improved.
Drawings
Fig. 1 is a flowchart of a log replay method according to a first embodiment of the present invention;
FIG. 2 is a diagram illustrating a data daemon system according to an embodiment of the invention;
FIG. 3 is a flow chart of a data modification process according to a first embodiment of the present invention;
fig. 4 is a flowchart of a log replay method in the second embodiment of the present invention;
fig. 5 is a schematic structural diagram of a log replaying apparatus according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of a log replaying apparatus according to a fourth embodiment of the present invention;
fig. 7 is a schematic structural diagram of a server in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a log replay method in an embodiment of the present invention, where the embodiment is applicable to a case of performing log replay on redo logs in a database, and the method may be executed by a log replay device configured in a master database, and the device may be implemented in a software and/or hardware manner, for example, the device may be configured in a server.
Fig. 2 is a schematic diagram of a data daemon system according to a first embodiment of the present invention, where the data daemon system is only an example, and the number of the backup databases may be multiple. The data daemon system in the figure comprises a main database 101 and a standby database 102, wherein the main database 101 is a shared storage database cluster comprising two database instances, the main database 101 comprises a database instance 1 and a database instance 2, and each database instance independently generates and manages redo logs of the database instance. The primary database 101 may send the redo logs of the database instance 1 and the database instance 2 to the standby database 102, and the standby database 102 may replay the received redo logs to achieve data synchronization with the primary database 101.
In this embodiment, the data daemon system in fig. 2 is used to explain the log replay method.
As shown in fig. 1, the method may specifically include:
and S110, if the current database instance is determined to modify the target data page for the first time, generating a redo log.
The database instance is an instance of a data service provided by a relational database server for a user, and the user can operate internal data in the data service provided by the database instance. In this embodiment, the number of database instances included in the master database is not limited. A data page may be a basic unit of management storage space in a database instance. The REDO (REDO) log may include a page-synchronized REDO log, which is a newly added REDO log type, and a conventional REDO log, where the page-synchronized REDO log may be a log record of related information that was modified last time before the current modification was performed on the data page, and a serial number of the synchronized log in the page-synchronized REDO log is incremented and unique.
Further, determining that the current database instance modifies the target data page for the first time may include: and if the target data page is determined to be subjected to disc brushing, or the target data page is not subjected to disc brushing, and the database instance corresponding to the last modification record in the global cache is different from the current database instance, determining that the current database instance carries out first modification on the target data page. Wherein the flushing refers to storing the target data page into the magnetic disk.
The method comprises the steps of inquiring a disk refreshing record in a global cache to determine whether a target data page is refreshed or not, if the target data page is determined to be refreshed, determining that other database instances except a current database instance modify and refresh the target data page, and loading a latest target data page from a disk, so that the current database instance can be determined to be the first modification of the current latest target data page. Or, if it is determined that the target data page is not flushed, the modification record in the global cache may be queried. If the database instance corresponding to the last modification record of the target data page in the global cache is different from the current database instance, that is, the database instance corresponding to the last modification of the target data page before the current database instance is modified is not the current database instance, it may be determined that the target data page is modified by other database instances other than the current database instance, and it is necessary to obtain the latest target data page from the database instance corresponding to the modification record, so that it may be determined that the current database instance is the first modification of the latest target data page. For example, database instance 1 modifies data page p1 and does not yet have to be flushed, the database instance corresponding to the last modified record (the last modified record) recorded in the global cache is database instance 1, and for database instances other than database instance 1, modifying data page p1 again is the first modification.
Further, generating the page synchronization redo log may include S111-S112 (not shown in the figure), specifically:
s111, determining a page log sequence number of the target data page before first modification as a synchronous log sequence number.
The page synchronous redo log is a newly added redo log type, the serial number of the synchronous log is increased progressively and is unique, and the position in the log file can be before the position of the conventional redo log. The content recorded in the page synchronization redo log may include an Identifier (ID) of a table space to which the target data page belongs, a file Identifier (ID) to which the target data page belongs, a page number of the target data page, a database instance number to which the target data page is located most recently, and a synchronization log sequence number of the target data page, where the identifier of the table space to which the target data page belongs may be represented by ts _ ID, the file identifier to which the target data page belongs may be represented by file _ ID, the page number of the target data page may be represented by page _ no, the database instance number to which the target data page is located most recently may be represented by ep _ se, and the synchronization log sequence number of the target data page may be represented by page _ lsn.
Before the current database instance modifies the target data page for the first time, the page log serial number recorded on the header of the target data page can be acquired, and the page log serial number is determined as the synchronous log serial number after the first modification.
And S112, generating a page synchronization redo log based on the synchronization log sequence number and the database instance corresponding to the last modification record of the target data page before the first modification.
After the sequence number of the synchronous log is determined, if the target data page is determined to be flushed, it is not known which database instance of the last modified target data page is, and at this time, page synchronous redo logs need to be generated for all database instances except the current database instance, that is, multiple page synchronous redo logs are generated. If the target data page is determined not to be overwritten, the database instance number of the database instance corresponding to the last modification record before the current database instance is modified for the first time can be determined through the modification record of the target data page in the global cache, the database instance number is the latest database instance number of the target data page, and only one page synchronous redo log is generated at the moment.
Furthermore, the identifier of the table space to which the target data page belongs, the identifier of the file to which the target data page belongs, and the page number of the target data page need to be determined. And generating a page synchronous redo log based on the synchronous log serial number, the latest database instance number of the target data page, the identifier of the table space of the target data page, the identifier of the file of the target data page and the page number of the target data page.
In addition, if the current database instance is determined to modify the target data page for the non-first time, a conventional redo log is generated. When the current database instance modifies the target data page for the Nth time, the generated redo log only comprises a conventional redo log, wherein N is an integer greater than or equal to 2.
And S120, sending the redo log to the standby database so that the standby database replays the log according to the log type of the redo log.
Specifically, after the redo log is generated by the master database, the redo log may be sent to the standby database, so that the standby database performs log replay according to the log type of the redo log. If the redo log is a page synchronous redo log, performing log replay on the current database instance after the database instance where the latest target data page recorded in the page synchronous redo log is located is replayed; and if the redo log is the conventional redo log, directly replaying the log through the parallel thread. The conventional redo logs can be multiple in number, and the replay performance is guaranteed through parallel processing.
In addition, in this embodiment, the page log serial number recorded on the data page header is used, and the dynamic modification system has assigned the maximum log serial number, so that it can be ensured that each database instance generates a synchronization log serial number in the page synchronization redo log when modifying the same data page, and the synchronization log serial number has incremental and unique characteristics. Therefore, for the page-synchronized redo log, log replay can be performed in the order of the synchronized log sequence numbers.
Fig. 3 is a flowchart of a data modification process in a first embodiment of the present invention, and referring to fig. 3, a process of modifying a target data page by a current database instance in a master database is as follows: and S11, initializing a local log serial number. Initializing the local log sequence number in the current database instance to 0, which may be denoted by PTX _ LSN. And S12, acquiring the modification authority of the target data page. And S13, acquiring the latest target data page. The latest target data page is loaded from other instances or disks through a cache fusion mechanism. And S14, modifying the target data page and generating a redo log. The current database instance modifies the target data page and generates a page-synchronized redo log and a conventional redo log. S15, judging whether the page log serial number is larger than the local log serial number. Reading a PAGE log sequence number recorded at the head of the target data PAGE before modification, wherein the PAGE log sequence number can be represented by Page _ LSN, if Page _ LSN > PTX _ LSN, the target data PAGE is modified by other database instances, and executing S16, otherwise executing S17. And S16, modifying the sequence number of the local log. PTX _ LSN is set to PAGE _ LSN + 1. And S17, whether the modification is finished. If PAGE _ LSN is less than or equal to PTX _ LSN, it indicates that no other database instance has modified the target data PAGE, and determines whether the modification is finished, if so, S18 is executed, otherwise, S12 is returned to. And S18, judging whether the sequence number of the system distribution log is smaller than the sequence number of the local log. Comparing the system allocation log sequence number with the local log sequence number, wherein the system allocation log sequence number can be represented by CUR _ LSN, if the CUR _ LSN is less than PTX _ LSN, which indicates that other database instances modify the target data page, S19 is executed, otherwise, if the CUR _ LSN is more than or equal to PTX _ LSN, which indicates that the target data page has not been modified so far after being modified last time, S20 is executed. And S19, modifying the system distribution log sequence number. CUR _ LSN is set to PTX _ LSN. S20, modifying the local log sequence number and the system allocation log sequence number. The modified PTX _ LSN is CUR _ LSN +1, the new CUR _ LSN is itself incremented by one. And S21, determining the page log sequence number. The PAGE log sequence number PAGE _ LSN is equal to PTX _ LSN. And S22, releasing the modification authority of the target data page.
Before modifying data pages, a database instance ensures that modification authority of the data pages is obtained first, the latest data pages are obtained, page synchronous redo logs of the database instance are generated after the data pages are modified, page log serial numbers recorded on the heads of the data pages are read and saved, and if the modification relates to a plurality of data pages, the maximum page log serial numbers are saved. And after the data page is modified, adjusting the serial number of the currently distributed maximum system distribution log of the system according to the maximum page log serial number collected by the current modification so as to determine the page synchronous redo log serial number of the page synchronous redo log.
Illustratively, if the master database includes three database instances: database example 1, database example 2, and database example 3, CUR _ LSN 1000 in database example 1, CUR _ LSN 2000 in database example 2, and CUR _ LSN 3000 in database example 3. Database instance 1 modifies data PAGE P1(P1 first modified, loaded from disk, PAGE _ LSN 800), initially sets PTX _ LSN 0, reads out data PAGE, sets PTX _ LSN 801, completes modification of data PAGE P1, modifies CUR _ LSN 1001, PTX _ LSN 1001, modifies PAGE _ LSN 1001, and generates two PAGE synchronization redo logs, respectively (database instance 2, 800), (database instance 3, 800). Database instance 3 continues to modify data PAGE P1, reads PAGE _ LSN 1001 from database instance 1, sets PTX _ LSN 1002, and upon completion of the modification of data PAGE P1, modifies CUR _ LSN 3001, PTX _ LSN 3001, and modifies PAGE _ LSN 3001. And generates a page-synchronized redo log as (database instance 1, 1001). Database instance 2 continues to modify data PAGE P1, reads PAGE _ LSN 3001 from database instance 3, sets PTX _ LSN 3002, after modification of data PAGE P1 is complete, finds PTX _ LSN > CUR _ LSN, modifies CUR _ LSN 3002, modifies PAGE _ LSN 3002, and generates a sync redo log as (database instance 3, 3001).
According to the technical scheme of the embodiment, when the main database determines that the current database instance modifies the target data page for the first time, the redo log is generated, and the redo log is sent to the standby database, so that the standby database performs log replay according to the log type of the redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log. In this embodiment, when the redo log is generated by the master database, a new redo log type of the page synchronization redo log is added, so that the backup database does not need extra logic judgment, and redo logs other than the page synchronization redo log can be processed in parallel according to the log type, thereby greatly improving the replay efficiency and performance of the backup database.
Example two
Fig. 4 is a flowchart of a log replay method in the second embodiment of the present invention. The present embodiment may be applicable to the case of performing log replay on redo logs in a database, and the method may be executed by a log replay apparatus configured in the database, where the apparatus may be implemented in a software and/or hardware manner, for example, the apparatus may be configured in a server. As shown in fig. 4, the method of this embodiment specifically includes:
s210, obtaining the redo log sent by the main database.
The redo log may include a page synchronization redo log and a regular redo log, among others.
S220, judging whether the redo log is a page synchronous redo log or not.
And performing log replay on the redo log according to the log type of the redo log, determining the log type of the redo log, namely determining whether the redo log is a page-synchronous redo log according to the log content, executing S230 if the redo log is the page-synchronous redo log, and executing S240 if the redo log is a conventional redo log.
And S230, performing log replay on the redo log according to the comparison result of the synchronous log serial number of the redo log and the replay completion log serial number.
When log replay is performed on the standby database, a replay task system can be distributed to each main database instance, and the replay task system can record replay conditions of logs of corresponding database instances. The replay completion log serial number is the latest log serial number of the replay completed log stored by the replay task system for the corresponding database instance, and may be specifically represented by APPLY _ LSN [ ep _ seqno ], where the replay completion log serial number meaning the database instance ep _ seqno is APPLY _ LSN.
And if the serial number of the replay completion log is less than the serial number of the synchronous log of the replay log, executing a waiting operation until the serial number of the replay completion log is greater than or equal to the serial number of the synchronous log of the replay log. If the database processes the page synchronous redo log in the redo log, whether the serial number of the redo log is greater than or equal to the serial number of the redo log or not can be judged, if the serial number of the redo log is less than the serial number of the redo log, the fact that the redo log with the log serial number equal to the serial number of the synchronous log exists in the database instance ep _ seqno is shown, the current replay session can wait for the replay thread of the database instance ep _ seqno to replay the log until the replay is completed and the log serial number is greater than or equal to the serial number of the redo log, and the replay is continued. If the replay completion log sequence number is greater than or equal to the synchronous log sequence number of the replay log, it indicates that the database instance ep _ seqno has replayed the replay log with the log sequence number equal to the synchronous log sequence number, and does not need to wait.
Illustratively, if the database instance 2 generates the page synchronous redo log of the database instance 1 when modifying the target data page, the backup database receives the conventional redo log of the database instance 1 first, and then receives the page synchronous redo log and the conventional redo log of the database instance 2, and in the concurrent situation, even if the database instance 2 has a fast thread replay speed and encounters the page synchronous redo log, and the replay completion log serial number is judged to be smaller than the synchronous log serial number of the page synchronous redo log, the database instance 2 waits for the replay completion of the replay thread of the database instance 1, and then resumes the replay of the conventional redo log.
If the target data page is flushed, the standby database receives the page synchronous redo log and the conventional redo log of the database instance 2, and the serial number of the replay completion log is smaller than that of the page synchronous redo log, but the principle that remote reading requires the log to be flushed first can be used for ensuring that other database instances do not modify the target data page, and the database instance 2 does not need to wait at the moment.
And S240, performing log replay on the conventional redo log in parallel.
Wherein the number of regular redo logs is at least one. And the backup database performs log replay on the conventional redo log through a parallel thread.
According to the technical scheme of the embodiment, the redo log sent by the main database is obtained through the backup database, and the redo log is replayed according to the log type of the redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log. In this embodiment, because the redo log includes a redo log type of the new page synchronous redo log, the redo log other than the page synchronous redo log can be processed in parallel directly according to the log type without additional logic judgment, and the replay efficiency and performance of the backup database are greatly improved.
EXAMPLE III
Fig. 5 is a schematic structural diagram of a log replay device in a third embodiment of the present invention, where the log replay device may be disposed in a master database, and this embodiment is applicable to a case of performing log replay on a redo log in the database. The log replay device provided by the embodiment of the invention can execute the log replay method provided by the embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
The device specifically comprises a log generation module 310 and a log sending module 320, wherein:
the log generating module 310 is configured to generate a redo log if it is determined that the current database instance modifies the target data page for the first time, where the redo log includes a page synchronization redo log and a conventional redo log;
the log sending module 320 is configured to send the redo log to the standby database, so that the standby database performs log replay according to the log type of the redo log.
According to the technical scheme of the embodiment, when the main database determines that the current database instance modifies the target data page for the first time, the redo log is generated, and the redo log is sent to the standby database, so that the standby database performs log replay according to the log type of the redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log. In this embodiment, when the redo log is generated by the master database, a new redo log type of the page synchronization redo log is added, so that the backup database does not need extra logic judgment, and redo logs other than the page synchronization redo log can be processed in parallel according to the log type, thereby greatly improving the replay efficiency and performance of the backup database.
Further, the log generating module 310 includes a first modification determining unit, and the first modification determining unit is specifically configured to:
and if the target data page is determined to be subjected to disc brushing, or the target data page is not subjected to disc brushing, and the database instance corresponding to the last modification record in the global cache is different from the current database instance, determining that the current database instance carries out first modification on the target data page.
Further, the log generating module 310 includes a page synchronization redo log unit, and the page synchronization redo log unit is specifically configured to:
determining a page log serial number of a target data page before first modification as a synchronous log serial number, wherein the synchronous log serial number is increased progressively and is unique;
and generating a page synchronization redo log based on the synchronous log sequence number and the database instance corresponding to the last modification record of the target data page before the first modification.
Further, the apparatus further comprises:
and the conventional log module is used for generating a conventional redo log if the current database instance is determined to modify the target data page for the non-first time.
The log replay device provided by the embodiment of the invention can execute the log replay method provided by the embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 6 is a schematic structural diagram of a log replay device in a fourth embodiment of the present invention, where the log replay device may be disposed in a backup database, and this embodiment is applicable to a case of performing log replay on a redo log in the database. The log replay device provided by the embodiment of the invention can execute the log replay method provided by the second embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
The device specifically includes a log obtaining module 410 and a log replaying module 420, wherein:
a log obtaining module 410, configured to obtain a redo log sent by a master database;
the log replay module 420 is configured to replay the redo log according to the log type of the redo log, where the redo log includes a page synchronization redo log and a conventional redo log.
According to the technical scheme of the embodiment, the redo log sent by the main database is obtained through the backup database, and the redo log is replayed according to the log type of the redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log. In this embodiment, because the redo log includes a redo log type of the new page synchronous redo log, the redo log other than the page synchronous redo log can be processed in parallel directly according to the log type without additional logic judgment, and the replay efficiency and performance of the backup database are greatly improved.
Further, the log replay module 420 includes:
and the first replay unit is used for replaying the redo log according to the comparison result of the synchronous log serial number of the redo log and the replay completion log serial number if the redo log is the page synchronous redo log.
Further, the first replay unit is specifically configured to:
and if the serial number of the replay completion log is less than the serial number of the synchronous log of the replay log, executing a waiting operation until the serial number of the replay completion log is greater than or equal to the serial number of the synchronous log of the replay log.
Further, the log replay module 420 includes:
and the second replay unit is used for replaying the logs of the conventional redo logs in parallel if the redo logs are the conventional redo logs, wherein the number of the conventional redo logs is at least one.
The log replay device provided by the embodiment of the invention can execute the log replay method provided by the second embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
Fig. 7 is a schematic structural diagram of a server in the fifth embodiment of the present invention. FIG. 7 illustrates a block diagram of an exemplary server 512 suitable for use in implementing embodiments of the present invention. The server 512 shown in fig. 7 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.
As shown in FIG. 7, the server 512 is in the form of a general purpose server. Components of server 512 may include, but are not limited to: one or more processors 516, a storage device 528, and a bus 518 that couples the various system components including the storage device 528 and the processors 516.
The server 512 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by server 512 and includes both volatile and nonvolatile media, removable and non-removable media.
A program/utility 540 having a set (at least one) of program modules 542 may be stored, for example, in storage 528, such program modules 542 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a network environment. The program modules 542 generally perform the functions and/or methods of the described embodiments of the invention.
The server 512 may also communicate with one or more external devices 514 (e.g., keyboard, pointing terminal, display 524, etc.), with one or more terminals that enable a user to interact with the server 512, and/or with any terminals (e.g., network card, modem, etc.) that enable the server 512 to communicate with one or more other computing terminals. Such communication may occur via input/output (I/O) interfaces 522. Further, server 512 may communicate with one or more networks (e.g., a Local Area Network (LAN), Wide Area Network (WAN), and/or a public Network such as the Internet) via Network adapter 520. As shown in FIG. 7, the network adapter 520 communicates with the other modules of the server 512 via the bus 518. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the server 512, including but not limited to: microcode, end drives, Redundant processors, external disk drive Arrays, RAID (Redundant Arrays of Independent Disks) systems, tape drives, and data backup storage systems, among others.
The processor 516 executes various functional applications and data processing by executing programs stored in the storage device 528, for example, implementing the log replay method provided by any embodiment of the present invention.
EXAMPLE six
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the log replay method provided in any embodiment of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (9)
1. A log replay method is applied to a master database and comprises the following steps:
if it is determined that the current database instance modifies the target data page for the first time, generating a redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log;
sending the redo log to a standby database so that the standby database performs log replay according to the log type of the redo log;
generating a page-synchronized redo log, comprising:
determining a page log sequence number of the target data page before the first modification as a synchronization log sequence number, wherein the synchronization log sequence number is incremented and unique;
generating the page synchronous redo log based on the synchronous log serial number and a database instance corresponding to a last modification record of the target data page before the first modification;
performing log replay according to the log type of the redo log, including:
if the redo log is the page synchronous redo log, performing log replay on the redo log according to a comparison result of a synchronous log serial number of the redo log and a replay completion log serial number;
and if the redo log is the conventional redo log, performing log replay on the conventional redo log in parallel, wherein the number of the conventional redo log is at least one.
2. The method of claim 1, wherein determining that the current database instance makes a first modification to the target data page comprises:
and if the target data page is determined to be disqualified, or the target data page is not disqualified, and the database instance corresponding to the last modification record in the global cache is different from the current database instance, determining that the current database instance modifies the target data page for the first time.
3. The method of claim 1, further comprising:
and if the current database instance is determined to modify the target data page for the non-first time, generating a conventional redo log.
4. The method of claim 1, applied to a database, comprising:
acquiring a redo log sent by a main database;
and performing log replay on the redo log according to the log type of the redo log, wherein the redo log comprises a page synchronization redo log and a conventional redo log.
5. The method of claim 1, wherein performing log replay on the redo log according to a comparison result between a synchronization log sequence number of the redo log and a replay completion log sequence number comprises:
and if the replay completion log serial number is less than the synchronous log serial number of the replay log, executing a waiting operation until the replay completion log serial number is greater than or equal to the synchronous log serial number of the replay log.
6. A log replaying apparatus configured in a master database, comprising:
the log generation module is used for generating a redo log if the current database instance is determined to modify the target data page for the first time, wherein the redo log comprises a page synchronization redo log and a conventional redo log;
the log sending module is used for sending the redo log to a standby database so as to enable the standby database to replay the log according to the log type of the redo log;
the page synchronization redo log unit is specifically configured to:
determining a page log serial number of the target data page before first modification as a synchronous log serial number, wherein the synchronous log serial number is increased progressively and is unique;
generating the page synchronous redo log based on the synchronous log serial number and a database instance corresponding to a last modification record of the target data page before the first modification;
the first replay unit is used for replaying the redo log according to a comparison result of a synchronous log serial number of the redo log and a replay completion log serial number if the redo log is a page synchronous redo log;
and the second replay unit is used for replaying the logs of the conventional redo logs in parallel if the redo logs are the conventional redo logs, wherein the number of the conventional redo logs is at least one.
7. The apparatus of claim 6, configured to provision a database, comprising:
the log acquisition module is used for acquiring a redo log sent by the main database;
and the log replay module is used for replaying the redo logs according to the log types of the redo logs, wherein the redo logs comprise page synchronous redo logs and conventional redo logs.
8. A server, characterized in that the server comprises:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the log replay method of any one of claims 1-5.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the log replay method as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910748717.7A CN110442560B (en) | 2019-08-14 | 2019-08-14 | Log replay method, device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910748717.7A CN110442560B (en) | 2019-08-14 | 2019-08-14 | Log replay method, device, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110442560A CN110442560A (en) | 2019-11-12 |
| CN110442560B true CN110442560B (en) | 2022-03-08 |
Family
ID=68435397
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910748717.7A Active CN110442560B (en) | 2019-08-14 | 2019-08-14 | Log replay method, device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110442560B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111290881B (en) * | 2020-01-21 | 2023-09-19 | 上海达梦数据库有限公司 | Data recovery method, device, equipment and storage medium |
| CN111324665B (en) * | 2020-01-23 | 2023-06-27 | 阿里巴巴集团控股有限公司 | Log playback method and device |
| CN111639087B (en) * | 2020-05-28 | 2023-09-08 | 北京金山云网络技术有限公司 | Data updating method and device in database and electronic equipment |
| CN113868028A (en) * | 2020-06-30 | 2021-12-31 | 华为技术有限公司 | Method for replaying log on data node, data node and system |
| CN112416654B (en) * | 2020-11-26 | 2024-04-09 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
| CN112637284A (en) * | 2020-12-09 | 2021-04-09 | 北京金山云网络技术有限公司 | Redo log storage method and device, electronic device and storage medium |
| CN112596953B (en) * | 2020-12-25 | 2023-08-29 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
| CN113239120B (en) * | 2021-06-07 | 2023-08-18 | 上海达梦数据库有限公司 | Log synchronization method, device, equipment and storage medium |
| CN113987078B (en) * | 2021-12-24 | 2022-04-19 | 中兴通讯股份有限公司 | Data synchronization method, device and computer readable storage medium |
| CN115114370B (en) * | 2022-01-20 | 2023-06-13 | 腾讯科技（深圳）有限公司 | Master-slave database synchronization method and device, electronic equipment and storage medium |
| CN114647624B (en) * | 2022-05-11 | 2022-08-02 | 成都云祺科技有限公司 | Method, system and storage medium for capturing database consistent point in block-level CDP |
| CN115454717B (en) * | 2022-09-16 | 2023-05-05 | 广州鼎甲计算机科技有限公司 | Database real-time backup method and device, computer equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102629250A (en) * | 2012-02-28 | 2012-08-08 | 杭州丰城信息技术有限公司 | Recovery method of redo log files for main memory database |
| CN103412803A (en) * | 2013-08-15 | 2013-11-27 | 华为技术有限公司 | Data recovering method and device |
| CN103793479A (en) * | 2014-01-14 | 2014-05-14 | 上海上讯信息技术股份有限公司 | Log management method and log management system |
| CN106776130A (en) * | 2016-11-30 | 2017-05-31 | 华为技术有限公司 | A kind of journal recovery method, storage device and memory node |
| CN108664359A (en) * | 2018-05-23 | 2018-10-16 | 上海达梦数据库有限公司 | A kind of database restoring method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897311B (en) * | 2015-12-21 | 2020-08-11 | 财团法人工业技术研究院 | Database batch updating method, data reduction log generating method and storage device |
-
2019
- 2019-08-14 CN CN201910748717.7A patent/CN110442560B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102629250A (en) * | 2012-02-28 | 2012-08-08 | 杭州丰城信息技术有限公司 | Recovery method of redo log files for main memory database |
| CN103412803A (en) * | 2013-08-15 | 2013-11-27 | 华为技术有限公司 | Data recovering method and device |
| CN103793479A (en) * | 2014-01-14 | 2014-05-14 | 上海上讯信息技术股份有限公司 | Log management method and log management system |
| CN106776130A (en) * | 2016-11-30 | 2017-05-31 | 华为技术有限公司 | A kind of journal recovery method, storage device and memory node |
| CN108664359A (en) * | 2018-05-23 | 2018-10-16 | 上海达梦数据库有限公司 | A kind of database restoring method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 基于Oracle Streams的数据库实时备份与恢复技术研究;成雅;《中国优秀硕士学位论文全文数据库 信息科技辑》;20130415(第4期);全文 * |
| 达梦数据库备份与恢复子系统的设计与实现;何儒汉;《中国优秀博硕士学位论文全文数据库 (硕士) 信息科技辑》;20060615(第2期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110442560A (en) | 2019-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442560B (en) | Log replay method, device, server and storage medium | |
| US8527459B2 (en) | System and method for data replication between heterogeneous databases | |
| US9229970B2 (en) | Methods to minimize communication in a cluster database system | |
| CN109325016B (en) | Data migration method, device, medium and electronic equipment | |
| US20080005183A1 (en) | Method And Apparatus for Propagating Tables While Preserving Cyclic Foreign Key Relationships | |
| WO2021012932A1 (en) | Transaction rollback method and device, database, system, and computer storage medium | |
| CN112416654B (en) | Database log replay method, device, equipment and storage medium | |
| CN111061690B (en) | RAC-based database log file reading method and device | |
| EP3933639A1 (en) | Transaction processing method, apparatus, and electronic device for blockchain | |
| CN111752482A (en) | Method and system for writing data in full-flash distributed storage system | |
| CN105302489A (en) | Heterogeneous multi-core remote embedded memory system and method | |
| US8521682B2 (en) | Transfer of data from transactional data sources to partitioned databases in restartable environments | |
| CN112231403B (en) | Consistency verification method, device, equipment and storage medium for data synchronization | |
| US11573876B2 (en) | Scalable exactly-once data processing using transactional streaming writes | |
| CN111290881B (en) | Data recovery method, device, equipment and storage medium | |
| CN117112522A (en) | Concurrent process log management method, device, equipment and storage medium | |
| CN111090782A (en) | Graph data storage method, device, equipment and storage medium | |
| CN112395141B (en) | Data page management method and device, electronic equipment and storage medium | |
| CN114896276A (en) | Data storage method and device, electronic equipment and distributed storage system | |
| CN114489480A (en) | Method and system for high-concurrency data storage | |
| CN111324668B (en) | Database data synchronous processing method, device and storage medium | |
| CN109740027B (en) | Data exchange method, device, server and storage medium | |
| CN109857523B (en) | Method and device for realizing high availability of database | |
| CN110046132B (en) | Metadata request processing method, device, equipment and readable storage medium | |
| US10268418B1 (en) | Accessing multiple data snapshots via one access point |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |