CN111324665A - Log playback method and device - Google Patents
Log playback method and device Download PDFInfo
- Publication number
- CN111324665A CN111324665A CN202010076416.7A CN202010076416A CN111324665A CN 111324665 A CN111324665 A CN 111324665A CN 202010076416 A CN202010076416 A CN 202010076416A CN 111324665 A CN111324665 A CN 111324665A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- data page
- updated
- identification information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a log playback method, which comprises the following steps: acquiring data page identification information of a data page to be updated; acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information; determining target log data corresponding to the data page to be updated according to the target log position information; and updating the data page to be updated by playing back the target log data. According to the method, after the data page identification information of the data page to be updated is obtained, the target log position information corresponding to the data page identification information is obtained, and the target log data corresponding to the data page to be updated is determined according to the target log position information, so that the target log data corresponding to the data page to be updated is played back in a targeted manner, the data page to be updated is updated, and the log playback speed can be increased.
Description
Technical Field
The application relates to the field of databases, in particular to a log playback method and device, electronic equipment and storage equipment. The application also relates to a method and a device for obtaining the log index data, the electronic equipment and the storage equipment.
Background
With the advent of the big data era, databases are more and more widely applied, and particularly with the gradual popularization of cloud platforms, databases based on shared storage are widely applied to various industries.
Currently, in order to maintain high availability of database services when a database is applied specifically, computation and storage separation is generally adopted, and a distributed architecture based on shared storage is used for providing high availability services. Generally speaking, a main and backup architecture is adopted to provide services, a main database serves as a readable and writable database node, namely a RW node, provides services to the outside, and writes data into a shared storage; the database is used as a readable node, namely the RO node, and the consistency of the externally provided data and the data in the RW node is kept through a log playback method based on a redo (redo) log generated by the RW node.
However, when the existing backup database provides data to the outside through a log playback method, it is usually necessary to acquire a complete redo log file generated by the primary database, and then perform log playback according to the acquired redo log file through a single process to ensure the consistency of the data provided between the primary and backup databases, which has the disadvantage that on one hand, the backup database is not targeted, that is, the backup database cannot update a certain data page only when the data page is accessed, but the backup database needs to update the data page according to the acquired redo log file as long as the data page is updated in the primary database regardless of whether the data page is accessed; on the other hand, correspondingly, when the backup database performs log playback, no matter whether the data page has the requirement of being accessed, the log playback needs to be performed according to the acquired redo log file, so that the problem that the playback speed is slow, and the performance of the main database is influenced indirectly exists.
In summary, the existing log playback method has the problems of no pertinence and slow playback speed.
Disclosure of Invention
The embodiment of the application provides a log playback method, which aims to solve the problems of no pertinence and low playback speed in the prior art.
The embodiment of the application provides a log playback method, which comprises the following steps:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
Optionally, the method includes: judging whether the existing data page in the cache region needs to be updated or not according to the metadata information of the log data to be redone; and if the existing data page in the cache region has a data page needing to be updated, setting an available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region.
Optionally, the method further includes: and if the existing data pages in the cache region do not have the data pages needing to be updated, generating the log index data corresponding to the metadata information of the log data to be redone.
Optionally, the obtaining of the data page identification information of the data page to be updated includes:
and if the transaction access request aiming at the data page is acquired, inquiring whether the data page is available according to the available state identification corresponding to the data page, and if the available state identification corresponding to the data page is in an unavailable state, taking the data page as the data page to be updated and acquiring the data page identification information of the data page to be updated.
Optionally, setting the available state identifier corresponding to the data page to be updated to an unavailable state includes: and setting the available state identifier corresponding to the data page needing to be updated into an unavailable state in a parallel or serial mode.
Optionally, the obtaining of the data page identification information of the data page to be updated includes: acquiring a transaction access request message sent by a client; obtaining the identification information of the data page requested to be accessed from the transaction access request message; and judging whether the data page requested to be accessed needs to be updated or not according to the metadata information of the log data to be redone, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring the data page identification information of the data page to be updated.
Optionally, the log index data is obtained by the following method: obtaining metadata information of redo log data generated by a main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the log data to be redone; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Optionally, the method further includes: and storing the log index data.
Optionally, the method includes: obtaining identification information for identifying the log index data; and updating metadata information associated with the log index data according to the identification information.
Optionally, the method further includes: generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
Optionally, the method includes: if the data page to be updated is not included in the cache region, reading the data page from the shared storage to the cache region before executing the step of updating the data page to be updated by playing back the target log data.
Optionally, the method further includes: if the data page to be updated is a data page corresponding to a data page requested to be accessed by a client and the data page to be updated is not included in the cache region, reading the data page from a shared memory to the cache region before executing the step of updating the data page to be updated by playing back the target log data.
The embodiment of the present application further provides a method for obtaining log index data, including:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
An embodiment of the present application further provides a log playback device, including:
the data page identification information acquisition unit is used for acquiring the data page identification information of the data page to be updated;
a target log position information obtaining unit, configured to obtain target log position information from log index data according to data page identification information of the data page to be updated, where the log index data is used to describe a correspondence between the data page identification information and the log position information;
the target log data determining unit is used for determining target log data corresponding to the data page to be updated according to the target log position information;
and the playback unit is used for updating the data page to be updated by playing back the target log data.
An embodiment of the present application further provides an electronic device, including:
a processor;
a memory for storing a program of a log playback method, the apparatus performing the following steps after being powered on and running the program of the log playback method by the processor:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
An embodiment of the present application further provides a storage device, in which a program of a log playback method is stored, where the program is run by a processor and executes the following steps:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
An embodiment of the present application further provides an apparatus for obtaining log index data, including:
the data acquisition unit is used for acquiring metadata information of redo log data generated by the main database;
an information obtaining unit, configured to obtain, from the metadata information, data page identification information of at least one data page, and obtain log location information of log data corresponding to the data page in the redo log data;
and the log index data obtaining unit is used for establishing the corresponding relation between the data page identification information and the log position information to obtain the log index data.
An embodiment of the present application further provides an electronic device, including:
a processor;
a memory for storing a program of an obtaining method of log index data, the apparatus performing the following steps after being powered on and running the program of the obtaining method of log index data by the processor:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
The embodiment of the present application further provides a storage device, in which a program of a method for obtaining log index data is stored, where the program is run by a processor and executes the following steps:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Compared with the prior art, the method has the following advantages:
the embodiment of the application provides a log playback method, which comprises the following steps: acquiring data page identification information of a data page to be updated; acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information; determining target log data corresponding to the data page to be updated according to the target log position information; and updating the data page to be updated by playing back the target log data. According to the method, after the data page identification information of the data page to be updated is obtained, the target log position information corresponding to the data page identification information is obtained from the log index data, and the target log data corresponding to the data page to be updated is determined according to the target log position information, so that the target log data corresponding to the data page to be updated is played back in a targeted manner to update the data page to be updated.
The embodiment of the present application further provides a method for obtaining log index data, including: obtaining metadata information of redo log data generated by a main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data. The method obtains the data page identification information of at least one data page by analyzing the metadata information after obtaining the metadata information of the redo log data generated by the main database, and obtaining the log position information of the log data corresponding to the data page in the summary of the redo log data, by establishing the corresponding relationship between the data page identification information and the log position information, the log index data is obtained, compared with the traditional method of performing log playback based on the whole complete redo log file, according to the log index data obtained by the method provided by the embodiment of the application, the log playback can be performed in a targeted manner only by transmitting the metadata information of the redo log file between the main database and the standby database, so that the log playback speed can be increased, and the data volume of network transmission between the main database and the standby database can be reduced.
Drawings
Fig. 1 is a schematic diagram of an application scenario of a log playback method according to a first embodiment of the present application.
Fig. 2 is a flowchart of a log playback method according to a first embodiment of the present application.
Fig. 3 is a schematic structural diagram of log index data according to a first embodiment of the present application.
Fig. 4 is a schematic diagram of a first stage of a log playback method according to a first embodiment of the present application.
Fig. 5 is a schematic diagram of a second stage of the log playback method provided in the first embodiment of the present application.
Fig. 6 is a schematic diagram of a method for concurrently implementing log playback according to a first embodiment of the present application.
Fig. 7 is a flowchart of a method for obtaining log index data according to a second embodiment of the present application.
Fig. 8 is a schematic diagram of a log playback apparatus according to a third embodiment of the present application.
Fig. 9 is a schematic diagram of an apparatus for obtaining log index data according to a fourth embodiment of the present application.
Fig. 10 is a schematic diagram of an electronic device according to a fifth embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.
In order to make those skilled in the art better understand the scheme of the present application, a detailed description is given below of a specific application scenario of an embodiment of the log playback method based on the log playback method provided by the present application. The log playback method can be particularly applied between a main database and a standby database of a distributed database based on shared storage, and further applied to the standby database of the distributed database to ensure the consistency of data provided by the standby database to a client and the data of the main database. Fig. 1 is a schematic view of an application scenario of a log playback method according to a first embodiment of the present application.
In specific implementation, in a distributed database based on shared storage, the main database 101 serves as a readable and writable node, when the content of a certain data page in the cache region of the main database 101 is changed and the main database 101 has not refreshed the change to the shared storage, in order to ensure consistency between the content of the data page provided by the standby database 102 to the client 103 and the content in the cache region of the main database 101, when the standby database 102 acquires an access request of the client 103 for the data page, the standby database 102 takes the data page as a data page to be updated and acquires data page identification information of the data page to be updated; then, the backup database 102 obtains target log position information from the log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relationship between the data page identification information and the log position information; the standby database 102 determines target log data corresponding to the data page to be updated according to the obtained target log position information, and updates the data page to be updated by revisiting the target log data; after the database 102 updates the data page to be updated, the content of the data page to be updated is provided to the client 103.
The client is a computing device for acquiring data from the database, and may be a conventional computer device, or a mobile terminal device, such as a mobile phone, a tablet computer, or the like, where the database may be a main database of a distributed database, or a backup database.
The master database refers to a database node serving as a readable and writable node to provide data services to clients in a distributed database based on shared storage, where the master database may also be referred to as a RW node, and for example, the master database may be a master database node in various commonly used distributed databases.
The backup database is a database node serving as a readable node to provide data service to a client in a distributed database based on shared storage, shares a bottom data storage space with a main database, and ensures the consistency of data in a cache region of the backup database node based on a redo log file generated by the main database, wherein the backup database can also be called as an RO node or a slave database node.
The core idea of the Redo log file, also called Write-Ahead Logging, is that modifications to a data page must occur after the modifications have been recorded in the Redo log file, that is, the actual modification operations to the data page can not be performed until the log records describing the data changes are flushed to shared storage.
The data page is a carrier of data stored in the database, and can also be called as a data block or a data file; the data may specifically take the form of a data table, a data index, or the like.
The data page identification information is information for uniquely identifying a data page, and may specifically be a string of character string information, for example, PageID.
In the above, three entities, namely, the primary database 101, the secondary database 102 and the client 103, are taken as examples to describe an application scenario of the log playback method provided in the first embodiment of the present application, and in a specific implementation, the method may also be applied only between the primary database 101 and the secondary database 102, and it is not necessary to trigger implementation of the log playback method provided in the first embodiment of the present application after the secondary database 102 acquires a transaction access request of the client 103.
For example, when the content of a certain data page in the buffer of the main database 101 is changed and the main database 101 has not refreshed the change to the shared storage, in order to ensure the consistency between the content in the buffer of the backup database 102 and the content in the buffer of the main database 101, the backup database 102 obtains the data page with the data change in the buffer of the main database 101 from the metadata information of the redo log data sent by the main database 101, and uses the data page as the data page to be updated, and obtains the data page identification information of the data page to be updated; then, the standby database 102 obtains the target log location information from the log index data according to the data page identification information of the data pages to be updated, determines the target log data corresponding to the data pages to be updated according to the obtained target log location information, and updates the data pages to be updated by playing back the target log data, so as to ensure the consistency between the content in the cache area of the standby database 102 and the content in the cache area of the main database 101.
It should be noted that the above application scenarios are only some specific embodiments of the log playback method described in the present application, and the above application scenario embodiments are provided to facilitate understanding of the log playback method of the present application, and are not used to limit the log playback method of the present application.
Referring to fig. 2, which is a flowchart of a log playback method according to a first embodiment of the present application, a detailed description is provided below with reference to fig. 2 for a log playback method according to the first embodiment of the present application.
Step S201, acquiring data page identification information of a data page to be updated.
Specifically, the log playback method provided in the first embodiment of the present application may be applied to a backup database based on a distributed database with shared storage, and when the method is implemented, it is first required to determine a data page to be updated in the backup database, that is, a data page in the backup database that is waiting for log playback to make its content consistent with the content in the buffer of the primary database, and the following method for determining a data page to be updated is described.
In a distributed database based on shared storage, all change operations for data pages need to record change records corresponding to the change operations into a redo log file before the change operations are actually applied to the data pages, and when the change operations for the data pages are recorded into the redo log file, metadata information of the redo log file is changed correspondingly. Therefore, after the backup database acquires the metadata information of the redo log file, the metadata information of the to-be-redo log data which is not subjected to redo operation can be acquired from the metadata information, the data pages with data change in the main database, namely changed data pages, are analyzed from the metadata information of the to-be-redo log data, and the data page identification information of the data pages is acquired; after obtaining the changed data pages in the master database, a more straightforward approach is: and directly taking all the obtained changed data pages as data pages to be updated. That is, the method for determining the data page to be updated includes: the method comprises the steps of obtaining metadata information of log data to be redone, analyzing the metadata information, obtaining a data page changed in a main database, and taking the data page changed in the main database as the data page to be updated, wherein the log data to be redone is redone log data waiting for redoing operation.
It should be noted that the MetaData (MetaData) information of the redo log file is data describing log data in the redo log file, or may be structural data for providing information related to the log data in the redo log file. The purpose of its use lies in: identifying and evaluating relevant log data in the redo log file, and tracking changes of the log data in the using process; it has the following characteristics: 1. once the metadata information is established, the metadata information can be shared; 2. metadata information is a coding scheme different from the traditional data coding scheme, and is specifically used for establishing a framework for redo log data in a log file, which can be understood by a computing device.
For example, the metadata information may include a data type, an action corresponding to the data type, and data page information. The data type can be used for identifying the btree type data or the heal type data of the log data; the action corresponding to the data type may be used to identify the operation corresponding to the data type, for example, the action may be an add, delete, or split action; the data page information may include location information of the data page, that is, location information of db. In particular, a piece of metadata information may contain information of multiple data pages, for example, a btree index split may involve three data pages; when the metadata information is analyzed, the data pages involved need to be locked in sequence according to different data types and actions, wherein the locking process can be called mini transaction to ensure atomicity of modification for the data pages. Whether a relevant page is locked in the mini transaction needs to be judged when the log index data is generated and retrieval is carried out based on the log index data.
In a first embodiment of the present application, in order to increase a log playback speed, reduce a log playback cost, and perform a log playback in a targeted manner, the method for determining a data page to be updated includes: judging whether the existing data page in the cache region needs to be updated or not according to the metadata information of the log data to be redone; and if the existing data page in the cache region has a data page needing to be updated, setting an available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region. Wherein, the setting the available state identifier corresponding to the data page to be updated to an unavailable state includes: and setting the available state identifier corresponding to the data page needing to be updated into an unavailable state in a parallel or serial mode.
In addition, the method further comprises: and if the existing data pages in the cache region do not have the data pages needing to be updated, generating the log index data corresponding to the metadata information of the log data to be redone. That is, if the data page that has been updated in the primary database does not exist in the buffer of the backup database, the backup database does not need to perform the log playback operation in the backup database, and the backup database only needs to generate the log index data corresponding to the obtained metadata information of the log data to be redone in the buffer thereof.
Fig. 4 is a schematic diagram of a first stage of a log playback method according to a first embodiment of the present application. As can be seen from fig. 4, in consideration that existing data pages in the cache region of the standby database are not always accessed by the client, after the standby database obtains metadata information of log data to be redone, log playback operation is not directly performed, but a corresponding available state identifier is maintained in the cache region of the standby database in parallel or in series for each existing data page, where the available state identifier is used to identify whether the content of the data page is directly usable; meanwhile, according to the metadata information, journal index data is generated in the backup database, and detailed description about the journal index data is explained in detail in the subsequent steps, which is not described herein again.
After determining the data page to be updated in the database, the acquiring the data page identification information corresponding to the data page to be updated includes: and if the transaction access request aiming at the data page is acquired, inquiring whether the data page is available according to the available state identification of the data page, and if the available state identification corresponding to the data page is in an unavailable state, taking the data page as the data page to be updated and acquiring the data page identification information of the data page to be updated.
That is, after the data is received from the transaction access request of the client, the data page accessed by the transaction request can be parsed from the transaction access request; if the corresponding available state identifier of the data page in the cache region of the standby database identifies that the data page is unavailable, the standby database may use the data page as a data page to be updated and obtain the data page identifier information of the data page to be updated, wherein the related description of the transaction in the database is not repeated herein because the related description in the prior art has detailed description.
As can be seen from the above description, the method specifically determines the data page to be updated in a targeted manner by maintaining the available state identifier in the cache region of the backup database, and performs log playback on the data page if the data page has a corresponding transaction access request; if the data page has no corresponding transaction access, the log playback is not carried out on the data page, and the method can reduce the access amount to the storage, thereby reducing the log playback cost and increasing the log playback speed; in addition, between the main database and the standby database, the data page to be updated can be determined only by transmitting the metadata information of the redo log file, and subsequent log playback processing is performed without transmitting the complete redo log file, so that the data volume of network transmission between the main database and the standby database can be reduced.
In addition, a first embodiment of the present application further provides a method for acquiring data page identification information of a data page to be updated, which specifically includes: acquiring a transaction access request message sent by a client; obtaining the identification information of the data page requested to be accessed from the transaction access request message; and judging whether the data page requested to be accessed needs to be updated or not according to the metadata information of the log data to be redone, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring the data page identification information of the data page to be updated.
Specifically, after the standby database receives a transaction access request from the client, the standby database obtains identification information of the data page requested to be accessed from the transaction access request; and then, the standby database judges whether the data page needs to be updated according to the metadata information of the log data to be redone, namely whether the content of the data page is changed in a cache region of the main database, if so, namely, the data page needs to be updated, the standby database takes the data page requested to be accessed as the data page to be updated, and acquires the data page identification information of the data page to be updated.
It should be noted that, in a specific implementation, one or a combination of multiple methods may be used to determine the data page to be updated and obtain the data page identification information of the data page to be updated at the same time, and of course, as the technology is continuously advanced, the data page to be updated and the data page identification information of the data page to be updated may also be determined by other methods or a method obtained by expanding the above methods.
As can be seen from the above description, the log playback method provided in the first embodiment of the present application can specifically determine the data page to be updated, so as to reduce the access amount to the network and storage, reduce the log playback cost, and increase the log playback speed.
Step S202, according to the data page identification information of the data page to be updated, obtaining target log position information from log index data, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information.
After the data page identification information of the data page to be updated is obtained in step S201, the subsequent log playback processing may be performed according to the data page identification information, that is, the target log position information is obtained from the log index data according to the data page identification information of the data page to be updated.
Before introducing step S202, a detailed description is first given of a method for obtaining log index data used in this step, as shown in fig. 3, which is a schematic structural diagram of the log index data provided in the first embodiment of the present application, where the log index data can be obtained by the following method: obtaining metadata information of redo log data generated by a main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the log data to be redone; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
After obtaining the log index data, further comprising: and storing the log index data. Wherein, while storing the log index data, comprising: obtaining identification information for identifying the log index data; and updating metadata information associated with the log index data according to the identification information. In addition, still include: generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
In the first embodiment of the present application, the log playback method specifically includes two stages, where the first stage has already been described in step S201, that is, log index data is continuously generated according to metadata information of a to-be-redo log file generated by main data in a cache region of a main database and a cache region of a standby database; meanwhile, the database also comprises: and updating the available state identifier of the data page in the cache region of the database according to the metadata information of the log file to be redone.
The generation process of the log index data specifically includes analyzing and obtaining data page identification information of at least one data page with data change in the master database according to metadata information of a redo log file generated by the master database, obtaining log position information of log data corresponding to the data page in the to-be-redo log data, and then obtaining the log index data by establishing a corresponding relationship between the data page identification information and the log position information.
The Log Location (LSN) information is information of a location for recording Log data in the redo Log file, and has a monotonically increasing attribute, and in a specific implementation, the Log location information may be 64-bit string information.
As can be seen from the above description, the log index data is specifically the corresponding relationship between the data page identification information and the log location information, that is, one piece of log index data is composed of the data page identification information and the log location information, and for example, may be composed of PageID and LSN. In the first embodiment of the present application, the log index data is stored in a hash table, the data page identification information is a key value of the hash table, and the log location information is a data value. In addition, in order to save storage space, in the obtained log index data, the log location information may be split into prefix information for describing which data page the log index file is for and suffix information which is a monotonically increasing sequence number.
It should be noted that, after the log index data is generated in the cache area of the primary database, the log index data may not be directly downloaded, but as shown in fig. 3, the log index data is written into an active table (active table), and after an active table is full of data, the active table full of data is set as an inactive table (inactive table) and waits to be written into the shared storage, so as to increase the retrieval speed of the log index data.
Furthermore, when the log index data is landed, i.e. written into the shared storage, the identification information of the log index file corresponding to the log index data can be correspondingly generated, and the metadata information associated with the log index data is updated according to the identification information, i.e. the metadata information of the log index file is updated; meanwhile, a corresponding filter for further increasing the retrieval speed for the log index data corresponding to the log index file is generated, for example, a bloom filter (BloomFilter) corresponding to the log index data is generated. How to generate the filter is described in detail in the prior art, and is not described in detail here.
It should be noted that, when the log index data is destaged to obtain the identification information of the log index file, the metadata information of the log index file is updated, because in the database architecture based on the shared storage, one master database node may correspond to a plurality of backup database nodes at the same time, and each backup database node may generate the log index data corresponding to the content in the cache area of the master database node in its cache area according to the metadata information of the redo log data generated by the master database node, in order to increase the utilization rate of the cache area, when a part of the log index data in the master database node, such as the log index data in the inactive table of the full data, is written into the shared storage, the backup database node may eliminate the corresponding log index data in the cache area of the backup database node at an appropriate time according to the metadata information of the updated log index file, to increase the utilization of the cache. As can be seen from fig. 3, in order to increase the retrieval speed of the log index data and improve the utilization rate of the cache area, the structure of the log index data provided in the first embodiment of the present application may specifically include a log index file, a bloom filter corresponding to the log index file, and metadata information corresponding to the log index file.
It should be noted that, the above is only one embodiment of the log index data provided in the first embodiment of the present application, and when the embodiment is specifically implemented, other methods may also be used to generate the log index data, and the log index data is set to other data structures, for example, LSM-Tree is used to implement the log index data, which is not described herein again.
After the log index data is generated in the cache region of the database, when the data page identification information of the data page to be updated is obtained and the data page to be updated needs to be updated through log playback, the target log position information corresponding to the data page to be updated can be obtained by retrieving the obtained log index data through the data page identification information of the data page to be updated.
Step S203, determining target log data corresponding to the data page to be updated according to the target log position information.
After the target log position information corresponding to the data page to be updated is obtained in S202, positioning may be performed in the redo log file generated by the master database according to the target log position information, so as to determine the target log data corresponding to the data page to be updated.
Step S204, the data page to be updated is updated by playing back the target log data.
The target log data corresponding to the data page to be updated is obtained through step S203, and by playing back the target log data, the data page to be updated can be updated,
fig. 5 is a schematic diagram of a second phase of the log playback method according to the first embodiment of the present application. As can be seen from fig. 5, when log playback is performed, if the data page to be updated is not included in the buffer, the data page may be read from the shared storage into the buffer before the step of updating the data page to be updated by playing back the target log data is performed.
That is, if the data page to be updated is not in the cache area of the standby database, the dropped contents with the updated data page need to be read into the cache area of the standby database from the shared storage, and then, the target log position information obtained through the log index data and the target log data obtained from the redo log file are used for log playback, so as to update the data page to be updated.
In addition, the method further comprises: if the data page to be updated is a data page corresponding to a data page requested to be accessed by a client and the data page to be updated is not included in the cache region, reading the data page from a shared memory to the cache region before executing the step of updating the data page to be updated by playing back the target log data.
That is, only when a transaction access request message sent by the client is received, and a data page in the transaction access request message is a data page needing to be updated, and the data page needing to be updated does not exist in the cache region of the standby database, the data page needing to be updated needs to be read from the shared storage into the cache region before the log playback operation is executed.
It should be noted that, because the log playback method provided in the first embodiment of the present application can perform log playback in a targeted manner through the log index data, in a specific implementation, the method may not be limited to perform log playback in a single process, but may perform log playback in parallel. As shown in fig. 6, which is a schematic diagram of a method for concurrently implementing log playback according to a first embodiment of the present application, it can be known from fig. 6 that, when the log playback method is implemented in detail, multiple transaction access requests sent by a client may be concurrently received, and data pages to be updated in a cache region of a backup database are concurrently updated, so as to increase the speed of log playback.
In summary, the log playback method provided in the first embodiment of the present application includes: acquiring data page identification information of a data page to be updated; acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information; determining target log data corresponding to the data page to be updated according to the target log position information; and updating the data page to be updated by playing back the target log data. According to the method, after the data page identification information of the data page to be updated is obtained, the target log position information corresponding to the data page identification information is obtained from the log index data, and the target log data corresponding to the data page to be updated is determined according to the target log position information, so that the target log data corresponding to the data page to be updated is played back in a targeted manner to update the data page to be updated.
In the above description, a log playback method is provided, which corresponds to the log playback method provided in the first embodiment of the present application, and the second embodiment of the present application further provides a method for obtaining log index data, please refer to fig. 7, which is a flowchart of the method for obtaining log index data provided in the second embodiment of the present application, wherein some steps have been described in detail in the first embodiment of the present application, so that the description herein is relatively simple, and for the relevant points, reference may be made to some descriptions in the method for obtaining log index data provided in the first embodiment of the present application, and the processing procedures described below are only schematic.
Step S701, obtaining metadata information of redo log data generated by the master database.
Step S702, obtaining, from the metadata information, data page identification information of at least one data page, and obtaining log position information of log data corresponding to the data page in the redo log data.
Step S703, establishing a corresponding relationship between the data page identification information and the log location information, and obtaining the log index data.
Optionally, the method further includes: and storing the log index data.
Optionally, the method includes: obtaining identification information for identifying the log index data; and updating metadata information associated with the log index data according to the identification information.
Optionally, the method further includes: generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
In correspondence with the log playback method provided in the first embodiment of the present application, a log playback apparatus is also provided in the third embodiment of the present application, please refer to fig. 8, which is a schematic diagram of the log playback apparatus provided in the third embodiment of the present application. A log playback apparatus provided in a third embodiment of the present application includes:
a data page identification information obtaining unit 801, configured to obtain data page identification information of a data page to be updated.
A target log location information obtaining unit 802, configured to obtain target log location information from log index data according to the data page identification information of the data page to be updated, where the log index data is used to describe a corresponding relationship between the data page identification information and the log location information.
A target log data determining unit 803, configured to determine, according to the target log location information, target log data corresponding to the data page to be updated.
A playback unit 804, configured to update the data page to be updated by playing back the target log data.
Optionally, the apparatus includes: the state setting unit is used for judging whether the existing data page in the cache region needs to be updated or not according to the metadata information of the log data to be redone; and if the existing data page in the cache region has a data page needing to be updated, setting an available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region.
Optionally, if there is no data page to be updated in the existing data pages in the cache region, generating log index data corresponding to the metadata information of the log data to be redone.
Optionally, the data page identification information obtaining unit is specifically configured to, when a transaction access request for a data page is obtained, query whether the data page is available according to an available state identification corresponding to the data page, if the available state identification corresponding to the data page is an unavailable state, use the data page as the data page to be updated, and obtain data page identification information of the data page to be updated.
Optionally, setting the available state identifier corresponding to the data page to be updated to an unavailable state includes: and setting the available state identifier corresponding to the data page needing to be updated into an unavailable state in a parallel or serial mode.
Optionally, the apparatus includes: the data page to be updated acquiring unit is used for acquiring a transaction access request message sent by the client; obtaining the identification information of the data page requested to be accessed from the transaction access request message; and judging whether the data page requested to be accessed needs to be updated or not according to the metadata information of the log data to be redone, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring the data page identification information of the data page to be updated.
Optionally, the apparatus includes: the log index data acquisition unit is used for acquiring metadata information of redo log data generated by the main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the log data to be redone; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Optionally, the playback unit is specifically configured to determine whether the data page to be updated is included in the buffer, and if not, read the data page from a shared storage to the buffer before executing the step of updating the data page to be updated by playing back the target log data.
Optionally, the playback unit is further configured to determine whether the data page to be updated is a data page corresponding to a data page requested to be accessed by a client, and whether the data page to be updated is included in the buffer, and if the data page to be updated is a data page corresponding to a data page requested to be accessed by a client and the data page to be updated is not included in the buffer, read the data page from a shared storage to the buffer before executing the step of updating the data page to be updated by playing back the target log data.
Corresponding to the method for obtaining log index data provided by the first embodiment of the present application, a fourth embodiment of the present application further provides a device for obtaining log index data, please refer to fig. 9, which is a schematic diagram of the device for obtaining log index data provided by the fourth embodiment of the present application. A device for obtaining log index data provided by a fourth embodiment of the present application includes the following parts:
a data obtaining unit 901, configured to obtain metadata information of redo log data generated by the master database.
An information obtaining unit 902, configured to obtain, from the metadata information, data page identification information of at least one data page, and obtain log location information of log data corresponding to the data page in the redo log data.
A log index data obtaining unit 903, configured to establish a corresponding relationship between the data page identification information and the log location information, and obtain the log index data.
In correspondence with the log playback method provided in the first embodiment of the present application, a fifth embodiment of the present application further provides an electronic device, please refer to fig. 10, which is a schematic diagram of an electronic device provided in the fifth embodiment of the present application. A fifth embodiment of the present application provides an electronic device including:
a processor 1001;
a memory 1002 for storing a program of a log playback method, the apparatus performing the following steps after being powered on and running the program of the log playback method by the processor:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
In correspondence with the log playback method provided in the first embodiment of the present application, a sixth embodiment of the present application further provides a storage device, since the storage device embodiment is substantially similar to the method embodiment, the description is relatively simple, and for relevant points, reference may be made to part of the description of the method embodiment, and the storage device embodiment described below is only illustrative. A storage device according to a sixth embodiment of the present application stores a program of a log playback method, where the program is executed by a processor to perform the following steps:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
Corresponding to the method for obtaining log index data provided in the second embodiment of the present application, a seventh embodiment of the present application further provides another electronic device, the structure of which is substantially similar to that of the electronic device described in the fifth embodiment of the present application, and since the embodiment of the electronic device is substantially similar to the embodiment of the method, the description is relatively simple, and for relevant points, reference may be made to part of the description of the embodiment of the method, and the embodiment of the electronic device described below is only illustrative. A seventh embodiment of the present application provides an electronic device including:
a processor;
a memory for storing a program of an obtaining method of log index data, the apparatus performing the following steps after being powered on and running the program of the obtaining method of log index data by the processor:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Corresponding to the method for obtaining log index data provided in the second embodiment of the present application, an eighth embodiment of the present application further provides a storage device, since the embodiment of the storage device is substantially similar to the embodiment of the method, the description is relatively simple, and for relevant points, reference may be made to part of the description of the embodiment of the method, and the embodiment of the storage device described below is only illustrative. A storage device according to an eighth embodiment of the present application stores a program of a method for obtaining log index data, where the program is executed by a processor to perform the following steps:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the claims that follow.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Claims (19)
1. A log playback method, comprising:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
2. The log playback method according to claim 1, comprising:
judging whether the existing data page in the cache region needs to be updated or not according to the metadata information of the log data to be redone;
and if the existing data page in the cache region has a data page needing to be updated, setting an available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region.
3. The log playback method according to claim 2, further comprising:
and if the existing data pages in the cache region do not have the data pages needing to be updated, generating the log index data corresponding to the metadata information of the log data to be redone.
4. The log playback method according to claim 2, wherein the obtaining of the data page identification information of the data page to be updated includes:
and if the transaction access request aiming at the data page is acquired, inquiring whether the data page is available according to the available state identification corresponding to the data page, and if the available state identification corresponding to the data page is in an unavailable state, taking the data page as the data page to be updated and acquiring the data page identification information of the data page to be updated.
5. The log playback method according to claim 2, wherein the identifying an available state corresponding to the data page needing to be updated as an unavailable state comprises:
and setting the available state identifier corresponding to the data page needing to be updated into an unavailable state in a parallel or serial mode.
6. The log playback method according to claim 1, wherein the obtaining of the data page identification information of the data page to be updated includes:
acquiring a transaction access request message sent by a client;
obtaining the identification information of the data page requested to be accessed from the transaction access request message;
and judging whether the data page requested to be accessed needs to be updated or not according to the metadata information of the log data to be redone, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring the data page identification information of the data page to be updated.
7. The log playback method according to claim 1, wherein the log index data is obtained by:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the log data to be redone;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
8. The log playback method according to claim 7, further comprising:
and storing the log index data.
9. The log playback method according to claim 8, comprising:
obtaining identification information for identifying the log index data;
and updating metadata information associated with the log index data according to the identification information.
10. The log playback method according to claim 8, further comprising:
generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
11. The log playback method according to claim 1, comprising:
if the data page to be updated is not included in the cache region, reading the data page from a shared memory to the cache region before executing the step of updating the data page to be updated by playing back the target log data.
12. The log playback method according to claim 11, further comprising:
if the data page to be updated is a data page corresponding to a data page requested to be accessed by a client and the data page to be updated is not included in the cache region, reading the data page from a shared memory to the cache region before executing the step of updating the data page to be updated by playing back the target log data.
13. A method for obtaining log index data is characterized by comprising the following steps:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
14. A log playback apparatus, comprising:
the data page identification information acquisition unit is used for acquiring the data page identification information of the data page to be updated;
a target log position information obtaining unit, configured to obtain target log position information from log index data according to data page identification information of the data page to be updated, where the log index data is used to describe a correspondence between the data page identification information and the log position information;
the target log data determining unit is used for determining target log data corresponding to the data page to be updated according to the target log position information;
and the playback unit is used for updating the data page to be updated by playing back the target log data.
15. An electronic device, comprising:
a processor;
a memory for storing a program of a log playback method, the apparatus performing the following steps after being powered on and running the program of the log playback method by the processor:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
16. A storage device storing a program of a log playback method, the program being executed by a processor to perform the steps of:
acquiring data page identification information of a data page to be updated;
acquiring target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
17. An apparatus for obtaining log index data, comprising:
the data acquisition unit is used for acquiring metadata information of redo log data generated by the main database;
an information obtaining unit, configured to obtain, from the metadata information, data page identification information of at least one data page, and obtain log location information of log data corresponding to the data page in the redo log data;
and the log index data obtaining unit is used for establishing the corresponding relation between the data page identification information and the log position information to obtain the log index data.
18. An electronic device, comprising:
a processor;
a memory for storing a program of an obtaining method of log index data, the apparatus performing the following steps after being powered on and running the program of the obtaining method of log index data by the processor:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
19. A storage device, characterized by storing a program of an acquisition method of log index data, the program being executed by a processor and executing the steps of:
obtaining metadata information of redo log data generated by a main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010076416.7A CN111324665B (en) | 2020-01-23 | 2020-01-23 | Log playback method and device |
| PCT/CN2021/073003 WO2021147935A1 (en) | 2020-01-23 | 2021-01-21 | Log playback method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010076416.7A CN111324665B (en) | 2020-01-23 | 2020-01-23 | Log playback method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111324665A true CN111324665A (en) | 2020-06-23 |
| CN111324665B CN111324665B (en) | 2023-06-27 |
Family
ID=71173128
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010076416.7A Active CN111324665B (en) | 2020-01-23 | 2020-01-23 | Log playback method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111324665B (en) |
| WO (1) | WO2021147935A1 (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112596953A (en) * | 2020-12-25 | 2021-04-02 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
| WO2021147935A1 (en) * | 2020-01-23 | 2021-07-29 | 阿里巴巴集团控股有限公司 | Log playback method and apparatus |
| CN113918535A (en) * | 2020-07-08 | 2022-01-11 | 腾讯科技（深圳）有限公司 | Data reading method, device, equipment and storage medium |
| WO2022048416A1 (en) * | 2020-09-04 | 2022-03-10 | 阿里云计算有限公司 | Operation request processing method and apparatus, and device, and readable storage medium, and system |
| CN114816247A (en) * | 2022-04-13 | 2022-07-29 | 阿里云计算有限公司 | Logic data acquisition method and device |
| CN117555884A (en) * | 2024-01-12 | 2024-02-13 | 腾讯科技（深圳）有限公司 | Method, device and equipment for reading data page and readable storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114510494B (en) * | 2022-04-18 | 2022-08-02 | 成方金融科技有限公司 | Log playback method and device and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120011106A1 (en) * | 2010-07-07 | 2012-01-12 | Microsoft Corporation | Shared log-structured multi-version transactional datastore with metadata to enable melding trees |
| CN103412803A (en) * | 2013-08-15 | 2013-11-27 | 华为技术有限公司 | Data recovering method and device |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
| CN104899117A (en) * | 2015-06-17 | 2015-09-09 | 江苏师范大学 | Memory database parallel logging method for nonvolatile memory |
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| US20180144015A1 (en) * | 2016-11-18 | 2018-05-24 | Microsoft Technology Licensing, Llc | Redoing transaction log records in parallel |
| CN110442560A (en) * | 2019-08-14 | 2019-11-12 | 上海达梦数据库有限公司 | Method, apparatus, server and storage medium are recurred in a kind of log |
| CN110569214A (en) * | 2019-08-02 | 2019-12-13 | 杭州云纪网络科技有限公司 | Index construction method and device for log file and electronic equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7308465B2 (en) * | 2003-04-03 | 2007-12-11 | Microsoft Corporation | Playback graph for data synchronization |
| CN102012851B (en) * | 2010-12-20 | 2012-09-26 | 浪潮(北京)电子信息产业有限公司 | Continuous data protection method and server |
| CN111324665B (en) * | 2020-01-23 | 2023-06-27 | 阿里巴巴集团控股有限公司 | Log playback method and device |
-
2020
- 2020-01-23 CN CN202010076416.7A patent/CN111324665B/en active Active
-
2021
- 2021-01-21 WO PCT/CN2021/073003 patent/WO2021147935A1/en active Application Filing
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120011106A1 (en) * | 2010-07-07 | 2012-01-12 | Microsoft Corporation | Shared log-structured multi-version transactional datastore with metadata to enable melding trees |
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| CN103412803A (en) * | 2013-08-15 | 2013-11-27 | 华为技术有限公司 | Data recovering method and device |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
| CN104899117A (en) * | 2015-06-17 | 2015-09-09 | 江苏师范大学 | Memory database parallel logging method for nonvolatile memory |
| US20180144015A1 (en) * | 2016-11-18 | 2018-05-24 | Microsoft Technology Licensing, Llc | Redoing transaction log records in parallel |
| CN110569214A (en) * | 2019-08-02 | 2019-12-13 | 杭州云纪网络科技有限公司 | Index construction method and device for log file and electronic equipment |
| CN110442560A (en) * | 2019-08-14 | 2019-11-12 | 上海达梦数据库有限公司 | Method, apparatus, server and storage medium are recurred in a kind of log |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021147935A1 (en) * | 2020-01-23 | 2021-07-29 | 阿里巴巴集团控股有限公司 | Log playback method and apparatus |
| CN113918535A (en) * | 2020-07-08 | 2022-01-11 | 腾讯科技（深圳）有限公司 | Data reading method, device, equipment and storage medium |
| WO2022048416A1 (en) * | 2020-09-04 | 2022-03-10 | 阿里云计算有限公司 | Operation request processing method and apparatus, and device, and readable storage medium, and system |
| CN112596953A (en) * | 2020-12-25 | 2021-04-02 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
| CN112596953B (en) * | 2020-12-25 | 2023-08-29 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
| CN114816247A (en) * | 2022-04-13 | 2022-07-29 | 阿里云计算有限公司 | Logic data acquisition method and device |
| CN117555884A (en) * | 2024-01-12 | 2024-02-13 | 腾讯科技（深圳）有限公司 | Method, device and equipment for reading data page and readable storage medium |
| CN117555884B (en) * | 2024-01-12 | 2024-04-26 | 腾讯科技（深圳）有限公司 | Method, device and equipment for reading data page and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111324665B (en) | 2023-06-27 |
| WO2021147935A1 (en) | 2021-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111324665B (en) | Log playback method and device | |
| CN106815275B (en) | Method and equipment for realizing synchronization of main database and standby database through standby database | |
| CN107153644B (en) | Data synchronization method and device | |
| EP3944556A1 (en) | Block data access method and apparatus, and block data storage method and apparatus | |
| CN111414403B (en) | Data access method and device and data storage method and device | |
| CN111177143B (en) | Key value data storage method and device, storage medium and electronic equipment | |
| US20180276267A1 (en) | Methods and system for efficiently performing eventual and transactional edits on distributed metadata in an object storage system | |
| CN113051221A (en) | Data storage method, device, medium, equipment and distributed file system | |
| CN113918535B (en) | Data reading method, device, equipment and storage medium | |
| CN114676135A (en) | Data storage method, readable medium and electronic device | |
| CN113806301A (en) | Data synchronization method, device, server and storage medium | |
| CN112948389B (en) | MD 5-based database table data comparison method and device | |
| CN116414828A (en) | Data management method and related device | |
| CN111680030A (en) | Data fusion method and device, and data processing method and device based on meta information | |
| CN112527841A (en) | Stream data merging processing method and device | |
| CN115934583B (en) | Hierarchical caching method, device and system | |
| CN115640353A (en) | Data reading method, computer storage medium and electronic device | |
| CN117493282A (en) | Metadata management method based on file system and related equipment thereof | |
| US20050246385A1 (en) | Database-rearranging program, database-rearranging method, and database-rearranging apparatus | |
| CN114896276A (en) | Data storage method and device, electronic equipment and distributed storage system | |
| CN114816247A (en) | Logic data acquisition method and device | |
| CN109376214B (en) | Data processing method, device and system, computer equipment and readable medium | |
| CN112699129A (en) | Data processing system, method and device | |
| CN113111138A (en) | Data processing method, device, computing equipment and medium | |
| CN112632211A (en) | Semantic information processing method and equipment for mobile robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230630 Address after: Room 1-2-A06, Yungu Park, No. 1008 Dengcai Street, Sandun Town, Xihu District, Hangzhou City, Zhejiang Province Patentee after: Aliyun Computing Co.,Ltd. Address before: Box 847, four, Grand Cayman capital, Cayman Islands, UK Patentee before: ALIBABA GROUP HOLDING Ltd. |