CN111324665B - Log playback method and device - Google Patents
Log playback method and device Download PDFInfo
- Publication number
- CN111324665B CN111324665B CN202010076416.7A CN202010076416A CN111324665B CN 111324665 B CN111324665 B CN 111324665B CN 202010076416 A CN202010076416 A CN 202010076416A CN 111324665 B CN111324665 B CN 111324665B
- Authority
- CN
- China
- Prior art keywords
- log
- data
- data page
- updated
- obtaining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a log playback method, which comprises the following steps: acquiring data page identification information of a data page to be updated; obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information; determining target log data corresponding to the data page to be updated according to the target log position information; and updating the data page to be updated by playing back the target log data. According to the method, after the data page identification information of the data page to be updated is obtained, the target log position information corresponding to the data page identification information is obtained, and the target log data corresponding to the data page to be updated is determined according to the target log position information, so that the target log data corresponding to the data page to be updated is played back in a targeted mode, the data page to be updated is updated, and the speed of log playback can be increased.
Description
Technical Field
The application relates to the field of databases, in particular to a log playback method, a log playback device, electronic equipment and storage equipment. The application also relates to a method and a device for obtaining the log index data, electronic equipment and storage equipment.
Background
With the advent of the big data age, databases have been increasingly used, and particularly with the increasing popularity of cloud platforms, databases based on shared storage are widely used in various industries.
Currently, in order to maintain high availability of database services when a database is specifically applied, computing and storage separation are generally adopted, and services with high availability are provided based on a distributed architecture of shared storage. In popular terms, a main and multi-standby architecture is adopted to provide service, a main database is used as a readable and writable database node, namely a RW node, to provide service outwards, and data is written into a shared storage; the backup database is used as a readable node, namely an RO node, and the consistency of the data provided outwards and the data in the RW node is maintained through a log playback method based on a redo (redox) log generated by the RW node.
However, when the current backup database provides data outwards through a log playback method, a complete redo log file generated by the main database is generally required to be acquired, and then log playback is performed through a single process according to the acquired redo log file to ensure consistency of data provided between the main database and the backup database, which has the disadvantage that the backup database cannot update a data page in a targeted manner only when the data page is accessed, but the backup database needs to update the data page according to the acquired redo log file whenever the data page is accessed or not; on the other hand, when the backup database performs log playback, no matter whether the data page has the accessed requirement, the backup database needs to perform log playback according to the obtained redo log file, so that the problem of low playback speed and further indirect influence on the performance of the main database also exists.
In summary, the current log playback method has the problems of no pertinence and low playback speed.
Disclosure of Invention
The embodiment of the application provides a log playback method to solve the problems of no pertinence and low playback speed in the prior art.
The embodiment of the application provides a log playback method, which comprises the following steps:
acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
Optionally, the method comprises the following steps: judging whether the existing data page in the buffer area needs to be updated or not according to metadata information of the log data to be reworked; and if the existing data page in the cache region has the data page needing to be updated, setting the available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region.
Optionally, the method further comprises: and if the existing data page in the cache area does not exist the data page needing to be updated, generating log index data corresponding to the metadata information of the log data to be reworked.
Optionally, the acquiring the data page identification information of the data page to be updated includes:
if a transaction access request for a data page is acquired, inquiring whether the data page is available or not according to an available state identifier corresponding to the data page, and if the available state identifier corresponding to the data page is an unavailable state, taking the data page as the data page to be updated and acquiring data page identification information of the data page to be updated.
Optionally, the step of setting the available status identifier corresponding to the data page to be updated to an unavailable status includes: and setting the available state identifier corresponding to the data page to be updated into an unavailable state in a parallel or serial mode.
Optionally, the acquiring the data page identification information of the data page to be updated includes: acquiring a transaction access request message sent by a client; obtaining identification information of a data page requesting access from the transaction access request message; judging whether the data page requested to be accessed needs to be updated according to metadata information of the log data to be reworked, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring data page identification information of the data page to be updated.
Optionally, the log index data is obtained by the following method: obtaining metadata information of the redo log data generated by the main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the log data to be reworked; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Optionally, the method further comprises: and storing the log index data.
Optionally, the method comprises the following steps: obtaining identification information for identifying the log index data; and updating metadata information associated with the log index data according to the identification information.
Optionally, the method further comprises: and generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
Optionally, the method comprises the following steps: if the data page to be updated is not included in the cache area, the data page is read from the shared storage to the cache area before the step of updating the data page to be updated by playing back the target log data is executed.
Optionally, the method further comprises: and if the data page to be updated is a data page corresponding to the data page requested to be accessed by the client, and the cache area does not comprise the data page to be updated, reading the data page from the shared storage to the cache area before the step of updating the data page to be updated by playing back the target log data is executed.
The embodiment of the application also provides a method for obtaining the log index data, which comprises the following steps:
obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
The embodiment of the application also provides a log playback device, which comprises:
a data page identification information acquisition unit for acquiring data page identification information of a data page to be updated;
a target log position information obtaining unit, configured to obtain target log position information from log index data according to data page identification information of the data page to be updated, where the log index data is used to describe a correspondence between the data page identification information and the log position information;
A target log data determining unit, configured to determine target log data corresponding to the data page to be updated according to the target log position information;
and the playback unit is used for updating the data page to be updated by playing back the target log data.
The embodiment of the application also provides electronic equipment, which comprises:
a processor;
a memory for storing a program of a log playback method, the apparatus, after being powered on and running the program of the log playback method by the processor, performing the steps of:
acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
The embodiment of the application also provides a storage device, which stores a program of the log playback method, the program is run by a processor, and the following steps are executed:
Acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
The embodiment of the application also provides a device for obtaining log index data, which comprises:
the data acquisition unit is used for acquiring metadata information of the redo log data generated by the main database;
an information obtaining unit configured to obtain, from the metadata information, data page identification information of at least one data page, and log position information of log data corresponding to the data page in the redo log data;
and the log index data obtaining unit is used for establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
The embodiment of the application also provides electronic equipment, which comprises:
A processor;
a memory for storing a program of a log index data obtaining method, the apparatus being powered on and executing the program of the log index data obtaining method by the processor, and performing the steps of:
obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
The embodiment of the application also provides a storage device, which stores a program of the log index data obtaining method, the program is executed by a processor, and the following steps are executed:
obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Compared with the prior art, the application has the following advantages:
the embodiment of the application provides a log playback method, which comprises the following steps: acquiring data page identification information of a data page to be updated; obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information; determining target log data corresponding to the data page to be updated according to the target log position information; and updating the data page to be updated by playing back the target log data. According to the method, after the data page identification information of the data page to be updated is obtained, the target log position information corresponding to the data page identification information is obtained from the log index data, and the target log data corresponding to the data page to be updated is determined according to the target log position information, so that the target log data corresponding to the data page to be updated is played back in a targeted mode, and the data page to be updated is updated.
The embodiment of the application also provides a method for obtaining the log index data, which comprises the following steps: obtaining metadata information of the redo log data generated by the main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data. According to the method, after metadata information of the redo log data generated by the main database is obtained, data page identification information of at least one data page is obtained through analyzing the metadata information, log position information of log data corresponding to the data page summarized in the redo log data is obtained, and the corresponding relation between the data page identification information and the log position information is established to obtain log index data.
Drawings
Fig. 1 is a schematic diagram of an application scenario of a log playback method according to a first embodiment of the present application.
Fig. 2 is a flowchart of a log playback method according to the first embodiment of the present application.
Fig. 3 is a schematic structural diagram of log index data according to the first embodiment of the present application.
Fig. 4 is a schematic diagram of a first stage of a log playback method according to a first embodiment of the present application.
Fig. 5 is a schematic diagram of a second stage of the log playback method according to the first embodiment of the present application.
Fig. 6 is a schematic diagram of a parallel implementation log playback method according to the first embodiment of the present application.
Fig. 7 is a flowchart of a method for obtaining log index data according to a second embodiment of the present application.
Fig. 8 is a schematic diagram of a log playback apparatus according to a third embodiment of the present application.
Fig. 9 is a schematic diagram of an apparatus for obtaining log index data according to a fourth embodiment of the present application.
Fig. 10 is a schematic diagram of an electronic device according to a fifth embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is, however, susceptible of embodiment in many other ways than those herein described and similar generalizations can be made by those skilled in the art without departing from the spirit of the application and the application is therefore not limited to the specific embodiments disclosed below.
In order to enable those skilled in the art to better understand the scheme of the present application, a specific application scenario of an embodiment thereof will be described in detail below based on the log playback method provided in the present application. The log playback method can be particularly applied between a main database and a standby database of a distributed database based on shared storage, and further can be applied to the standby database of the distributed database so as to ensure consistency of data provided by the standby database to a client and data of the main database. Fig. 1 is a schematic application scenario diagram of a log playback method according to a first embodiment of the present application.
In a specific implementation, in a distributed database based on shared storage, a main database 101 is used as a readable and writable node, when the content of a certain data page in a cache area of the main database 101 is changed and the main database 101 does not refresh the change to shared storage, in order to ensure consistency between the content of the data page provided by a standby database 102 to a client 103 and the content in the cache area of the main database 101, when the standby database 102 acquires an access request of the client 103 for the data page, the standby database 102 takes the data page as a data page to be updated and acquires data page identification information of the data page to be updated; then, the backup database 102 obtains target log position information from log index data according to data page identification information of a data page to be updated, wherein the log index data is used for describing a corresponding relation between the data page identification information and the log position information; the backup database 102 determines target log data corresponding to the data page to be updated according to the obtained target log position information, and updates the data page to be updated by revising the target log data; after the backup database 102 updates the data page to be updated, the contents of the data page to be updated are provided to the client 103.
The client is a computing device for acquiring data from a database, which can be a traditional computer device or a mobile terminal device, such as a mobile phone, a tablet computer and the like, wherein the database can be a main database or a standby database of a distributed database.
The master database refers to a database node which is used as a readable and writable node to provide data service to clients in a distributed database based on shared storage, wherein the master database can also be called RW node, for example, the master database can be a master database node in various commonly used distributed databases.
The backup database is a database node which is used as a readable node to provide data service for clients in a distributed database based on shared storage, shares a bottom data storage space with a main database, and ensures the consistency of data in a cache region of the main and backup database nodes based on a redo log file generated by the main database, wherein the backup database can also be called an RO node or a slave database node.
The core idea of the Redo (redox) log file, also called Write-Ahead log (Write-Ahead log), is that modifications to the data page must occur after these modifications have been recorded in the Redo log file, i.e. the actual modification operations can be performed on the data page after the log record describing the data change has been flushed to shared storage.
The data page is a carrier of data stored in the database, and can also be called a data block or a data file; the data may specifically take the form of a data table, a data index, etc.
The data page identification information is information for uniquely identifying a data page, and may specifically be a string of character string information, for example, may be a PageID.
In the above, three entities of the main database 101, the standby database 102 and the client 103 are taken as examples to illustrate an application scenario of the log playback method provided in the first embodiment of the present application, and in a specific implementation, the method may be applied only between the main database 101 and the standby database 102, without triggering to implement the log playback method provided in the first embodiment of the present application after the standby database 102 obtains the transaction access request of the client 103.
For example, when the content of a certain data page in the cache area of the main database 101 is changed and the main database 101 has not refreshed the change to the shared storage, in order to ensure consistency between the content in the cache area of the backup database 102 and the content in the cache area of the main database 101, the backup database 102 obtains the data page with the data change in the cache area of the main database 101 from the metadata information of the redo log data sent by the main database 101, and uses the data page as the data page to be updated, and obtains the data page identification information of the data page to be updated; then, the backup database 102 obtains target log position information from the log index data according to the data page identification information of the data pages to be updated, determines target log data corresponding to the data pages to be updated according to the obtained target log position information, and updates the data pages to be updated by playing back the target log data so as to ensure consistency of the content in the cache area of the backup database 102 and the content in the cache area of the main database 101.
It should be noted that the above application scenario is only some specific embodiments of the log playback method described in the present application, and the purpose of the application scenario embodiments is to facilitate understanding of the log playback method of the present application, and is not limited to the log playback method of the present application.
Fig. 2 is a flowchart of a log playback method according to the first embodiment of the present application, and the log playback method according to the first embodiment of the present application is described in detail below with reference to fig. 2.
Step S201, acquiring data page identification information of a data page to be updated.
Specifically, the log playback method provided in the first embodiment of the present application may be applied to a backup database of a distributed database based on shared storage, and when the method is implemented, first, a data page to be updated in the backup database needs to be determined, that is, a data page waiting for log playback in the backup database to keep its content consistent with that in a cache area of a main database, and the method for determining a data page to be updated is described below.
In a distributed database based on shared storage, all the change operations for the data page need to be recorded in a redo log file before being actually applied to the data page, and when the change operations for the data page are recorded in the redo log file, the metadata information of the redo log file is correspondingly changed. Therefore, after obtaining the metadata information of the redo log file, the standby database can obtain the metadata information of the log data to be redone which is not subjected to the redo operation from the metadata information, and analyze the data pages which are changed in the main database from the metadata information of the log data to be redone, namely changed data pages, and obtain the data page identification information of the data pages; after obtaining the changed data page in the master database, a more straightforward method is: and directly taking all the acquired changed data pages as data pages to be updated. That is, the determination method of the data page to be updated includes: and acquiring metadata information of the log data to be reworked, analyzing the metadata information to obtain a data page which is changed in a main database, and taking the data page which is changed in the main database as the data page to be updated, wherein the log data to be reworked is the log data to be reworked waiting for the reworked operation.
MetaData (MetaData) information of the redo log file is data describing log data in the redo log file, or may be structural data for providing information related to the log data in the redo log file. The use purpose is that: identifying and evaluating relevant log data in the redo log file, and tracking changes in the log data during use; it has the following characteristics: 1. once the metadata information is established, it can be shared; 2. metadata information is a coding scheme that is different from traditional data coding schemes, and is specifically used to create a framework that allows computing devices to understand for the purpose of re-doing log data in log files.
For example, the metadata information may include a data type, an action corresponding to the data type, and data page information. The data type can be used for identifying the data of the btree type or the data of the soap type of the log data; actions corresponding to the data types can be used for identifying operations corresponding to the data types, for example, actions such as adding, deleting or splitting; the data page information may include location information of the data page, that is, db. In practice, a piece of metadata information may contain multiple data pages of information, e.g., a btree index split may involve three data pages; the data pages involved need to be locked in sequence according to different data types and actions when the metadata information is analyzed, wherein the locking process can be called mini transaction to ensure the atomicity of modification for the data pages. When the log index data is generated and the search is performed based on the log index data, whether the related page is locked in the mini transaction is required to be judged.
In a first embodiment of the present application, in order to increase a log playback speed, reduce a log playback cost, and pertinently perform log playback, the method for determining a data page to be updated includes: judging whether the existing data page in the buffer area needs to be updated or not according to metadata information of the log data to be reworked; and if the existing data page in the cache region has the data page needing to be updated, setting the available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region. The step of setting the available state identifier corresponding to the data page to be updated as an unavailable state includes: and setting the available state identifier corresponding to the data page to be updated into an unavailable state in a parallel or serial mode.
In addition, the method further comprises: and if the existing data page in the cache area does not exist the data page needing to be updated, generating log index data corresponding to the metadata information of the log data to be reworked. That is, if the data page in the main database, in which the update has occurred, does not exist in the buffer of the backup database, then the log playback operation does not need to be performed in the backup database, and the backup database only needs to generate log index data corresponding to the obtained metadata information of the log data to be reworked in the buffer thereof.
Fig. 4 is a schematic diagram of a first stage of a log playback method according to a first embodiment of the present application. As can be seen from fig. 4, considering that existing data pages in the buffer area of the standby database are not all accessed by the client at any time, after the standby database obtains metadata information of log data to be reworked, log playback operation is not directly performed, but a corresponding available state identifier is maintained for each existing data page in parallel or in series in the buffer area of the standby database, where the available state identifier is used to identify whether the content of the data page can be directly used; meanwhile, according to the metadata information, log index data is generated in the backup database, and detailed description of the log index data is described in detail in the subsequent steps, which will not be repeated here.
After determining the data page to be updated in the backup database, the acquiring the data page identification information corresponding to the data page to be updated includes: if a transaction access request for a data page is acquired, inquiring whether the data page is available or not according to the available state identification of the data page, and if the available state identification corresponding to the data page is an unavailable state, taking the data page as the data page to be updated, and acquiring the data page identification information of the data page to be updated.
That is, after the standby data receives the transaction access request of the client, the data page accessed by the transaction request can be resolved from the transaction access request; if the data page is identified as unavailable by the corresponding available status identifier in the cache area of the standby database, the standby database may take the data page as the data page to be updated and obtain the data page identification information of the data page to be updated, where the related description about the transaction in the database is not repeated herein because of the detailed description in the prior art.
The foregoing is a first method for obtaining data page identification information of a data page to be updated according to the first embodiment of the present application, and according to the foregoing description, it is known that the method specifically determines the data page to be updated in a targeted manner by maintaining an available state identifier in a cache area of a standby database, and if the data page has a corresponding transaction access request, performs log playback on the data page; if the data page has no corresponding transaction access, the data page is not subjected to log playback, and the method can reduce the access amount to storage, thereby reducing the log playback cost and increasing the log playback speed; in addition, between the main database and the standby database, the data page to be updated can be determined only by transmitting the metadata information of the redo log file, and subsequent log playback processing is performed, so that the transmission of the complete redo log file is not required, and the data volume of network transmission between the main database and the standby database can be reduced.
In addition, the first embodiment of the present application further provides a method for acquiring data page identification information of a data page to be updated, which specifically includes: acquiring a transaction access request message sent by a client; obtaining identification information of a data page requesting access from the transaction access request message; judging whether the data page requested to be accessed needs to be updated according to metadata information of the log data to be reworked, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring data page identification information of the data page to be updated.
That is, describing the determination process of the data page to be updated from another angle, specifically, after the standby database receives the transaction access request of the client, the standby database obtains the identification information of the data page requesting access from the transaction access request; and then, the backup database judges whether the data page needs to be updated or not according to the metadata information of the log data to be reworked, namely, whether the content of the data page is changed or not in a cache area of the main database, if so, the data page needs to be updated, the backup database takes the data page which is requested to be accessed as the data page to be updated, and acquires the data page identification information of the data page to be updated.
It should be noted that, in implementation, one or more of the above methods may be used simultaneously to determine the data page to be updated and obtain the data page identification information of the data page to be updated, and, of course, with the continuous progress of the technology, other methods or a method obtained by expanding the above methods may also be used to determine the data page to be updated and obtain the data page identification information of the data page to be updated.
The above describes how to determine the data page to be updated and obtain the data page identification information of the data page to be updated in detail, and it can be known from the above description that the log playback method provided by the first embodiment of the present application can determine the data page to be updated in a targeted manner, thereby reducing the access amount to the network and the storage, reducing the log playback cost, and increasing the log playback speed.
Step S202, obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information.
After the data page identification information of the data page to be updated is obtained in step S201, subsequent log playback processing may be performed according to the data page identification information, that is, the target log position information may be obtained from the log index data according to the data page identification information of the data page to be updated.
Before introducing the step S202, a detailed description is first given of a method for obtaining log index data used in the step, as shown in fig. 3, which is a schematic diagram of a structure of log index data provided in the first embodiment of the present application, where the log index data may be specifically obtained by the following method: obtaining metadata information of the redo log data generated by the main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the log data to be reworked; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
After obtaining the log index data, further comprising: and storing the log index data. Wherein, while storing the log index data, it includes: obtaining identification information for identifying the log index data; and updating metadata information associated with the log index data according to the identification information. In addition, the method further comprises the steps of: and generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
In the first embodiment of the present application, the log playback method specifically includes two stages, where the first stage has been described in the step S201, that is, in the buffer area of the main database and the buffer area of the backup database, log index data is continuously generated according to metadata information of the log file to be reworked generated by the main data; meanwhile, in the backup database, the method further comprises the following steps: and updating the available state identification of the data page in the buffer area of the backup database according to the metadata information of the log file to be redone.
The generation process of the log index data specifically includes analyzing and obtaining data page identification information of at least one data page with data change in a main database according to metadata information of a redo log file generated by the main database, obtaining log position information of log data corresponding to the data page in the log data to be redo, and then obtaining the log index data by establishing a corresponding relation between the data page identification information and the log position information.
The log location (LSN, log Sequence Number) information is information of a location for recording log data in the redo log file, and has a monotonically increasing attribute, and in a specific implementation, the log location information may be 64-bit character string information.
As can be seen from the above description, the log index data is specifically a correspondence relationship between the data page identification information and the log location information, that is, one piece of log index data is formed by the data page identification information and the log location information, for example, may be formed by the PageID and the LSN. In the first embodiment of the present application, the log index data is stored in a hash table, the data page identification information is a key value of the hash table, and the log location information is a data value. In addition, in order to save storage space, in the obtained log index data, the log position information may be split into prefix information and suffix information, where the prefix information is used to describe which data page the log index file is an index file for, and the suffix information is a monotonically increasing sequence number.
It should be noted that, after the log index data is generated in the buffer area of the main database, the log index data may be written into an active table (active table) first, as shown in fig. 3, after one active table is full of data, the active table full of data is set as an inactive table (inactive table), and the log index data is waited to be written into the shared storage, so as to increase the retrieval speed of the log index data.
In addition, when log index data is dropped, i.e. written into the shared storage, the identification information of the log index file corresponding to the log index data can be correspondingly generated, and the metadata information associated with the log index data, i.e. the metadata information of the log index file, is updated according to the identification information; meanwhile, a filter for further increasing the retrieval speed for the log index data, for example, a bloom filter (bloom filter) corresponding to the log index data is generated correspondingly to the log index file. How to generate the filter is not described in detail herein because of the prior art.
It should be noted that, when the log index data is dropped to obtain the identification information of the log index file, the metadata information of the log index file is updated, because in the database architecture based on shared storage, one master database node may simultaneously correspond to a plurality of backup database nodes, and each backup database node may generate, in its cache area, log index data corresponding to the content in the cache area of the master database node according to the metadata information of the redone log data generated by the master database node, so as to increase the utilization rate of the cache area, when a part of the log index data, such as the log index data in the inactive table of the full data, in the master database node is written into the shared storage, the backup database node may discard the corresponding log index data in the cache area of the backup database node at an appropriate time according to the metadata information of the updated log index file, so as to increase the utilization rate of the cache area. As can be seen from fig. 3, in order to increase the retrieval speed of the log index data and improve the utilization rate of the buffer, the structure of the log index data provided in the first embodiment of the present application may specifically include a log index file, a bloom filter corresponding to the log index file, and metadata information corresponding to the log index file.
It should be noted that the above is only one embodiment of log index data provided in the first embodiment of the present application, and other methods may be used to generate the log index data and set the log index data to other data structures, for example, the log index data is implemented by LSM-Tree, which is not described herein.
After log index data are generated in a buffer area of the standby database, when data page identification information of a data page to be updated is obtained and the data page to be updated needs to be updated through log playback, target log position information corresponding to the data page to be updated can be obtained through searching in the obtained log index data through the data page identification information of the data page to be updated.
Step S203, determining target log data corresponding to the data page to be updated according to the target log position information.
After the target log position information corresponding to the data page to be updated is obtained through S202, positioning can be performed in the redo log file generated in the master database according to the target log position information, so as to determine the target log data corresponding to the data page to be updated.
Step S204, updating the data page to be updated by playing back the target log data.
Obtaining target log data corresponding to a data page to be updated in step S203, and by playing back the target log data, updating the data page to be updated,
fig. 5 is a schematic diagram of a second stage of the log playback method according to the first embodiment of the present application. As can be seen from fig. 5, when performing log playback, if the data page to be updated is not included in the buffer, the data page may be read from the shared memory to the buffer before the step of updating the data page to be updated by playing back the target log data is performed.
That is, if the data page to be updated is not in the buffer area of the backup database, the dropped content of the data page with the update needs to be read from the shared storage into the buffer area of the backup database, and then the data page to be updated is updated by performing log playback through the target log position information acquired by the log index data and the target log data acquired from the redo log file.
In addition, the method further comprises: and if the data page to be updated is a data page corresponding to the data page requested to be accessed by the client, and the cache area does not comprise the data page to be updated, reading the data page from the shared storage to the cache area before the step of updating the data page to be updated by playing back the target log data is executed.
That is, only when a transaction access request message sent by a client is received and a data page in the transaction access request message is a data page that needs to be updated and the data page that needs to be updated does not yet exist in a buffer of the standby database, the data page that needs to be updated needs to be read from the shared storage into the buffer before the log playback operation is performed.
It should be noted that, since the log playback method provided in the first embodiment of the present application may perform log playback with pertinence through the log index data, in a specific implementation, the method may not be limited to a method for performing log playback by a single process, but may perform log playback in parallel. Fig. 6 is a schematic diagram of a method for parallel implementation of log playback according to the first embodiment of the present application, and as can be seen from the content of fig. 6, when the method for log playback described in the present application is implemented, a plurality of transaction access requests sent by a client may be received in parallel, and data pages to be updated in a cache area of a standby database may be updated in parallel, so as to increase the speed of log playback.
In summary, the log playback method provided in the first embodiment of the present application includes: acquiring data page identification information of a data page to be updated; obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information; determining target log data corresponding to the data page to be updated according to the target log position information; and updating the data page to be updated by playing back the target log data. According to the method, after the data page identification information of the data page to be updated is obtained, the target log position information corresponding to the data page identification information is obtained from the log index data, and the target log data corresponding to the data page to be updated is determined according to the target log position information, so that the target log data corresponding to the data page to be updated is played back in a targeted mode, and the data page to be updated is updated.
In the above description, a log playback method is provided, corresponding to the log playback method provided in the first embodiment of the present application, and the second embodiment of the present application further provides a log index data obtaining method, please refer to fig. 7, which is a flowchart of a log index data obtaining method provided in the second embodiment of the present application, wherein part of the steps are described in detail in the first embodiment of the present application, so that the description herein is relatively simple, and the relevant point is referred to the part of the description in the log index data obtaining method provided in the first embodiment of the present application, and the processing procedure described below is only schematic.
Step S701, obtaining metadata information of the redo log data generated by the master database.
Step S702, obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data.
Step S703, establishing a correspondence between the data page identification information and the log position information, to obtain the log index data.
Optionally, the method further comprises: and storing the log index data.
Optionally, the method comprises the following steps: obtaining identification information for identifying the log index data; and updating metadata information associated with the log index data according to the identification information.
Optionally, the method further comprises: and generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
In correspondence with a log playback method provided in the first embodiment of the present application, a log playback device is further provided in the third embodiment of the present application, please refer to fig. 8, which is a schematic diagram of the log playback device provided in the third embodiment of the present application, and since the device embodiment is substantially similar to the method embodiment, the description is relatively simple, and the relevant point is referred to the partial description of the method embodiment, and the device embodiment described below is merely illustrative. The log playback device provided in the third embodiment of the present application includes the following parts:
a data page identification information acquisition unit 801 for acquiring data page identification information of a data page to be updated.
A target log position information obtaining unit 802, configured to obtain target log position information from log index data according to the data page identification information of the data page to be updated, where the log index data is used to describe a correspondence between the data page identification information and the log position information.
And a target log data determining unit 803, configured to determine target log data corresponding to the data page to be updated according to the target log position information.
And a playback unit 804, configured to update the data page to be updated by playing back the target log data.
Optionally, the apparatus includes: the state setting unit is used for judging whether the existing data page in the buffer area needs to be updated or not according to the metadata information of the log data to be reworked; and if the existing data page in the cache region has the data page needing to be updated, setting the available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region.
Optionally, if there is no data page to be updated in the existing data page in the buffer area, log index data corresponding to metadata information of the log data to be reworked is generated.
Optionally, the data page identification information obtaining unit is specifically configured to query whether the data page is available according to an available state identifier corresponding to the data page when a transaction access request for the data page is obtained, and if the available state identifier corresponding to the data page is an unavailable state, take the data page as the data page to be updated, and obtain data page identification information of the data page to be updated.
Optionally, the step of setting the available status identifier corresponding to the data page to be updated to an unavailable status includes: and setting the available state identifier corresponding to the data page to be updated into an unavailable state in a parallel or serial mode.
Optionally, the apparatus includes: the data page to be updated obtaining unit is used for obtaining the transaction access request message sent by the client; obtaining identification information of a data page requesting access from the transaction access request message; judging whether the data page requested to be accessed needs to be updated according to metadata information of the log data to be reworked, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring data page identification information of the data page to be updated.
Optionally, the apparatus includes: the log index data obtaining unit is used for obtaining metadata information of the redo log data generated by the main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the log data to be reworked; and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Optionally, the playback unit is specifically configured to determine whether the buffer area includes the data page to be updated, and if not, read the data page from the shared storage to the buffer area before executing the step of updating the data page to be updated by playing back the target log data.
Optionally, the playback unit is further configured to determine whether the data page to be updated is a data page corresponding to a data page requested to be accessed by the client, and whether the data page to be updated is included in the cache area, and if the data page to be updated is a data page corresponding to a data page requested to be accessed by the client, and the cache area does not include the data page to be updated, read the data page from the shared storage to the cache area before the step of updating the data page to be updated by playing back the target log data is performed.
In correspondence with the method for obtaining log index data provided in the first embodiment of the present application, the fourth embodiment of the present application further provides a device for obtaining log index data, please refer to fig. 9, which is a schematic diagram of the device for obtaining log index data provided in the fourth embodiment of the present application, and since the device embodiment is substantially similar to the method embodiment, the description is relatively simple, and the relevant points refer to a part of the description of the method embodiment, and the device embodiment described below is merely illustrative. The device for obtaining log index data provided in the fourth embodiment of the present application includes the following parts:
A data obtaining unit 901, configured to obtain metadata information of the redo log data generated by the master database.
An information obtaining unit 902, configured to obtain data page identification information of at least one data page from the metadata information, and obtain log position information of log data corresponding to the data page in the redo log data.
A log index data obtaining unit 903, configured to establish a correspondence between the data page identification information and the log position information, and obtain the log index data.
In correspondence with the log playback method provided in the first embodiment of the present application, the fifth embodiment of the present application further provides an electronic device, please refer to fig. 10, which is a schematic diagram of an electronic device provided in the fifth embodiment of the present application, and since the electronic device embodiment is substantially similar to the method embodiment, the description is relatively simple, and the relevant points refer to a part of the description of the method embodiment, and the electronic device embodiment described below is merely illustrative. An electronic device provided in a fifth embodiment of the present application includes:
a processor 1001;
a memory 1002 for storing a program of a log playback method, the apparatus, after being powered on and running the program of the log playback method by the processor, performs the steps of:
Acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
In correspondence with a log playback method provided in the first embodiment of the present application, the sixth embodiment of the present application further provides a storage device, and since the storage device embodiment is substantially similar to the method embodiment, the description is relatively simple, and the relevant points are merely referred to in the partial description of the method embodiment, and the storage device embodiment described below is merely illustrative. A storage device provided in a sixth embodiment of the present application stores a program of a log playback method, the program being executed by a processor to perform the steps of:
acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is used for describing the corresponding relation between the data page identification information and the log position information;
Determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
In correspondence with the method for obtaining log index data provided in the second embodiment of the present application, the seventh embodiment of the present application further provides another electronic device, whose structure is substantially similar to that of the electronic device described in the fifth embodiment of the present application, and since the electronic device embodiment is substantially similar to the method embodiment, the description is relatively simple, and the relevant points refer to the part of the description of the method embodiment, and the electronic device embodiment described below is merely illustrative. An electronic device provided in a seventh embodiment of the present application includes:
a processor;
a memory for storing a program of a log index data obtaining method, the apparatus being powered on and executing the program of the log index data obtaining method by the processor, and performing the steps of:
obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
And establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
In correspondence with the method for obtaining log index data provided in the second embodiment of the present application, the eighth embodiment of the present application further provides a storage device, and since the storage device embodiment is substantially similar to the method embodiment, the description is relatively simple, and the relevant points refer to a part of the description of the method embodiment, and the storage device embodiment described below is merely illustrative. A storage device according to an eighth embodiment of the present application stores a program of a log index data obtaining method, the program being executed by a processor to perform the steps of:
obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
While the preferred embodiment has been described, it is not intended to limit the invention thereto, and any person skilled in the art may make variations and modifications without departing from the spirit and scope of the present invention, so that the scope of the present invention shall be defined by the claims of the present application.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer readable media, as defined herein, does not include non-transitory computer readable media (transmission media), such as modulated data signals and carrier waves.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Claims (18)
1. A log playback method, comprising:
acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is obtained by the following method: obtaining metadata information of log data to be reworked generated by a main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the database in the log data to be reworked; establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data;
Determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
2. The log playback method as recited in claim 1, comprising:
judging whether the existing data page in the buffer area needs to be updated or not according to metadata information of the log data to be reworked;
and if the existing data page in the cache region has the data page needing to be updated, setting the available state identifier corresponding to the data page needing to be updated as an unavailable state in the cache region.
3. The log playback method as recited in claim 2, further comprising:
and if the existing data page in the cache area does not exist the data page needing to be updated, generating log index data corresponding to the metadata information of the log data to be reworked.
4. The log playback method as set forth in claim 2, wherein the acquiring the data page identification information of the data page to be updated includes:
if a transaction access request for a data page is acquired, inquiring whether the data page is available or not according to an available state identifier corresponding to the data page, and if the available state identifier corresponding to the data page is an unavailable state, taking the data page as the data page to be updated and acquiring data page identification information of the data page to be updated.
5. The method for log playback according to claim 2, wherein the step of setting the available status flag corresponding to the data page to be updated to an unavailable status includes:
and setting the available state identifier corresponding to the data page to be updated into an unavailable state in a parallel or serial mode.
6. The log playback method as set forth in claim 1, wherein the acquiring the data page identification information of the data page to be updated includes:
acquiring a transaction access request message sent by a client;
obtaining identification information of a data page requesting access from the transaction access request message;
judging whether the data page requested to be accessed needs to be updated according to metadata information of the log data to be reworked, if so, taking the data page requested to be accessed as the data page to be updated, and acquiring data page identification information of the data page to be updated.
7. The log playback method as recited in claim 1, further comprising:
and storing the log index data.
8. The log playback method as recited in claim 7, comprising:
obtaining identification information for identifying the log index data;
And updating metadata information associated with the log index data according to the identification information.
9. The log playback method as recited in claim 7, further comprising:
and generating a filter corresponding to the log index data, wherein the filter is used for increasing the retrieval speed of the log index data.
10. The log playback method as recited in claim 2, comprising:
if the data page to be updated is not included in the cache area, the data page is read from the shared storage to the cache area before the step of updating the data page to be updated by playing back the target log data is executed.
11. The log playback method as recited in claim 10, further comprising:
and if the data page to be updated is a data page corresponding to the data page requested to be accessed by the client, and the cache area does not comprise the data page to be updated, reading the data page from the shared storage to the cache area before the step of updating the data page to be updated by playing back the target log data is executed.
12. A method for obtaining log index data, comprising:
Obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
13. A log playback apparatus, comprising:
a data page identification information acquisition unit for acquiring data page identification information of a data page to be updated;
a target log position information obtaining unit, configured to obtain target log position information from log index data according to data page identification information of the data page to be updated, where the log index data is obtained by: obtaining metadata information of log data to be reworked generated by a main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the database in the log data to be reworked; establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data;
A target log data determining unit, configured to determine target log data corresponding to the data page to be updated according to the target log position information;
and the playback unit is used for updating the data page to be updated by playing back the target log data.
14. An electronic device, comprising:
a processor;
a memory for storing a program of a log playback method, the apparatus, after being powered on and running the program of the log playback method by the processor, performing the steps of:
acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is obtained by the following method: obtaining metadata information of log data to be reworked generated by a main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the database in the log data to be reworked; establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data;
Determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
15. A storage device storing a program of a log playback method, the program being executed by a processor to perform the steps of:
acquiring data page identification information of a data page to be updated;
obtaining target log position information from log index data according to the data page identification information of the data page to be updated, wherein the log index data is obtained by the following method: obtaining metadata information of log data to be reworked generated by a main database; obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the database in the log data to be reworked; establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data;
determining target log data corresponding to the data page to be updated according to the target log position information;
and updating the data page to be updated by playing back the target log data.
16. An apparatus for obtaining log index data, comprising:
the data acquisition unit is used for acquiring metadata information of the redo log data generated by the main database;
an information obtaining unit configured to obtain, from the metadata information, data page identification information of at least one data page, and log position information of log data corresponding to the data page in the redo log data;
and the log index data obtaining unit is used for establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
17. An electronic device, comprising:
a processor;
a memory for storing a program of a log index data obtaining method, the apparatus being powered on and executing the program of the log index data obtaining method by the processor, and performing the steps of:
obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
And establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
18. A storage device, characterized in that a program of an obtaining method of log index data is stored, the program being executed by a processor, the steps of:
obtaining metadata information of the redo log data generated by the main database;
obtaining data page identification information of at least one data page from the metadata information, and obtaining log position information of log data corresponding to the data page in the redo log data;
and establishing a corresponding relation between the data page identification information and the log position information to obtain the log index data.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010076416.7A CN111324665B (en) | 2020-01-23 | 2020-01-23 | Log playback method and device |
| PCT/CN2021/073003 WO2021147935A1 (en) | 2020-01-23 | 2021-01-21 | Log playback method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010076416.7A CN111324665B (en) | 2020-01-23 | 2020-01-23 | Log playback method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111324665A CN111324665A (en) | 2020-06-23 |
| CN111324665B true CN111324665B (en) | 2023-06-27 |
Family
ID=71173128
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010076416.7A Active CN111324665B (en) | 2020-01-23 | 2020-01-23 | Log playback method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111324665B (en) |
| WO (1) | WO2021147935A1 (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324665B (en) * | 2020-01-23 | 2023-06-27 | 阿里巴巴集团控股有限公司 | Log playback method and device |
| CN111930788B (en) * | 2020-09-04 | 2021-03-23 | 阿里云计算有限公司 | Processing method, device and equipment of operation request, readable storage medium and system |
| CN112596953B (en) * | 2020-12-25 | 2023-08-29 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
| CN114510494B (en) * | 2022-04-18 | 2022-08-02 | 成方金融科技有限公司 | Log playback method and device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103412803A (en) * | 2013-08-15 | 2013-11-27 | 华为技术有限公司 | Data recovering method and device |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
| CN104899117A (en) * | 2015-06-17 | 2015-09-09 | 江苏师范大学 | Memory database parallel logging method for nonvolatile memory |
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| CN110442560A (en) * | 2019-08-14 | 2019-11-12 | 上海达梦数据库有限公司 | Method, apparatus, server and storage medium are recurred in a kind of log |
| CN110569214A (en) * | 2019-08-02 | 2019-12-13 | 杭州云纪网络科技有限公司 | Index construction method and device for log file and electronic equipment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7308465B2 (en) * | 2003-04-03 | 2007-12-11 | Microsoft Corporation | Playback graph for data synchronization |
| US8412689B2 (en) * | 2010-07-07 | 2013-04-02 | Microsoft Corporation | Shared log-structured multi-version transactional datastore with metadata to enable melding trees |
| CN102012851B (en) * | 2010-12-20 | 2012-09-26 | 浪潮(北京)电子信息产业有限公司 | Continuous data protection method and server |
| US20180144015A1 (en) * | 2016-11-18 | 2018-05-24 | Microsoft Technology Licensing, Llc | Redoing transaction log records in parallel |
| CN111324665B (en) * | 2020-01-23 | 2023-06-27 | 阿里巴巴集团控股有限公司 | Log playback method and device |
-
2020
- 2020-01-23 CN CN202010076416.7A patent/CN111324665B/en active Active
-
2021
- 2021-01-21 WO PCT/CN2021/073003 patent/WO2021147935A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105122241A (en) * | 2013-03-15 | 2015-12-02 | 亚马逊科技公司 | Database system with database engine and separate distributed storage service |
| CN103412803A (en) * | 2013-08-15 | 2013-11-27 | 华为技术有限公司 | Data recovering method and device |
| CN104281506A (en) * | 2014-07-10 | 2015-01-14 | 中国科学院计算技术研究所 | Data maintenance method and system for file system |
| CN104899117A (en) * | 2015-06-17 | 2015-09-09 | 江苏师范大学 | Memory database parallel logging method for nonvolatile memory |
| CN110569214A (en) * | 2019-08-02 | 2019-12-13 | 杭州云纪网络科技有限公司 | Index construction method and device for log file and electronic equipment |
| CN110442560A (en) * | 2019-08-14 | 2019-11-12 | 上海达梦数据库有限公司 | Method, apparatus, server and storage medium are recurred in a kind of log |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021147935A1 (en) | 2021-07-29 |
| CN111324665A (en) | 2020-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111324665B (en) | Log playback method and device | |
| CN107153644B (en) | Data synchronization method and device | |
| US20110320532A1 (en) | Data operating method, system, client, and data server | |
| EP2863310B1 (en) | Data processing method and apparatus, and shared storage device | |
| US9418094B2 (en) | Method and apparatus for performing multi-stage table updates | |
| US20150254320A1 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| CN107193827B (en) | Idempotent control method and device for sub-base and sub-table | |
| CN111414403A (en) | Data access method and device and data storage method and device | |
| EP3944556A1 (en) | Block data access method and apparatus, and block data storage method and apparatus | |
| CN112214468B (en) | Small file acceleration method, device, equipment and medium for distributed storage system | |
| US20150169623A1 (en) | Distributed File System, File Access Method and Client Device | |
| CN112416880A (en) | Method and device for optimizing storage performance of mass small files based on real-time merging | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| US20180276267A1 (en) | Methods and system for efficiently performing eventual and transactional edits on distributed metadata in an object storage system | |
| CN113051221A (en) | Data storage method, device, medium, equipment and distributed file system | |
| CN111221814B (en) | Method, device and equipment for constructing secondary index | |
| CN114647658A (en) | Data retrieval method, device, equipment and machine-readable storage medium | |
| CN114428681A (en) | Method and device for expanding computing capacity of database system | |
| CN111399753B (en) | Method and device for writing pictures | |
| CN112948389A (en) | MD 5-based database table data comparison method and equipment | |
| CN109885552B (en) | Metadata dynamic management method of distributed file system and distributed file system | |
| CN113918535A (en) | Data reading method, device, equipment and storage medium | |
| CN114328437B (en) | Method, device, equipment and medium for rapidly deleting historical data | |
| CN114003607A (en) | Block chain data storage method and device | |
| CN117632864A (en) | File access processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230630 Address after: Room 1-2-A06, Yungu Park, No. 1008 Dengcai Street, Sandun Town, Xihu District, Hangzhou City, Zhejiang Province Patentee after: Aliyun Computing Co.,Ltd. Address before: Box 847, four, Grand Cayman capital, Cayman Islands, UK Patentee before: ALIBABA GROUP HOLDING Ltd. |
|
| TR01 | Transfer of patent right |