CN115576494B

CN115576494B - Data storage method and computing device

Info

Publication number: CN115576494B
Application number: CN202211343759.0A
Authority: CN
Inventors: 邓旭玥
Original assignee: XFusion Digital Technologies Co Ltd
Current assignee: XFusion Digital Technologies Co Ltd
Priority date: 2022-10-31
Filing date: 2022-10-31
Publication date: 2023-11-03
Anticipated expiration: 2042-10-31
Also published as: CN115576494A

Abstract

The embodiment of the application provides a data storage method and computing equipment, which are characterized in that when a transaction begins to be submitted, a data page is flushed, then a post-write type log is flushed, the post-write type log comprises a data modification log, and the data modification log comprises a data page modification position and a modification operation type. Because the position and the modification type of the modification data are recorded in the data modification log, the data before modification and the data after modification do not need to be recorded, and the data quantity corresponding to the position and the modification type of the modification data is smaller, the log quantity is reduced, and the IO quantity during data modification is reduced. And because the data page disc brushing is completed before the post-writing type log disc brushing, the problem of half page writing does not occur when the post-writing type log disc brushing is performed, and therefore a full page writing mechanism is not needed to be written, and the log data volume is further reduced.

Description

Data storage method and computing device

Technical Field

The present application relates to the field of data technologies, and in particular, to a data storage method and a computing device.

Background

In the process of executing the transaction by the database, the transaction log needs to be recorded, so that the durability of the transaction is ensured. The current method for recording transaction logs is mainly WAL (Write-ahead logging).

Under the condition of the traditional mechanical disk storage medium, the capability of random IO is far behind sequential IO, so that the Oracle, mySQL, postgreSQL database and other databases are used for solving the problem of random IO submitted by each transaction on the premise of ensuring the consistency of the databases, and the WAL technology is adopted to write the data modification content contained in the random dirty pages into the sequential log. The log is flushed when the transaction is submitted, and dirty pages are not required to be flushed.

However, since the data modification related to the transaction needs to be written into the log before the transaction is submitted, the log IO amount increases dramatically, and the database is likely to stop working due to the disk fullness caused by the log expansion.

Thus, how to reduce the log size is a currently important issue.

Disclosure of Invention

The embodiment of the application provides a data storage method and computing equipment, which are used for reducing the log quantity and achieving the purpose of improving the transaction processing performance of a database.

In a first aspect, the present application provides a data storage method, comprising:

when a transaction begins to be submitted, the data page is brushed;

and brushing the disk of the post-write type log, wherein the post-write type log comprises a data modification log, and the data modification log records the position and the modification type of modification data corresponding to the modification operation when the modification operation is carried out on the data page in the buffer zone of the memory.

According to the data storage method provided by the embodiment of the application, when the transaction begins to be submitted, the data page is flushed, and after the data page is flushed, the post-writing log is flushed. Because the post-write type log comprises the data modification log, when the data modification log records modification operation on the data page in the buffer area of the memory, the position and the modification type of the modification data corresponding to the modification operation are not required to record the data before modification and the data after modification, the log quantity of the data modification log is reduced, and the log quantity of the post-write type log is further reduced.

Optionally, the transaction includes a plurality of modification operations on the data page in the buffer of the memory;

when the transaction begins to be submitted, the data page is flushed, which specifically comprises the following steps:

and writing the data page in the buffer area of the disk into the disk when the first modification operation in the plurality of modification operations in the transaction starts to be written into the buffer area of the disk.

According to the data storage method provided by the embodiment, when a first modification operation in a plurality of modification operations in a transaction starts to be written into a buffer of a disk, a data page in the buffer of the disk is written into the disk, so that when the transaction starts to be submitted, the data page is flushed.

Optionally, the writing-after log brushing disc specifically includes:

when the data page in the buffer area of the memory is modified, writing the position of modified data corresponding to the modification operation and the modification type into the log buffer area of the memory to generate a data modification log;

and brushing the data modification log.

According to the data storage method provided by the embodiment, when the data page in the buffer area of the memory is modified, the position and the modification type of the modification data corresponding to the modification operation are written into the log buffer area of the memory to generate the data modification log, so that the modification data is determined according to the data modification log in the disk.

Optionally, the modifying operation includes at least one of an updating operation, a deleting operation, and an adding operation, and the data modifying log includes at least one of a transaction updating log, a transaction deleting log, and a transaction adding log;

when the modification operation is performed on the data page in the buffer area of the memory, writing the position of the modification data corresponding to the modification operation into the log buffer area of the memory to generate a data modification log, which specifically includes:

when the data page in the buffer area of the memory is updated, writing the position of update data corresponding to the update operation into the log buffer area of the memory to generate a transaction update log;

When deleting the data page in the buffer area of the memory, writing the position of the deleted data corresponding to the deleting operation into the log buffer area of the memory to generate a transaction deleting log;

when the data page in the buffer area of the memory is added, writing the position of the added data corresponding to the adding operation into the log buffer area of the memory to generate a transaction adding log.

According to the data storage method provided by the embodiment, the modification operation comprises at least one of an update operation, a delete operation and an add operation, when the update operation is performed, update data corresponding to the update operation is written into a log buffer area of a memory to generate a transaction update log, when the delete operation is performed, delete data corresponding to the delete operation is written into the log buffer area of the memory to generate a transaction delete log, and when the add operation is performed, add data corresponding to the add operation is written into the log buffer area of the memory to generate a transaction add log, so that the log corresponding to each operation is obtained.

Optionally, before the transaction begins to commit, the method further comprises:

when a data page in a buffer area of a memory is modified, calculating the position of modified data corresponding to the modification operation;

And locking the position of the modification data corresponding to the modification operation.

According to the data storage method provided by the embodiment, after the position of the modification data corresponding to the modification operation is calculated, the position corresponding to the modification data is locked, so that the data quantity of locking is reduced, and further performance consumption caused by locking is reduced.

Optionally, when performing a modification operation on a data page in a buffer area of the memory, calculating a location of modification data corresponding to the modification operation specifically includes:

when the data page in the buffer area of the memory is modified, the position of the modified data is calculated according to the end position of the last data of the modified data corresponding to the modified operation and the disk space occupied by the modified data.

According to the data storage method, when the data page in the buffer area of the memory is modified, the position of the modified data is calculated according to the end position of the last data of the modified data corresponding to the modification operation and the disk space occupied by the modified data, so that the position of the modified data is accurately obtained.

Optionally, when performing the modification operation on the data page in the buffer area of the memory, calculating the position of the modified data according to the end position of the last data of the modified data corresponding to the modification operation and the disk space occupied by the modified data specifically includes:

When the data page in the buffer area of the memory is updated, calculating the position of the updated data according to the end position of the last data of the updated data corresponding to the updated operation and the disk space occupied by the updated data;

when deleting the data page in the buffer area of the memory, calculating the position of the deleted data according to the end position of the last data of the deleted data corresponding to the deleting operation and the disk space occupied by the deleted data;

when the data page in the buffer area of the memory is added, the position of the added data is calculated according to the end position of the last data of the added data corresponding to the added operation and the disk space occupied by the added data.

According to the data storage method, when updating operation is performed on a data page in a buffer area of a memory, the position of the updated data is calculated according to the end position of the last data of the updated data corresponding to the updating operation and the disk space occupied by the updated data, when deleting operation is performed on the data page in the buffer area of the memory, the position of the updated data is calculated according to the end position of the last data of the deleted data corresponding to the deleting operation and the disk space occupied by the deleted data, and when adding operation is performed on the data page in the buffer area of the memory, the position of the added data is calculated according to the end position of the last data corresponding to the adding operation and the disk space occupied by the added data, and therefore the positions of the data corresponding to the updating operation, the deleting operation and the adding operation are obtained.

Optionally, the post-write log includes a transaction commit log or a transaction rollback log, the transaction commit log records data pages in a buffer of a memory that are committed to a buffer of a disk, and the transaction rollback log records data pages for which a modification operation has been performed before a failure operation;

the writing-after log brushing disc specifically comprises the following steps:

after flushing the data pages in the buffer area of the memory, writing the data pages submitted to the buffer area of the disk in the buffer area of the memory into the log buffer area of the memory, generating a transaction commit log, and flushing the transaction commit log;

or writing the data page which has executed the modification operation before the failure operation in the buffer area of the memory into the log buffer area of the memory, generating a transaction rollback log, and brushing the transaction rollback log.

According to the data storage method provided by the embodiment, after the data pages in the buffer area of the memory are flushed, the data pages submitted to the buffer area of the disk in the buffer area of the memory are written into the log buffer area of the memory, a transaction commit log is generated, the data pages which have been subjected to modification operation before failure operation in the buffer area of the memory are written into the log buffer area of the memory, and a transaction rollback log is generated, and then the post-write log can also comprise the transaction commit log or the transaction rollback log, so that the transaction commit record and the transaction rollback record can be obtained according to the post-write log.

Optionally, after the post-writing log is brushed, the method further includes:

obtaining a target log from a disk, playing back the target log, and judging whether the target log is the last log in a post-writing log;

if not, continuing to acquire a target log from the disk until the target log is the last log in the post-writing logs;

if yes, the control computer receives a request for modifying the data in the disk.

According to the data storage method provided by the embodiment, the target log can be obtained from the disk, and the target log is played back, so that the modification record of the data in the disk can be obtained according to the target log.

In a second aspect, the present application provides a computing device comprising a processor configured to perform the data storage method described above.

According to the data storage method provided by the application, when a transaction begins to be submitted, a data page is flushed, then a post-writing type log is flushed, the post-writing type log comprises a data modification log, and the data modification log comprises a data page modification position and a modification operation type. Because the position and the modification type of the modification data are recorded in the data modification log, the data before modification and the data after modification do not need to be recorded, and the data quantity corresponding to the position and the modification type of the modification data is smaller, the log quantity is reduced, and the IO quantity during data modification is reduced. And because the data page disc brushing is completed before the post-writing type log disc brushing, the problem of half page writing does not occur when the post-writing type log disc brushing is performed, and therefore a full page writing mechanism is not needed to be written, and the log data volume is further reduced.

Drawings

In order to clearly illustrate the application or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the application, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIGS. 1-3 are flow diagrams of various embodiments of a data storage method;

FIG. 4 is an architecture diagram of a data storage system.

Detailed Description

The terminology used in the description of the embodiments of the application herein is for the purpose of describing particular embodiments of the application only and is not intended to be limiting of the application.

For ease of understanding, related art terms related to the embodiments of the present application are explained and explained first.

Data tuple (tuple): from the user's perspective, in the database, each table includes multiple rows of data; from the database kernel implementation, each row of data, plus metadata that needs to be added to manage the row of data (e.g., whether the row of data is expired), constitutes a data tuple.

Transaction (Transaction): generally refers to what is to be done or what is done. In computer terminology, refers to one or more program execution units (units) that access and possibly update data items in a database. Transactions are typically caused by execution of user programs written in a database manipulation language or programming language (e.g., SQL, C++, or Java) and are defined by shapes such as begin transaction and end transaction statements (or function calls), etc. A transaction consists of a collective operation performed between the start of the transaction (begin transaction) and the end of the transaction (end transaction).

ACID (atom, consistency, isolation, persistence): characteristics of the transaction; atomicity refers to the fact that the transaction is either successful or failed, the system rolls back to a state before the transaction is started after the failure, and the situation that part of the transaction is successful and affects omission in the system and the other part of the transaction is failed does not exist; the consistency refers to that after the transaction is successfully executed, the system enters the next consistency state from the consistency before the execution; isolation refers to that the transaction is not influenced by other transactions during execution, and the influence generated in the process of executing the transaction is not perceived by other transactions; persistence refers to that after the transaction is successfully executed, the effect of the transaction persists in the system, and the effect of the transaction cannot disappear or be incomplete due to the problem of system faults (such as restarting of a server, crashing of a database process and the like).

Redo log (Redo log): after the transaction is successfully executed, the transaction modifies the data, and its modification effect may disappear (modification has not yet flushed the dirty page in the memory) due to a failure (e.g., a server restart, a database process crash, etc.). To avoid this problem, a redox log is introduced, recording transaction modification operation types (modification, deletion, addition, etc.), data tuple IDs to which the modification relates, the new data tuple itself (deletion type does not have this data), etc. Based on the redox log, the transaction modification operation can be redone, and the modification effect can be reappeared.

Undo log (rollback log): in the event that the transaction is not successfully executed, if the transaction ever modifies data, its modifying effect may remain in the system due to the operating system brushing dirty pages. To eliminate this reservation effect, an undo log is introduced, and modifications made by failed transactions can be cleared by executing the undo log in reverse.

Log log (commit log): a transaction commit history log records the state log of all transaction commits (or rollbacks) that have been executed in the system.

WAL: (Write-ahead logging): WAL refers to flushing a transaction log first and then flushing modified data pages, i.e., the transaction log is flushed before, at the time of transaction commit.

WBL: (Write-head logging, post-Write log): WBL refers to flushing the modified data page first and then flushing the transaction log, i.e., flushing the transaction log later, at the time of transaction commit.

Out-place update (not in place): refers to writing a new page of data to other locations instead of writing the new page of data at the original page when the page of data is written.

Brushing dirty pages: the flow of the operating system modifying data is as follows: (1) Reading data into a memory according to pages (the size is generally 4K) from a disk file; (2) Modifying a data page in a memory, wherein the data page is a dirty page; (3) this dirty page is written back to disk file. Step (3) is to brush dirty pages.

4K page atomic write: when data is read into the memory from the disk file and the disk file is written out from the memory, 4K is used as a reading and writing unit, namely one page. The operating system ensures that the data in the page is complete and consistent data at a certain moment when the whole 4K page is read and written every time, and the situation that the previous part of data in the 4K page is updated or older than other data does not exist, and the operation characteristic of writing the data is called 4K page atomic writing.

8K page half page write: the data page size of the database is 8K, and the database consists of two operating system pages, wherein the operating system can ensure 4K page atomic writing, but does not ensure that 2 4K page atomic writing modes write out one 8K page, and before the next page is written out, the operating system fails, so that half of data in the 8K page of the database, namely one 4K operating system page, stores new data, and the other half stores old data.

Full Page Write: when the database system modifies the 8K data page for the first time (before the data page is read from the disk file to the memory or the dirty page is brushed on the data page), the whole data page is written into the database transaction log before the modification, so that the log can store a clean data page, and the clean and consistent data page can be recovered from the log if a half-page writing problem occurs later. The Full page write increases the database log IO size.

Checkpoint (or manufacturing Checkpoint): the database (calling the operating system API) executes the operation of brushing dirty pages on all the modified data pages, records the operation in a database transaction log after the completion of brushing dirty pages, and writes information marking the current processing progress of the database in a contracted position, wherein the whole process is called a manufacturing check point (in this case, the checkpoint is a verb). The manufacture checkpointing information is recorded in the log, and the current process progress information written into the database at the contracted position is called a checkpoint (in this case, checkpoint is a noun). If the system crashes, the state of the database can be restored to the corresponding database processing progress of the check point by the disk data file and the check point data.

RTO (Recovery Time Objective, system recovery time): the database system is restarted due to a crash for some reason or is restarted as scheduled by maintenance, and the database system is restored to the state before the restart by replaying the redox logging operation after the restart. And restarting the database until the database is completely restored, wherein the state is the system restoration time. During this time, the database system cannot provide services to the outside, and in general, the shorter the system recovery time, the better.

Random IO: the read-write contents are scattered at different positions of the disk, and need to be searched back and forth.

Sequential IO: the read-write contents are stored in a block of the disk in a concentrated manner, and are sequentially read from front to back.

Dirty pages: when the memory data page and the disk data page are inconsistent, the memory page is called a dirty page.

Brushing dirty pages: the operation of writing a page of memory data to disk is referred to as flushing the page.

Transaction log: which is a separate file from the database file, stores all changes made to the database, and all records insert, update, delete, commit, rollback, and database schema changes.

In the process of executing the transaction by the database, the transaction log needs to be recorded, so that the durability of the transaction is ensured. There are two types of methods for logging transactions: WAL and WBL.

Under the condition of the traditional mechanical disk storage medium, the capability of random IO is far behind that of sequential IO, so that the Oracle, mySQL, postgreSQL database and other databases are used for solving the problem of random IO submitted by each transaction on the premise of ensuring the consistency of the databases, the WAL technology is adopted, the random dirty pages are written into sequential logs, and the log is utilized to brush the disks, so that the dirty pages do not need to be brushed when the transaction is submitted.

However, since data related to the transaction needs to be written into the log before the transaction is submitted, the log IO amount is increased rapidly, and the database is easy to stop working due to the fact that the disk is full caused by the expansion of the log.

In addition, the operating system can only guarantee 4K Page atomic writing, so that the Full Page Write mechanism is written for preventing the 8K Page half Page writing problem, and the mechanism solves the 8K Page half Page writing problem, but leads to larger log data volume.

In addition, when the log is written, the log is written in the order of generating the log by the database, but a plurality of transactions are executed in the database at the same time, in order to force the log to be arranged in the order of generating the log by the database, the log needs to be mutually exclusive locked so that the log is written in series, and therefore, the concurrency is not advantageous.

For WBL, when the data page is modified, the modified data page is not covered with the data page before modification, but the modified data page is written to other positions, and the data of the WBL is inconsistent with the instantaneous data in an abnormal scene. When the database is restarted and restored, in order to ensure that the data page modification is consistent with the data modification of the log record, more data page modification is required to be rolled off than the data modification of the log record, so that the transaction batch is submitted under an abnormal scene, more rollback operations are required to be carried out, and the RTO of the system is reduced.

Based on the above, the application provides a data storage method, when a transaction begins to be submitted, a data page is flushed, and then a WBL log is flushed, wherein the WBL log comprises a data modification log, and the data modification log comprises a data page modification position and a modification operation type. The existing WAL log includes pre-modification data and post-modification data, and then the log records the contents before the transaction modifies the database and the contents after the database is modified, namely, the old value and the new value are recorded. When the recovery is met, if the new value is not written, the recovery can be realized after the new value is found; if the previous write half fails, the old value can be recovered by finding the old value, the data modification log comprises the position and the modification type of the modification data, the data volume of the old value and the new value in the WAL log is larger, and the data volume corresponding to the position and the modification type of the modification data in the data modification log is smaller, so that the log volume is reduced, and the IO volume during data modification is reduced. And because the WBL log is flushed before the data page flushing is completed, half-page writing problem does not occur when the WBL log is flushed, and therefore a full-page writing mechanism is not needed to be written, and the log data volume is further reduced.

The data storage method provided by the embodiment of the application is described in detail below through a specific implementation manner.

Fig. 1 is a flowchart of a data storage method according to an embodiment of the present application. Referring to fig. 1, the data storage method provided by the embodiment of the application includes:

s101, when the transaction begins to be submitted, the data page is flushed.

A transaction includes a collective operation performed between the beginning of the transaction and the end of the transaction, then the transaction may include one or more operations that may include adding, updating, deleting, etc., data in memory. When the transaction comprises one operation, the transaction start can be executed for the operation start, the time of the transaction start can be the time of the operation start execution, and when the transaction comprises a plurality of operations, the transaction start can be executed for the first operation start, and the time of the transaction start can be the time of the first operation start execution.

Illustratively, the transaction may include one or more DML statements (Data Manipulation Language, data manipulation statement), and the collective operations performed between the beginning of the transaction and the end of the transaction may be the execution of one or more DML statements in the transaction. When the transaction includes a DML statement, the time at which the processor starts executing the DML statement may be the time at which the transaction starts; when a transaction includes a plurality of DML statements, the time at which the processor starts executing the first DML statement is taken as the time at which the transaction starts. The transaction may also include a DDL statement (Data Definition Language, data defining statement), and the collective operation performed between the beginning of the transaction and the end of the transaction may be the execution of the DDL statement, with the time at which the processor begins executing the DDL statement being the time at which the transaction begins. The transaction may further include a DCL statement (Data Control Language, data control statement), and then the collective operation performed between the start of the transaction and the transaction may be the execution of the DCL statement, and the time at which the processor starts executing the DCL statement is the time at which the transaction starts.

Transaction end includes normal transaction end and abnormal transaction end, normal transaction end means that all operations in the transaction are completed normally, and transaction end means that a certain operation in the transaction fails to execute.

It should be noted that when all operations in the transaction are normally completed, the transaction commit is performed, and the transaction commit refers to writing all operations in a buffer area of the memory into the buffer area in the disk, and when the transaction commit is performed, it may be determined that the transaction is ended, and the time when the transaction begins to commit may be taken as the time when the transaction is ended. When a transaction includes an operation, the time when the transaction begins to commit may be the time when the operation begins to write to the buffer of the disk, and when the transaction includes a plurality of operations, the time when the transaction begins to commit may be the time when the first operation in the transaction begins to write to the buffer of the disk.

When the execution of a certain operation in the transaction fails, the transaction rollback is carried out, wherein the transaction rollback refers to the occurrence of faults in the execution process of the certain operation, and all completed operations in the transaction are withdrawn. Then the transaction end may be determined when the transaction rollback is performed and the time at which the transaction begins to rollback may be taken as the time at which the transaction ends. The transaction includes a plurality of operations, and part of the operations are executed before some operation fails, the rollback of the transaction needs to cancel all the executed part of the operations, and the time when the transaction ends can be the time when the first operation in the executed part of the operations starts to rollback.

The processor may execute all operations in the transaction, commit the transaction after receiving a commit instruction sent by the user, or perform the transaction rollback after receiving a rollback instruction sent by the user, or automatically commit the transaction after executing all operations in the transaction, or automatically rollback after failing to execute a certain operation in the transaction. For example, when a transaction includes one or more DML statements, the transaction COMMIT may be performed after receiving a COMMIT statement sent by the user, or the transaction ROLLBACK may be performed after receiving a ROLLBACK statement sent by the user. When a transaction includes a DDL statement or a DCL statement, a transaction commit or a transaction rollback is automatically performed after the DDL statement or DCL statement is executed.

When the data page in the disk is required to be updated, the data page in the disk is not directly updated, but is read into the buffer area of the memory, and then the data page in the buffer area of the memory is updated, wherein the updating of the data page in the buffer area of the memory can be the updating of partial data or all data in the data page. The data pages in the buffer area of the memory are inconsistent with the content of the data pages in the disk due to updating, the data pages in the buffer area of the memory are written into the buffer area of the disk, the reading and writing times of the disk are reduced, and then the data pages in the buffer area of the disk are written into the disk, so that the data pages in the memory are consistent with the data pages in the disk, and therefore the data page brushing is realized.

For example, when a user needs to perform data addition on data in a disk, an addition instruction is sent to a processor, the processor reads a data page corresponding to the addition instruction from the disk, reads the data page corresponding to the addition instruction into a buffer pool of a memory, and then the processor adds the data in the data page in the buffer pool of the memory, and writes the data page after the data addition into the disk. When a user needs to update data in a disk, an update instruction is sent to a processor, the processor reads a data page corresponding to the update instruction from the disk, reads the data page corresponding to the update instruction into a buffer pool of a memory, and then the processor updates the data in the data page in the buffer pool of the memory and writes the data page after updating the data into the disk. When a user needs to delete data in a disk, a deleting instruction is sent to the processor, the processor reads a data page corresponding to the deleting instruction from the disk, the data page corresponding to the deleting instruction is read into a buffer pool of the memory, the data is deleted from the data page in the buffer pool of the memory by the processor, and the data page after the data deletion is written into the disk by the processor.

The transaction starting submitting refers to writing the operation starting to the buffer area of the memory into the buffer area of the disk, for example, when the data page in the buffer area of the memory is updated, writing the data page in the buffer area of the memory for data update into the buffer area of the disk; when data is added to the data page in the buffer area of the memory, writing the data page with the data added in the buffer area of the memory into the buffer area of the disk; when the data page in the buffer area of the memory is deleted, the data page in the buffer area of the memory, which is deleted, is written into the buffer area of the disk. For example, the fsync function may be used to synchronize pages of data in a buffer of memory into a buffer of disk.

When the transaction begins to commit, the data pages in the buffer area of the disk are written into the disk, so that when the transaction begins to commit, the data pages in the buffer area of the memory are written into the disk, and when the transaction begins to commit, the data pages are flushed.

S102, after the data page is flushed, flushing the WBL log.

When the data pages in the buffer pool of the memory are operated, all the operations of the data pages are recorded in the log buffer area of the memory, so that a WBL log is generated, and then the WBL log records all the operations of updating, deleting, adding, submitting, rolling back and the like of the data pages in the buffer area of the memory. The WBL log includes one or more of a data modification log, a transaction commit log, and a transaction rollback log. The data modification log may record a location of modification data in a data page in a buffer of the memory, for example, when performing an update operation on data in the data page in the buffer of the memory, the data modification log may record a location of update data; when deleting the data in the data page in the buffer area of the memory, the data modification log can record the position of deleting the data; when the data page in the buffer area of the memory is added, the data modification log can record the position of the added data. The data modification log may also record the type of operation, e.g., update operation, delete operation, update operation, etc., on the data page in the buffer of the memory. The data modification log may also record the ID of the data page in which the modification data is located.

Illustratively, the data modification log may include at least one of a transaction update log, a transaction delete log, and a transaction add log. The data update log records the update operation and the position of the update data of the data page in the buffer area of the memory, the data deletion log records the deletion operation and the position of the deletion data of the data page in the buffer area of the memory, and the transaction addition log records the addition operation and the position of the addition data of the data page in the buffer area of the memory.

The transaction commit log records data pages committed to the disk in the memory's buffer. The transaction rollback log records at least one of a data page in a buffer of the memory in which an update operation has been performed before an execution failure of an operation, a data page in which a delete operation has been performed before an execution failure of an operation, and a data page in which an add operation has been performed before an execution failure of an operation.

In some embodiments, when performing an update operation on a data page in a buffer of the memory, the update operation and a location of update data corresponding to the update operation may be written into a log buffer of the memory, so as to generate a transaction update log. When the deleting operation is performed on the data page in the buffer area of the memory, the deleting operation and the position of the deleting data corresponding to the deleting operation can be written into the log buffer area of the memory, and a transaction deleting log is generated. When the data page in the buffer area of the memory is added, the adding operation and the position of the adding data corresponding to the adding operation can be written into the log buffer area of the memory to generate a transaction adding log. After flushing the data pages in the buffer area of the memory, the data pages submitted to the buffer area in the disk in the buffer area of the memory can be written into the log buffer area of the memory to generate a transaction commit log. And writing the data page for the transaction rollback in the buffer area of the memory into the log buffer area of the memory to generate a transaction rollback log.

Because the WBL log comprises the data modification log, and for operations such as updating, deleting, adding and the like of the data page, the data modification log records the position and the modification type of the modified data in the data page, and the record of the data before modification and the data after modification is not needed, the log quantity of the data modification log is reduced, and therefore the log quantity of the WBL log is reduced. And because the WBL log is flushed before the data page flushing is completed, half-page writing problem does not occur when the WBL log is flushed, and therefore a full-page writing mechanism is not needed to be written, and the log data volume is further reduced.

At present, in order to improve the security of data, a main storage mode is generally adopted to store the data, for example, the data can be stored in the magnetic discs of two computers, for convenience of description, the two computers are called a host computer and a standby computer, and a data page brush disc and a WBL log brush disc can be completed in the host computer, so that the data page and the WBL log are stored in the magnetic discs of the host computer. When the WBL log in the disk of the host computer is copied to the disk of the standby computer, the existing primary-standby copy mode needs to take 8K as the minimum unit to avoid the 8K page half page writing problem. Because the WBL log is flushed before being flushed, the problem of half page writing does not occur when the WBL log is flushed, and the WBL log in the host computer can be sent to the standby computer by taking 4K as the minimum unit.

After receiving the primary and secondary synchronization instruction sent by the user, the host computer sends the WBL log in the disk of the host computer to the secondary computer, and the secondary computer analyzes the WBL log and modifies the data page in the disk of the secondary computer according to the WBL log.

As an implementation manner, the data modification log may record the last data ending position of the modification data and the disk space occupied by the modification data, and then the position of the modification data may be calculated according to the last data ending position of the modification data and the disk space occupied by the modification data.

For example, when the data in the data page in the buffer area of the memory is updated, the data modification log may record the last data end position of the updated data and the disk space occupied by the updated data, and then the position of the updated data may be determined according to the last data end position of the updated data and the disk space occupied by the updated data. When deleting data in a data page in a buffer area of the memory, the data modification log can record the end position of the last data of the deleted data and the disk space occupied by the deleted data, and then the position of the deleted data can be determined according to the end position of the last data of the deleted data and the disk space occupied by the deleted data. When the data page in the buffer area of the memory is added, the data modification log can record the last data end position of the added data and the disk space occupied by the added data, and then the position of the added data can be determined according to the last data end position of the added data and the disk space occupied by the added data.

In some embodiments, when modifying a data page in a buffer pool of a memory, a location where modified data to be modified in the data page is located may be locked, so that performance consumption caused by a large concurrent locking conflict is reduced. Illustratively, the minimum granularity of locking may be byte-level locking.

For example, when an update operation is performed on a data page in a buffer pool of the memory, a lock may be performed on a location of update data corresponding to the update operation in the data page. When the data page in the buffer pool of the memory is subjected to the adding operation, the position of the added data corresponding to the adding operation in the data page can be locked.

According to the data storage method provided by the application, after the data page is firstly flushed, the WBL log is flushed, and as the WBL log comprises the data modification log, the position and the modification type of the modified data in the data page are recorded in the data modification log, the data before modification and the data after modification do not need to be recorded, and the log quantity of the data modification log is reduced, so that the log quantity of the WBL log is reduced.

Fig. 2 is a flowchart of a data storage method according to an embodiment of the present application. Referring to fig. 2, the data storage method provided by the embodiment of the application includes:

S201, when the data page in the buffer pool of the memory is modified, calculating the position of modified data in the data page.

When the data page in the buffer pool of the memory is modified, the end position of the last data of the modified data in the data page and the disk space occupied by the modified data are obtained, and the position of the modified data is calculated according to the end position of the last data of the modified data and the disk space occupied by the modified data. The modification data comprises at least one of update data, deletion data and addition data, wherein the update data refers to update operation of data in a data page, the deletion data refers to deletion operation of data in the data page, and the addition data refers to addition operation of the data page.

For example, when the data in the data page in the buffer area of the memory is updated, the position of the update data is calculated according to the end position of the last data of the update data and the disk space occupied by the update data. When deleting the data in the data page in the buffer area of the memory, the position of the deleted data can be calculated according to the end position of the last data of the deleted data and the disk space occupied by the deleted data. When the data page in the buffer area of the memory is added, the position of the added data can be calculated according to the end position of the last data of the added data and the disk space occupied by the added data.

S202, locking the position of the modified data in the data page.

Because the data in the disk is a shared resource, the access of a plurality of users or a plurality of application programs is supported, and the processes of each access are independent, when a plurality of users access the data in the disk concurrently, a situation that a plurality of transactions access the same data simultaneously occurs in the disk. Incorrect data may be read and stored if concurrent operations are not controlled, disrupting database consistency. When a user needs to modify a data page in a disk, a request is sent to a processor to lock the data page. After the processor locks the data page, the processor does not modify the data page according to modification instructions sent by other users before the data page is unlocked. And locking the position of the modified data in the data page, so that the performance consumption is reduced.

S203, when the transaction begins to be submitted, the data page is flushed.

After the data page in the buffer area of the memory is updated, when the transaction begins to be submitted, the data page in the buffer area of the memory, which is updated, is written into the buffer area of the disk, and is written into the disk from the buffer area of the disk. After the data page in the buffer area of the memory is added, the data page in the buffer area of the memory, which is added, is written into the buffer area of the disk, and is written into the disk from the buffer area of the disk. When deleting the data page in the buffer area of the memory, writing the data page in the buffer area of the memory for deleting operation into the buffer area of the disk, and writing the data page into the disk from the buffer area of the disk.

S204, after the data page is flushed, flushing the WBL log.

The WBL log includes one or more of a data modification log, a transaction commit log, and a transaction rollback log. The data modification log may include at least one of a transaction update log, a transaction delete log, and a transaction add log.

When the data page in the buffer area of the memory is updated, the update operation and the position of the update data corresponding to the update operation can be written into the log buffer area of the memory to generate a transaction update log. When the deleting operation is performed on the data page in the buffer area of the memory, the deleting operation and the position of the deleting data corresponding to the deleting operation can be written into the log buffer area of the memory, and a transaction deleting log is generated. When the data page in the buffer area of the memory is added, the adding operation and the position of the adding data corresponding to the adding operation can be written into the log buffer area of the memory to generate a transaction adding log. After flushing the data pages in the buffer area of the memory, the data pages submitted to the buffer area in the disk in the buffer area of the memory can be written into the log buffer area of the memory to generate a transaction commit log. And writing the data page for the transaction rollback in the buffer area of the memory into the log buffer area of the memory to generate a transaction rollback log. I.e., generate WBL logs.

After the data page is flushed, the WBL log is flushed. For example, the data modification log in the log buffer of the memory is written into the disk, the transaction commit log in the log buffer of the memory is written into the disk, and the transaction rollback log in the log buffer of the memory is written into the disk. It should be noted that, the transaction commit log and the transaction rollback log may be referred to as commit logs, and writing the transaction commit log and the transaction rollback log to the disk may be referred to as clog log drop.

According to the data storage method provided by the application, when the data page in the buffer pool of the memory is modified, the position of the modified data in the data page is calculated, and then the position of the modified data of the data page is locked, so that the data quantity of locking is reduced, and further, the performance consumption caused by locking is reduced. And after the data page is flushed, flushing the WBL log, wherein the WBL log comprises a data modification log, the data modification log records the position and the modification type of the modification data, the data before modification and the data after modification are not required to be recorded, and the log quantity of the WBL log is reduced. And because the WBL log is flushed before the data page flushing is completed, half-page writing problem does not occur in the WBL log flushing process, and a full-page writing mechanism is not needed, so that the log data volume is further reduced. In addition, the server needs to perform corresponding processing according to the WBL log when restarting and recovering, and the actual operation can be reduced and the system recovery time can be reduced when restarting and recovering due to the reduction of the log quantity of the WBL log.

Fig. 3 is a flowchart of a server restart recovery method according to an embodiment of the present application. Referring to fig. 3, the method for restarting the server according to the embodiment of the present application includes:

s301, when the transaction begins to be submitted, the data page is flushed.

S302, after the data page is flushed, the WBL log is flushed.

S303, obtaining a target log from a disk of the server.

After the operating system of the server is down, the server needs to be restarted, and operations of WBL log records need to be played back in the process of restarting the server. The operations recorded in the transaction rollback log in the WBL log are replayed, so that the data page is restored to a state before the fault occurs, and after the server is started, the processor acquires a target log from a disk of the server, wherein the target log can be a data modification log, a transaction commit log or a transaction rollback log in the WBL log.

S304, playing back the target log, and judging whether the target log is the last log in the WBL log.

And after one log is acquired, continuing to acquire the next log, if the next log is not acquired, determining that the currently acquired log is the last log in the WBL logs, and if the next log is acquired, determining that the currently acquired log is not the last log in the WBL logs. The starting log entry for WBL log playback may be the first log entry after the last full checkpoint.

If yes, go to step S305, if no, go to step S306.

S305, the control server receives a request for modifying data in the disk.

And when the acquired log is the last log in the WBL log, indicating that the playback of the WBL log is finished. The server may receive a request from a user to modify data in the disk.

S306, judging whether the target log is a transaction modification log.

The type of the target log may be determined based on the information recorded in the target log. For example, if the target log records the location of the update data, the target data is determined to be the transaction update log. And if the target log records the position of the deleted data, determining the target data as the transaction deleted log. And if the target log records the position of the added data, determining the target data as the transaction added log.

If yes, go to step S303, if no, go to step S307.

S307, judging whether the target log is a transaction commit log.

Judging whether the target log records data pages submitted to the buffer area of the disk in the buffer area of the memory, if so, determining that the target log is a transaction commit log, and if not, determining that the target log is a non-transaction commit log.

If yes, go to step S303, if no, go to step S308.

The data page is submitted to the buffer zone of the disk from the buffer zone of the memory, and the data page submitted to the buffer zone of the disk from the buffer zone of the memory is recorded in the log. The clog log includes a transaction commit log and a transaction rollback log, then data pages committed by the memory's buffer to the disk's buffer may be recorded in the transaction commit log.

S308, determining the target log as a transaction rollback log.

And then step S303 is performed.

According to the server restarting recovery method provided by the application, the WBL logs in the disk are subjected to playback operation, and the actual operation can be reduced when the server restarts to recover due to the reduction of the log quantity of the WBL logs, so that the system recovery time is reduced.

An embodiment of the present application further provides a data storage system 10, referring to fig. 4, the data storage system 10 includes: a transaction processing module 101, the transaction processing module 101 including a transaction management module 1011 and a log management module 1012;

the transaction management module 1011 is used for flushing the data page when the transaction begins to be submitted;

the log management module 1012 is used to flush WBL logs, including PCM logs, including data page modification locations and modification operation types.

The transaction module 101 may guarantee the transaction ACID characteristics.

In some embodiments, transaction processing module 101 may also include concurrency control module 1013 and lock manager 1014, which are used to ensure that multiple transactions access the same data in the database at the same time without breaking isolation and consistency of the transactions and consistency of the database. The lock manager 1014 is used to lock data modification locations, reducing performance consumption due to large concurrent locking conflicts.

Data storage system 10 may also include a connection management system controller 102, a computing module 103, and a storage module 104. The connection management system controller 102 is used to establish connections with clients, verify rights, maintain and manage connections. The computation module 103 includes a query parsing module 1031, a query optimization module 1032, and a query execution module 1033. The query parsing module 1031 is configured to parse a query statement sent by a client to obtain a query condition, and is mainly configured to verify whether the statement is correct according to a grammar rule, for example, whether a keyword of SQL is correct, and whether an order of the keywords is correct. The query optimization module 1032 is configured to optimize query conditions, a query may be executed in a variety of ways, and ultimately return the same results, and the query optimization module 1032 is configured to find an optimal execution plan therein. For example, a first query condition and a second query condition are obtained, the first query condition is greater than 7, the second query condition is greater than 11, and the first query condition and the second query condition are optimized because the second query condition comprises the first query condition, the first query condition is removed, and then the query is performed only according to the second query condition. The query execution module 1033 is used for executing queries, i.e., executing transactions, according to a query plan. The storage module 104 includes an index management module 1041, a memory management module 1042, and a disk file management module 1043, where the index management module 1041 is used to manage an index, and the index is a decentralized storage structure created to accelerate the retrieval of data lines in a table, and each line in an index page contains a logical pointer to accelerate the retrieval of physical data. The memory management module 1042 is used for managing data in the memory, such as modifying, deleting, and adding data in the memory. The disk management module 1043 is configured to manage a disk, for example, receive a data page brush in a memory and a WBL log brush.

The embodiment of the application also provides a computing device, which comprises a processor, wherein the processor is used for executing the data storage method.

The computer devices may include cell phones, tablet computers, notebook computers, palm top computers, personal digital assistants (Personal Digital Assistant, PDA), servers, and the like.

In the description of the present application, it should be noted that, unless explicitly stated and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, indirectly connected through an intermediary, or may be in communication with each other between two elements or in an interaction relationship between two elements. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art according to the specific circumstances.

The application should not be construed as limited to the particular orientations and configurations or operations of the device or element in question. In the description of the present application, the meaning of "a plurality" is two or more, unless specifically stated otherwise.

The terms in the description and claims of the application and the above-described figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.

Claims

1. A method of data storage, comprising:

when a transaction begins to be submitted, the data page is brushed;

brushing a disk of a post-write type log, wherein the post-write type log comprises a data modification log, and the data modification log records the position and the modification type of modification data corresponding to the modification operation when the modification operation is carried out on a data page in a buffer zone of a memory;

the writing-after log brushing disc specifically comprises the following steps:

Brushing the data modification log;

the post-write log further comprises a transaction commit log or a transaction rollback log, wherein the transaction commit log records data pages submitted to a buffer area of a disk in the buffer area of a memory, and the transaction rollback log records data pages for which modification operation has been executed before failure operation;

the method for brushing the post-writing log onto the disk specifically further comprises the following steps:

or writing the data page which has executed the modification operation before the failure operation in the buffer area of the memory into the log buffer area of the memory, generating a transaction rollback log, and brushing the transaction rollback log;

and in the restarting process of the server, playing back a target log, wherein the target log is one or more of the data modification log, the transaction commit log and the transaction rollback log.

2. The method of claim 1, wherein the transaction includes a plurality of modification operations to a page of data in a buffer of the memory;

3. The method of claim 1, wherein the modification operation comprises at least one of an update operation, a delete operation, and an add operation, and wherein the data modification log comprises at least one of a transaction update log, a transaction delete log, and a transaction add log;

4. A method according to any of claims 1-3, wherein before a transaction begins to commit, the method further comprises:

5. The method of claim 4, wherein when performing a modification operation on a data page in a buffer of the memory, calculating a location of modification data corresponding to the modification operation specifically includes:

6. The method of claim 5, wherein when performing the modification operation on the data page in the buffer area of the memory, calculating the position of the modified data according to the end position of the last data of the modified data corresponding to the modification operation and the disk space occupied by the modified data, specifically includes:

7. A method according to any one of claims 1-3, wherein after the brushing of the post-write log on the disk, the method further comprises:

8. A computing device comprising a processor configured to perform the data storage method of any of claims 1-7.