CN113076220A - Data processing method and device, electronic equipment and computer readable medium - Google Patents
Data processing method and device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN113076220A CN113076220A CN202010011632.3A CN202010011632A CN113076220A CN 113076220 A CN113076220 A CN 113076220A CN 202010011632 A CN202010011632 A CN 202010011632A CN 113076220 A CN113076220 A CN 113076220A
- Authority
- CN
- China
- Prior art keywords
- data
- memory table
- storage device
- persistent storage
- dump
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 230000002085 persistent effect Effects 0.000 claims abstract description 110
- 238000000034 method Methods 0.000 claims abstract description 38
- 238000011084 recovery Methods 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 20
- 230000008859 change Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 10
- 238000013500 data storage Methods 0.000 claims description 8
- 230000001680 brushing effect Effects 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 2
- 230000037430 deletion Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 10
- 230000002829 reductive effect Effects 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000007246 mechanism Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011010 flushing procedure Methods 0.000 description 2
- 238000002194 freeze distillation Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004984 smart glass Substances 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
Abstract
The embodiment of the application provides a data processing method and device, electronic equipment and a computer readable medium, and relates to the field of data processing. Wherein the method comprises the following steps: determining a dump site of write data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure; constructing a dump task for the write data in the memory table based on the dump site; and dumping the write data positioned before the dumping site in the memory table to a data layer which does not support read operation in a persistent storage device by executing the dumping task so as to advance the checkpoint of the storage engine. By the embodiment of the application, the resource consumption of the storage engine in the pushing process of the check point of the storage engine is effectively reduced while the downtime recovery efficiency of the storage engine is considered, so that the performance influence on the storage engine is effectively reduced.
Description
Technical Field
The embodiment of the application relates to the field of data processing, and in particular, to a data processing method and apparatus, an electronic device, and a computer-readable medium.
Background
Storage engines based on Log structured merge trees (LSMtree) typically implement failure recovery by way of writes wal (Write Ahead Log). Specifically, the data is persisted wal log data before being written to the active memory table. After the data in the memory table is written into the persistent storage device, wal log data corresponding to the data written into the persistent storage device can be recovered, so that the check point of the storage engine is pushed forward, and the problems of excessive log data wal and excessive downtime and recovery time of the storage engine are solved. When the storage engine is down to recover, wal log data after the check point is played back, so that the data lost in the memory table can be recovered, and the data in the memory can not be lost. If the data in the memory table is refreshed more frequently, the earlier the check point of the storage engine is pushed, and the shorter the downtime of the storage engine is recovered. However, the data in the memory table is frequently flushed, which results in frequent merging of data in the data layer of the persistent storage device, aggravates write amplification, consumes a large amount of I/O read-write data, and also requires data comparison by the CPU, which increases resource consumption of the storage engine and adversely affects performance of the storage engine. Therefore, how to effectively reduce the performance influence on the storage engine while considering the downtime recovery efficiency of the storage engine becomes a technical problem to be solved urgently at present.
Disclosure of Invention
The present application aims to provide a data processing method, an apparatus, an electronic device, and a computer-readable medium, which are used to solve the technical problem in the prior art that how to effectively reduce the performance impact on a storage engine while considering the downtime recovery efficiency of the storage engine.
According to a first aspect of embodiments of the present application, a data processing method is provided. The method comprises the following steps: determining a dump site of write data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure; constructing a dump task for the write data in the memory table based on the dump site; and dumping the write data positioned before the dumping site in the memory table to a data layer which does not support read operation in a persistent storage device by executing the dumping task so as to advance the checkpoint of the storage engine.
According to a second aspect of embodiments of the present application, there is provided a data processing apparatus. The device comprises: the system comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining a dump site of write-in data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure; the building module is used for building a dump task for the write-in data in the memory table based on the dump site; and the dumping module is used for dumping the write data in the memory table before the dumping site to a data layer which does not support read operation in the persistent storage equipment by executing the dumping task so as to advance the check point of the storage engine.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including: one or more processors; a computer readable medium configured to store one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data processing method as described in the first aspect of the embodiments above.
According to a fourth aspect of embodiments of the present application, there is provided a computer-readable medium, on which a computer program is stored, which when executed by a processor, implements the data processing method as described in the first aspect of the embodiments above.
According to the technical scheme provided by the embodiment of the application, based on snapshot data of a memory table of a storage engine adopting an LSM tree structure, a dump site of write data in the memory table is determined, based on the dump site, a dump task for the write data in the memory table is constructed, then the dump task is executed, the write data in the memory table before the dump site is dumped to a data layer which does not support read operation in persistent storage equipment to promote the check point of the storage engine, compared with the existing other modes, by constructing and executing the dump task, the write data in the memory table before the dump site is dumped to the data layer which does not support read operation in the persistent storage equipment, the data in the data layer which does not support read operation and is dumped to the persistent storage equipment is not required to be merged, and while the downtime recovery efficiency of the storage engine is considered, the resource consumption of the storage engine in the pushing process of the checkpoint of the storage engine is effectively reduced, and therefore the performance influence on the storage engine is effectively reduced.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
fig. 1A is a schematic diagram of a log-structured merge tree according to an embodiment of the present application;
FIG. 1B is a flowchart illustrating steps of a data processing method according to an embodiment of the present disclosure;
FIG. 2A is a flowchart illustrating steps of a data processing method according to a second embodiment of the present application;
FIG. 2B is a diagram illustrating a data processing procedure according to the second embodiment of the present application;
FIG. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present application;
FIG. 4 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device in a fifth embodiment of the present application;
fig. 6 is a hardware structure of an electronic device according to a sixth embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1A, according to the LSMtree mechanism, an application (english: application) of the host sends a write instruction to the storage engine 11 through the API interface, after receiving the write instruction, the storage engine 11 generates a LOG file (english: LOG), the LOG file records Version information (english: Version) of data, and then the storage engine 11 stores the data in a memory table in the storage. And when the data volume of all the data stored in the memory table reaches a preset threshold value, converting the memory table into an immutable memory table. The immutable memory table will not receive new data any more, and a new memory table will be created in the memory. The new memory table will continue to receive data. Thus, there may be one or more immutable memory tables, but only one. The data in each immutable memory table is solidified, and the memory table can continuously receive newly written data. In the LSM-Tree mechanism, data in each immutable memory Table is dumped in a Static Sorted Table (SST) file (hereinafter referred to as a file) of the persistent storage device. It is understood that the persistent storage device contains a plurality of files, and each file corresponds to an immutable memory table. The file is used for receiving data dumped from the immutable memory table. In this embodiment, the first-level file includes data directly dumped from the immutable memory table, and the first-level file is also a file that is not merged. The specific process of the method is described below by taking an example in which data in an immutable memory table is dumped in a file. First, the storage engine 11 allocates a file identifier (file identifier is short for file identifier) for a file to be dumped, where the file identifier is used to uniquely indicate the file to be dumped. Then, the storage engine 11 allocates a segment of logical address to the data in the file to be dumped, and saves the correspondence between the file identifier and the saved logical address. The storage engine 11 sends the data and the assigned logical address to the persistent storage device 44, the persistent storage device 44 writes the data into an erase block, and the actual address where the data is stored in the erase block is the physical address of the data. The persistent storage 44 records the correspondence between the logical address and the physical address, and transmits the correspondence between the logical address and the physical address to the storage engine 11. The storage engine 11 saves the correspondence between the logical address and the physical address in the memory.
In the storage engine, the write log and the write data file are two operations with the largest IO consumption in the storage engine, in the two operations, the write data file belongs to scatter write, and the write log file is sequential write, so to ensure the performance of the storage engine, the storage engine generally ensures that the log is written into the log file before the completion of the commit, and the write data in the memory table is stored in the memory and then written into the data file in an irregular and batch manner. That is, log write and commit operations are synchronous, while data write and commit operations are asynchronous. Therefore, a problem exists in that it cannot be guaranteed that all write-in data of the memory table in the memory is written into the data file when the storage engine is down, and thus, when the storage engine is started, the log file is used for performing recovery operation to recover the storage engine to a state before breakdown, so that the consistency of the data is guaranteed. Checkpointing is an important mechanism in this process by which it is determined which log data should be scanned and applied to recover lost data in the memory table at recovery time. In the prior art, the storage engine based on the LSM tree usually relies on flushing the data of the memory table to push the check point, so as to avoid the problems of excessive log files and excessive downtime of the storage engine. However, frequent disk flushes can cause frequent merging of data, aggravate write amplification, and cause more cpu and io consumption, thereby causing a decrease in performance of the storage engine. Therefore, the data processing method provided by the embodiment of the application dumps the write-in data before the dump site in the memory table to the data layer which does not support the read operation in the persistent storage device by constructing and executing the dump task, does not need to execute the merge operation on the data which is dumped to the data layer which does not support the read operation in the persistent storage device, and effectively reduces the resource consumption of the storage engine in the pushing process of the check point of the storage engine while considering the downtime recovery efficiency of the storage engine, thereby effectively reducing the performance influence on the storage engine. The data processing scheme provided by the embodiment of the present application will be described in detail in the first embodiment and the second embodiment provided below.
Referring to fig. 1B, a flowchart illustrating steps of a data processing method according to a first embodiment of the present application is shown.
Specifically, the data processing method of the present embodiment includes the following steps:
in step S101, a dump location of write data in a memory table of a storage engine adopting an LSM tree structure is determined based on snapshot data of the memory table.
In the embodiment of the present application, the snapshot data may be data generated by performing a snapshot operation on a memory table of a storage engine adopting an LSM tree structure at a certain time. In particular, a copy-on-write approach may be used to perform snapshot operations on memory tables of a storage engine that employs an LSM tree structure. The dump location can be understood as a location required when dumping the write data in the memory table, that is, location information required when dumping the write data in the memory table. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, when determining a dump site of write data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure, obtaining a first maximum data version number of global commit data at a current time based on snapshot data of commit data generated by transaction commit in the storage engine; acquiring a second maximum data version number of write-in data in a memory table at the current moment based on snapshot data of the memory table of the storage engine; and taking the smaller of the first maximum data version number and the second maximum data version number as a dump site of the write data in the memory table. Therefore, the dump site of the written data in the memory table can be accurately determined by acquiring the first maximum data version number and the second maximum data version number, and determining the dump site of the written data in the memory table based on the first maximum data version number and the second maximum data version number. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, the snapshot data of the commit data generated by the transaction commit in the storage engine may be understood as data generated by performing a snapshot operation on the commit data generated by the transaction commit in the storage engine at the current time, and the data includes the data version number of all the commit data generated by the transaction commit in the storage engine. Therefore, the first maximum data version number of the global commit data at the current time can be acquired. The snapshot data of the memory table of the storage engine may be understood as data generated by performing a snapshot operation on the memory table of the storage engine at the current time, where the data includes data version numbers of all written data in the memory table. Therefore, the second maximum data version number of the write data in the memory table at the current time can be obtained. Specifically, the first maximum data version number of the global commit data at the current time may be compared with the second maximum data version number of the write data in the memory table at the current time, and the smaller of the first maximum data version number and the second maximum data version number may be used as a dump site of the write data in the memory table. In addition, the smaller of the two can also be used as a playback point of the pre-written log data of the current LSM tree structure, and the pre-written log data before the playback point can be recycled. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In step S102, a dump task for the write data in the memory table is constructed based on the dump site.
In this embodiment, the storage engine may have a plurality of LSM tree structures, and each LSM tree structure corresponds to one memory table. A dump task for write data in the memory table may be constructed for each of a plurality of LSM tree structures. A plurality of dump tasks corresponding to the plurality of LSM tree structures respectively can be placed in a task scheduling queue to wait for execution of dump threads. Meanwhile, the memory table keeps working, can read and write normally, and does not influence normal freezing and disk refreshing. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In step S103, by executing the dump task, the write data in the memory table before the dump site is dumped to a data layer in the persistent storage device that does not support a read operation, so as to advance a checkpoint of the storage engine.
In the embodiment of the present application, the write data in the memory table before the dump site may be understood as write data with a data version number smaller than or equal to the smaller of the two. The persistent storage device may be understood as a device for persistently storing data, such as a solid state disk, a magnetic disk, a mechanical hard disk, or the like. By dumping the write data in the memory table before the dumping site to the data layer which does not support the read operation in the persistent storage device, the merging operation does not need to be executed on the data dumped to the data layer which does not support the read operation in the persistent storage device. This is because, when performing the merge operation, it is necessary to read the data of the two data layers to be merged from the persistent storage device into the memory, merge and sort the data of the two data layers read into the memory, and write the result after merging and sorting into the persistent storage device, but in this embodiment, the write-in data before the dump site in the memory table is dumped to the data layer that does not support the read operation, so the merge operation cannot be performed on the data that is dumped to the data layer that does not support the read operation in the persistent storage device, that is, the data that is dumped to the data layer that does not support the read operation in the persistent storage device does not participate in the merge operation. Therefore, while the downtime recovery efficiency of the storage engine is considered, the resource consumption of the storage engine in the pushing process of the check point of the storage engine is effectively reduced, and the performance influence on the storage engine is effectively reduced. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, when dumping the write data in the memory table before the dump site to a data layer in a persistent storage device that does not support a read operation, scanning the write data in the memory table before the dump site; and writing the write data in the memory table before the dump site to a data layer which does not support the read operation in the persistent storage device according to the data storage format of the persistent storage device. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, the write-in data of which the data version number is smaller than or equal to the smaller of the two in the memory table is scanned, and the write-in data of which the data version number is smaller than or equal to the smaller of the two in the memory table is written to a data layer, which does not support a read operation, in the persistent storage device according to the data storage format of the persistent storage device. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, the method further comprises: if the data layer exists in the persistent storage device when the write data of the memory table is flushed, determining whether the flushed write data of the memory table is redundant with the data of the data layer; and if the written data of the disk brushing in the memory table is determined to be redundant with the data of the data layer, deleting the data layer. Therefore, when the data redundancy of the written data of the disk brushing and the data layer in the memory table is determined, the data layer which does not support the reading operation can be deleted, the redundancy of the data stored in the persistent storage equipment can be eliminated, and the storage space of the persistent storage equipment is saved. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, the method further comprises: and recording the change operation of the dump task on the metadata of all data layers of the persistent storage device by using a log file. Therefore, the integrity of the dump task when the storage engine is down to recover can be ensured by using the log file to record the change operation of the dump task on the metadata of all the data layers of the persistent storage device. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In one particular example, the metadata of the data tier of the persistent storage device may include an identification of each data file of the data tier, the largest and smallest keywords in a single data file, the span of keywords contained by each data file, and other metadata. Other metadata includes, but is not limited to: the data storage method comprises the steps of storing the data in a physical address of a persistent storage device, writing a logical address and writing records for writing the data, and corresponding relation between the physical address and the identification of the file to which the physical address belongs. The write record includes the manner and version in which the user writes the data. Similar to the normal merge operation, the change operation of the dump task on the metadata of the data layer in the persistent storage device is also a transaction process, and a log file of the change operation of the dump task on the metadata of the data layer in the persistent storage device needs to be recorded, so as to ensure the integrity of the dump task when the storage engine is down to recover. Wherein the log file is a redo log file. The content recorded in the redo log may include: the execution step of the change operation, the data content after the execution of the change operation and the like. The redo log file is also called a redo log file and is used for ensuring the durability of the transaction. The redo log file can prevent the shutdown of the storage engine and the situation that the metadata of the data layer of the persistent storage device is not written into the persistent storage device, and the redo log file can be used for redoing to restore the metadata of the data layer of the persistent storage device which is not written into the memory of the storage engine, so that the persistence of the transaction is ensured. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, the method further comprises: and when the storage engine is down to recover, converting the data layer into the highest data layer of the persistent storage device to support reading operation, and recovering the written data which is positioned behind the dump site in the memory table by playing back log data corresponding to the written data which is positioned behind the dump site in the memory table. Wherein the Log data may be wal (Write Ahead Log) Log data. Therefore, the data layer which does not support the read operation is converted into the highest data layer which supports the read operation in the persistent storage device, and then the written data which is positioned behind the dump site in the memory table is recovered by replaying the log data corresponding to the written data which is positioned behind the dump site in the memory table, so that the number of the replayed log data when the storage engine is shut down and recovered is effectively reduced, and the efficiency of the shut down and recovery of the storage engine is remarkably improved. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, in converting the data tier to a highest data tier of the persistent storage, the data tier is converted to the highest data tier of the persistent storage by altering metadata of the data tier. Wherein the highest data tier may be a tier 0 data tier of a persistent storage device. Therefore, the data layer which does not support the read operation can be conveniently converted into the highest data layer of the persistent storage device by changing the metadata of the data layer which does not support the read operation. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, before the converting the data tier to a highest data tier of the persistent storage device, the method further comprises: restoring the metadata of all data layers of the persistent storage device in the memory of the storage engine based on the checkpoint of the metadata of all data layers of the persistent storage device and a log file for recording the change operation of the dump task on the metadata of all data layers of the persistent storage device. Therefore, the metadata of all the data layers of the persistent storage device is restored in the memory of the storage engine based on the check point of the metadata of all the data layers of the persistent storage device and the log file for recording the change operation of the dump task on the metadata of all the data layers of the persistent storage device, and the integrity of the dump task when the storage engine is down to recover can be ensured. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
According to the data processing method provided by the embodiment of the application, based on the snapshot data of the memory table of the storage engine adopting the LSM tree structure, the dump site of the write data in the memory table is determined, based on the dump site, the dump task for the write data in the memory table is constructed, then the dump task is executed, the write data in the memory table before the dump site is dumped to the data layer which does not support the read operation in the persistent storage equipment, so as to promote the check point of the storage engine, compared with the existing other modes, by constructing and executing the dump task, the write data in the memory table before the dump site is dumped to the data layer which does not support the read operation in the persistent storage equipment, the merging operation of the data in the data layer which does not support the read operation and is not required to be executed in the persistent storage equipment, and while the downtime recovery efficiency of the storage engine is considered, the resource consumption of the storage engine in the pushing process of the checkpoint of the storage engine is effectively reduced, and therefore the performance influence on the storage engine is effectively reduced.
The data processing method of the present embodiment may be performed by any suitable device having data processing capabilities, including but not limited to: a camera, a terminal, a mobile terminal, a PC, a server, an in-vehicle device, an entertainment device, an advertising device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a handheld game console, smart glasses, a smart watch, a wearable device, a virtual display device, a display enhancement device, or the like.
Referring to fig. 2A, a flowchart illustrating steps of a data processing method according to a second embodiment of the present application is shown.
Specifically, the data processing method of the present embodiment includes the following steps:
in step S201, a dump location of write data in a memory table of a storage engine adopting an LSM tree structure is determined based on snapshot data of the memory table.
Since step S201 is similar to step S101 described above, it is not described herein again.
In step S202, a dump task for the write data in the memory table is constructed based on the dump site.
Since step S202 is similar to step S102, it is not repeated herein.
In step S203, by executing the dump task, the write data in the memory table before the dump site is dumped to a data layer in the persistent storage device that does not support a read operation, so as to advance a checkpoint of the storage engine.
Since step S203 is similar to step S103, it is not described herein again.
In step S204, if it is determined that the old data layer which does not support the read operation exists in the persistent storage device, deleting the old data layer which does not support the read operation.
In a specific example, the implementation of the data processing scheme provided by this embodiment mainly includes the following components: 1. and (3) building a dump task, recording a dump site according to the current snapshot of the memory table, and putting the dump task built based on the dump site into a task scheduling queue to wait for execution, as shown in fig. 2B. 2. And the dump function is realized by writing the data in the memory table to the data layer M0 which does not support the read operation in the persistent storage device according to the data storage format of the persistent storage device and deleting the old data layer M0. 3. And the storage engine is down to recover, the normal LSM tree structure is recovered firstly during recovery, then the data layer M0 is recovered to be the latest data layer L0, and the written data lost in the memory table can be recovered by replaying wal log data corresponding to the written data after the dump site. The details of the implementation of each component are described in detail below. 1. And (5) building a dump task. When a dump task is constructed, the maximum data version number of globally submitted data and the maximum data version number of write data in a current memory table are obtained, the smaller of the maximum data version number of globally submitted data and the maximum data version number of write data in the current memory table is used as a dump point and also used as a log playback point of a current LSM tree structure, and wal logs before the playback point can be recycled. Meanwhile, the current memory table keeps working, can be read and written normally, and does not influence normal freezing and disk refreshing. And a plurality of dump tasks corresponding to the LSM tree structures are put into a task scheduling queue to wait for the execution of dump threads. 2. And (5) realizing a dump function. During dumping, all data with the data version number smaller than or equal to the smaller of the two in the memory table are scanned and written to the data layer M0 according to the data storage format of the persistent storage device. Meanwhile, if the old data tier M0 exists in the persistent storage device, it is deleted, and the data tier M0 does not support a read operation, and thus does not participate in merging of data, as shown in fig. 2B. The memory table also needs to check whether the data layer M0 exists in the persistent storage device when the disk is normally flushed, because the data flushed is redundant with the data of the data layer M0, and the data layer M0 needs to be deleted. Similar to normal merging, the change of the metadata of the data layer of the persistent storage device by the dump task is also a transaction process, and a redo log of the change operation of the metadata of the data layer of the persistent storage device by the dump task needs to be recorded, so as to ensure the integrity of the dump task when the storage engine is down to recover. 3. And storing the implementation of engine downtime recovery. At recovery time, the metadata of all data layers in the persistent storage device is restored to memory, including data layer M0, through the checkpoint and redo log of the metadata of the data layers. Thereafter, the data of data tier M0 is added to the highest data tier L0 in the persisted storage, becoming the newest tier of data L0 and the data tier M0 is deleted, where the addition does not involve reading data to modify only the metadata of data tier M0, as shown in FIG. 2B. And then, by constructing a playback site stored during the dump task, the log data after the playback site is played back to the memory table. Therefore, the data in the memory table is redundantly dumped, so that the influence on the performance of the storage engine caused by frequent disk refreshing and merging is avoided, and the playback log amount can be reduced and the recovery efficiency can be improved by recovering the data layer M0 to the highest data layer L0. The dump task is a background task, and can do more work to advance the check point in the low peak period of the storage engine so as to achieve shorter recovery time. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
On the basis of the first embodiment of the present application, if it is determined that the old data layer which does not support the read operation exists in the persistent storage device, the old data layer which does not support the read operation is deleted, and compared with other existing manners, if it is determined that the old data layer which does not support the read operation exists in the persistent storage device, the old data layer which does not support the read operation is deleted, so that redundancy of data stored in the persistent storage device can be eliminated, and thus a storage space is saved for the persistent storage device.
The data processing method of the present embodiment may be performed by any suitable device having data processing capabilities, including but not limited to: a camera, a terminal, a mobile terminal, a PC, a server, an in-vehicle device, an entertainment device, an advertising device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a handheld game console, smart glasses, a smart watch, a wearable device, a virtual display device, a display enhancement device, or the like.
Referring to fig. 3, a schematic structural diagram of a data processing apparatus according to a third embodiment of the present application is shown.
The data processing apparatus of the present embodiment includes: a first determining module 301, configured to determine, based on snapshot data of a memory table of a storage engine adopting an LSM tree structure, a dump location of write data in the memory table; a building module 302, configured to build a dump task for the write data in the memory table based on the dump site; the dumping module 303 is configured to dump, by executing the dumping task, the write data in the memory table located before the dumping site to a data layer in the persistent storage device that does not support a read operation, so as to advance a checkpoint of the storage engine.
The data processing apparatus of this embodiment is configured to implement the corresponding data processing method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Referring to fig. 4, a schematic structural diagram of a data processing apparatus in the fourth embodiment of the present application is shown.
The data processing apparatus of the present embodiment includes: a first determining module 401, configured to determine, based on snapshot data of a memory table of a storage engine adopting an LSM tree structure, a dump location of write data in the memory table; a building module 402, configured to build a dump task for the write data in the memory table based on the dump site; a dumping module 403, configured to dump, by executing the dumping task, write data in the memory table located before the dumping site to a data layer in a persistent storage device that does not support a read operation, so as to advance a checkpoint of the storage engine.
Optionally, the first determining module 401 is specifically configured to: acquiring a first maximum data version number of global commit data at the current moment based on snapshot data of the commit data generated by transaction commit in the storage engine; acquiring a second maximum data version number of write-in data in a memory table at the current moment based on snapshot data of the memory table of the storage engine; and taking the smaller of the first maximum data version number and the second maximum data version number as a dump site of the write data in the memory table.
Optionally, the dump module 403 is specifically configured to: scanning write data in the memory table before the dump site; and writing the write data in the memory table before the dump site to a data layer which does not support the read operation in the persistent storage device according to the data storage format of the persistent storage device.
Optionally, the apparatus further comprises: a first deletion module 404 for deleting an old data layer if it is determined that the old data layer exists in the persistent storage.
Optionally, the apparatus further comprises: a second determining module 406, configured to determine whether the written data flushed from the memory table is redundant with data of the data layer if it is determined that the data layer exists in the persistent storage device when the written data of the memory table is flushed; a second deleting module 407, configured to delete the data layer if it is determined that the written data of the disk-flushing in the memory table is redundant with the data of the data layer.
Optionally, the apparatus further comprises: a recording module 405, configured to record, using a log file, a change operation of the dump task on metadata of all data layers of the persistent storage device.
Optionally, the apparatus further comprises: the first recovery module 409 is configured to, when the storage engine is down to recover, convert the data layer into the highest data layer of the persistent storage device to support a read operation, and recover, by playing back log data corresponding to write data located after the dump site in the memory table, the write data located after the dump site in the memory table.
Optionally, the first recovery module 409 is specifically configured to: converting the data tier to a highest data tier of the persistent storage device by altering metadata of the data tier.
Optionally, before the first recovery module 409, the apparatus further includes: a second recovery module 408, configured to recover the metadata of all data layers of the persistent storage device in the memory of the storage engine based on the checkpoint of the metadata of all data layers of the persistent storage device and a log file used for recording change operations of the dump task on the metadata of all data layers of the persistent storage device.
The data processing apparatus of this embodiment is configured to implement the corresponding data processing method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Fig. 5 is a schematic structural diagram of an electronic device in a fifth embodiment of the present application; the electronic device may include:
one or more processors 501;
a computer-readable medium 502, which may be configured to store one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data processing method according to the first embodiment or the second embodiment.
Fig. 6 is a hardware structure of an electronic device according to a sixth embodiment of the present application; as shown in fig. 6, the hardware structure of the electronic device may include: a processor 601, a communication interface 602, a computer-readable medium 603, and a communication bus 604;
wherein the processor 601, the communication interface 602, and the computer readable medium 603 communicate with each other via a communication bus 604;
alternatively, the communication interface 602 may be an interface of a communication module, such as an interface of a GSM module;
the processor 601 may be specifically configured to: determining a dump site of write data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure; constructing a dump task for the write data in the memory table based on the dump site; and dumping the write data positioned before the dumping site in the memory table to a data layer which does not support read operation in a persistent storage device by executing the dumping task so as to advance the checkpoint of the storage engine.
The Processor 601 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The computer-readable medium 603 may be, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code configured to perform the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. The computer program, when executed by a Central Processing Unit (CPU), performs the above-described functions defined in the method of the present application. It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access storage media (RAM), a read-only storage media (ROM), an erasable programmable read-only storage media (EPROM or flash memory), an optical fiber, a portable compact disc read-only storage media (CD-ROM), an optical storage media piece, a magnetic storage media piece, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code configured to carry out operations for the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may operate over any of a variety of networks: including a Local Area Network (LAN) or a Wide Area Network (WAN) -to the user's computer, or alternatively, to an external computer (e.g., through the internet using an internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions configured to implement the specified logical function(s). In the above embodiments, specific precedence relationships are provided, but these precedence relationships are only exemplary, and in particular implementations, the steps may be fewer, more, or the execution order may be modified. That is, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present application may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a first determination module, a build module, and a dump module. The names of these modules do not constitute a limitation on the module itself in some cases, for example, the first determining module may also be described as "a module that determines a dump location of write data in a memory table of a storage engine adopting an LSM tree structure based on snapshot data of the memory table".
As another aspect, the present application also provides a computer-readable medium on which a computer program is stored, which when executed by a processor, implements the data processing method as described in the first or second embodiment.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: determining a dump site of write data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure; constructing a dump task for the write data in the memory table based on the dump site; and dumping the write data positioned before the dumping site in the memory table to a data layer which does not support read operation in a persistent storage device by executing the dumping task so as to advance the checkpoint of the storage engine.
The expressions "first", "second", "said first" or "said second" used in various embodiments of the present disclosure may modify various components regardless of order and/or importance, but these expressions do not limit the respective components. The above description is only configured for the purpose of distinguishing elements from other elements. For example, the first user equipment and the second user equipment represent different user equipment, although both are user equipment. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure.
When an element (e.g., a first element) is referred to as being "operably or communicatively coupled" or "connected" (operably or communicatively) to "another element (e.g., a second element) or" connected "to another element (e.g., a second element), it is understood that the element is directly connected to the other element or the element is indirectly connected to the other element via yet another element (e.g., a third element). In contrast, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (a second element), no element (e.g., a third element) is interposed therebetween.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (20)
1. A method of data processing, the method comprising:
determining a dump site of write data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure;
constructing a dump task for the write data in the memory table based on the dump site;
and dumping the write data positioned before the dumping site in the memory table to a data layer which does not support read operation in a persistent storage device by executing the dumping task so as to advance the checkpoint of the storage engine.
2. The method of claim 1, wherein determining a dump location of write data in a memory table of a storage engine in an LSM tree structure based on snapshot data of the memory table comprises:
acquiring a first maximum data version number of global commit data at the current moment based on snapshot data of the commit data generated by transaction commit in the storage engine;
acquiring a second maximum data version number of write-in data in a memory table at the current moment based on snapshot data of the memory table of the storage engine;
and taking the smaller of the first maximum data version number and the second maximum data version number as a dump site of the write data in the memory table.
3. The method of claim 1, wherein dumping write data in the memory table prior to the dump site to a data layer in a persistent storage device that does not support a read operation comprises:
scanning write data in the memory table before the dump site;
and writing the write data in the memory table before the dump site to a data layer which does not support the read operation in the persistent storage device according to the data storage format of the persistent storage device.
4. The method of claim 3, further comprising:
deleting an old data layer if it is determined that the old data layer exists in the persistent storage device.
5. The method of claim 1, further comprising:
if the data layer exists in the persistent storage device when the write data of the memory table is flushed, determining whether the flushed write data of the memory table is redundant with the data of the data layer;
and if the written data of the disk brushing in the memory table is determined to be redundant with the data of the data layer, deleting the data layer.
6. The method of claim 1, further comprising:
and recording the change operation of the dump task on the metadata of all data layers of the persistent storage device by using a log file.
7. The method of claim 1, further comprising:
and when the storage engine is down to recover, converting the data layer into the highest data layer of the persistent storage device to support reading operation, and recovering the written data which is positioned behind the dump site in the memory table by playing back log data corresponding to the written data which is positioned behind the dump site in the memory table.
8. The method of claim 7, wherein converting the data tier to a highest data tier of the persistent storage device comprises:
converting the data tier to a highest data tier of the persistent storage device by altering metadata of the data tier.
9. The method of claim 7, wherein prior to said converting the data tier to a highest data tier of the persistent storage device, the method further comprises:
restoring the metadata of all data layers of the persistent storage device in the memory of the storage engine based on the checkpoint of the metadata of all data layers of the persistent storage device and a log file for recording the change operation of the dump task on the metadata of all data layers of the persistent storage device.
10. A data processing apparatus, characterized in that the apparatus comprises:
the system comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining a dump site of write-in data in a memory table based on snapshot data of the memory table of a storage engine adopting an LSM tree structure;
the building module is used for building a dump task for the write-in data in the memory table based on the dump site;
and the dumping module is used for dumping the write data in the memory table before the dumping site to a data layer which does not support read operation in the persistent storage equipment by executing the dumping task so as to advance the check point of the storage engine.
11. The apparatus of claim 10, wherein the first determining module is specifically configured to:
acquiring a first maximum data version number of global commit data at the current moment based on snapshot data of the commit data generated by transaction commit in the storage engine;
acquiring a second maximum data version number of write-in data in a memory table at the current moment based on snapshot data of the memory table of the storage engine;
and taking the smaller of the first maximum data version number and the second maximum data version number as a dump site of the write data in the memory table.
12. The apparatus of claim 10, wherein the dump module is specifically configured to:
scanning write data in the memory table before the dump site;
and writing the write data in the memory table before the dump site to a data layer which does not support the read operation in the persistent storage device according to the data storage format of the persistent storage device.
13. The apparatus of claim 12, further comprising:
a first deletion module to delete an old data layer if it is determined that the old data layer exists in the persistent storage device.
14. The apparatus of claim 10, further comprising:
a second determining module, configured to determine whether the written data flushed from the disk in the memory table is redundant with data of the data layer if it is determined that the data layer exists in the persistent storage device when the written data flushed from the disk in the memory table;
and the second deleting module is used for deleting the data layer if the written data of the disk refreshing in the memory table and the data redundancy of the data layer are determined.
15. The apparatus of claim 10, further comprising:
and the recording module is used for recording the change operation of the dump task on the metadata of all the data layers of the persistent storage device by using a log file.
16. The apparatus of claim 10, further comprising:
and the first recovery module is used for converting the data layer into the highest data layer of the persistent storage device to support read operation when the storage engine is recovered in a downtime manner, and recovering the write data which are positioned behind the dump site in the memory table by playing back log data corresponding to the write data which are positioned behind the dump site in the memory table.
17. The apparatus of claim 16, wherein the first recovery module is specifically configured to:
converting the data tier to a highest data tier of the persistent storage device by altering metadata of the data tier.
18. The apparatus of claim 16, wherein the first recovery module is preceded by:
and the second recovery module is used for recovering the metadata of all the data layers of the persistent storage device in the memory of the storage engine based on the checkpoint of the metadata of all the data layers of the persistent storage device and a log file for recording the change operation of the dump task on the metadata of all the data layers of the persistent storage device.
19. An electronic device, comprising:
one or more processors;
a computer readable medium configured to store one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data processing method as claimed in any one of claims 1-9.
20. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010011632.3A CN113076220A (en) | 2020-01-06 | 2020-01-06 | Data processing method and device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010011632.3A CN113076220A (en) | 2020-01-06 | 2020-01-06 | Data processing method and device, electronic equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113076220A true CN113076220A (en) | 2021-07-06 |
Family
ID=76608972
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010011632.3A Pending CN113076220A (en) | 2020-01-06 | 2020-01-06 | Data processing method and device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113076220A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114513558A (en) * | 2022-01-29 | 2022-05-17 | 建信金融科技有限责任公司 | User request processing method and device |
| CN115061858A (en) * | 2022-08-19 | 2022-09-16 | 湖南视拓信息技术股份有限公司 | Data persistence method and device, computer equipment and storage medium |
| CN115202588A (en) * | 2022-09-14 | 2022-10-18 | 云和恩墨(北京)信息技术有限公司 | Data storage method and device and data recovery method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100106695A1 (en) * | 2008-10-24 | 2010-04-29 | Microsoft Corporation | Scalable blob storage integrated with scalable structured storage |
| CN102831034A (en) * | 2012-08-14 | 2012-12-19 | 中国电力科学研究院 | Multimode backup and quick recovery method based on embedded Linux device |
| CN104360824A (en) * | 2014-11-10 | 2015-02-18 | 北京奇虎科技有限公司 | Data merging method and device |
| US20150082210A1 (en) * | 2013-09-19 | 2015-03-19 | Oracle International Corporation | System and method for providing a visual preview of a user interface heap dump |
| CN104461768A (en) * | 2013-09-22 | 2015-03-25 | 华为技术有限公司 | Copy storing device and copy storing method |
| CN109254874A (en) * | 2018-08-29 | 2019-01-22 | 郑州云海信息技术有限公司 | Method for transformation, device and the equipment of a kind of piece of storing data snapshot type |
| EP3477490A1 (en) * | 2017-10-26 | 2019-05-01 | Druva Technologies Pte. Ltd. | Deduplicated merged indexed object storage file system |
| CN110008129A (en) * | 2019-04-11 | 2019-07-12 | 苏州浪潮智能科技有限公司 | A kind of method for testing reliability, device and equipment storing timing snapshot |
-
2020
- 2020-01-06 CN CN202010011632.3A patent/CN113076220A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100106695A1 (en) * | 2008-10-24 | 2010-04-29 | Microsoft Corporation | Scalable blob storage integrated with scalable structured storage |
| CN102831034A (en) * | 2012-08-14 | 2012-12-19 | 中国电力科学研究院 | Multimode backup and quick recovery method based on embedded Linux device |
| US20150082210A1 (en) * | 2013-09-19 | 2015-03-19 | Oracle International Corporation | System and method for providing a visual preview of a user interface heap dump |
| CN104461768A (en) * | 2013-09-22 | 2015-03-25 | 华为技术有限公司 | Copy storing device and copy storing method |
| CN104360824A (en) * | 2014-11-10 | 2015-02-18 | 北京奇虎科技有限公司 | Data merging method and device |
| EP3477490A1 (en) * | 2017-10-26 | 2019-05-01 | Druva Technologies Pte. Ltd. | Deduplicated merged indexed object storage file system |
| CN109254874A (en) * | 2018-08-29 | 2019-01-22 | 郑州云海信息技术有限公司 | Method for transformation, device and the equipment of a kind of piece of storing data snapshot type |
| CN110008129A (en) * | 2019-04-11 | 2019-07-12 | 苏州浪潮智能科技有限公司 | A kind of method for testing reliability, device and equipment storing timing snapshot |
Non-Patent Citations (2)
| Title |
|---|
| "可扩展数据库管理系统中的数据复制", 中国优秀硕士学位论文全文数据库, no. 1 * |
| 张雪峰;党德玉;罗洪霞;: "一种基于快照管理的非一致性分析法", 计算机与现代化, no. 07 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114513558A (en) * | 2022-01-29 | 2022-05-17 | 建信金融科技有限责任公司 | User request processing method and device |
| CN114513558B (en) * | 2022-01-29 | 2023-06-16 | 建信金融科技有限责任公司 | User request processing method and device |
| CN115061858A (en) * | 2022-08-19 | 2022-09-16 | 湖南视拓信息技术股份有限公司 | Data persistence method and device, computer equipment and storage medium |
| CN115202588A (en) * | 2022-09-14 | 2022-10-18 | 云和恩墨(北京)信息技术有限公司 | Data storage method and device and data recovery method and device |
| CN115202588B (en) * | 2022-09-14 | 2022-12-27 | 本原数据(北京)信息技术有限公司 | Data storage method and device and data recovery method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8856469B2 (en) | Apparatus and method for logging optimization using non-volatile memory | |
| US9183236B2 (en) | Low level object version tracking using non-volatile memory write generations | |
| US8250033B1 (en) | Replication of a data set using differential snapshots | |
| US8548948B2 (en) | Methods and apparatus for a fine grained file data storage system | |
| US7395278B2 (en) | Transaction consistent copy-on-write database | |
| CN113076220A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| US10049036B2 (en) | Reliable distributed messaging using non-volatile system memory | |
| US9767015B1 (en) | Enhanced operating system integrity using non-volatile system memory | |
| CN107665219B (en) | Log management method and device | |
| US9760480B1 (en) | Enhanced logging using non-volatile system memory | |
| JP2017079053A (en) | Methods and systems for improving storage journaling | |
| CN108628542B (en) | File merging method and controller | |
| Kim et al. | Exploiting compression-induced internal fragmentation for power-off recovery in SSD | |
| CN110019063B (en) | Method for computing node data disaster recovery playback, terminal device and storage medium | |
| US11556423B2 (en) | Using erasure coding in a single region to reduce the likelihood of losing objects maintained in cloud object storage | |
| KR20160012388A (en) | Method and apparatus for fsync system call processing using ordered mode journaling with file unit | |
| CN116257460B (en) | Trim command processing method based on solid state disk and solid state disk | |
| US11487428B2 (en) | Storage control apparatus and storage control method | |
| US10430115B2 (en) | System and method for optimizing multiple packaging operations in a storage system | |
| US20230297510A1 (en) | Write-behind optimization of covering cache | |
| CN110928890B (en) | Data storage method and device, electronic equipment and computer readable storage medium | |
| US7949632B2 (en) | Database-rearranging program, database-rearranging method, and database-rearranging apparatus | |
| KR20190113359A (en) | Data storing and restoring method based on In-memory database using NVDIMM | |
| CN113791934A (en) | Data recovery method, computing device and storage medium | |
| CN112416261A (en) | Data storage method, device, equipment and system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40056497 Country of ref document: HK |