CN110928890A - Data storage method and device, electronic equipment and computer readable medium - Google Patents
Data storage method and device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN110928890A CN110928890A CN201911088468.XA CN201911088468A CN110928890A CN 110928890 A CN110928890 A CN 110928890A CN 201911088468 A CN201911088468 A CN 201911088468A CN 110928890 A CN110928890 A CN 110928890A
- Authority
- CN
- China
- Prior art keywords
- data
- page
- log
- check code
- time stamp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a data storage method and device, electronic equipment and a computer readable medium, and relates to the field of data storage. The method is applied to a storage class memory which comprises a memory and a magnetic disk, and comprises the following steps: when data in the memory needs to be written into the disk, generating a log file of the data and a corresponding first check code; writing the data into a data page of the disk, and updating a second check code of the data page; when the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp; and updating the data of the data page according to the log record corresponding to the page time stamp. The method and the device have the advantages that the disordered writing and partial writing of the data in the SCM are avoided through the mechanisms of the check codes, the logs and the check points, and the influence on the performance of the storage system is reduced.
Description
Technical Field
The present application relates to the field of data storage, and more particularly, to a data storage method and apparatus, an electronic device, and a computer-readable medium.
Background
Storage Class Memory (SCM), as a new type of memory device, can access a memory bus like Random Access Memory (RAM), and the CPU directly accesses the SCM through a store and load instruction. However, partial writes as well as out-of-order writes may occur to the data of the SCM.
Disclosure of Invention
The application provides a data storage method, a data storage device, an electronic device and a computer readable medium, so as to overcome the defects.
In a first aspect, an embodiment of the present application provides a data storage method, where the method includes: when data in the memory need to be written into a disk, generating a log file of the data and a corresponding first check code; writing data into a data page of a disk, and updating a second check code of the data page; when the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from a log file according to the page time stamp; and updating the data of the data page according to the log record corresponding to the page time stamp.
In a second aspect, an embodiment of the present application further provides a data storage device, where the data storage device includes: the device comprises a first check code generation module, a second check code updating module, a log record acquisition module and a data updating module. The first check code generation module is used for generating a log file of data and a corresponding first check code when the data in the memory need to be written into a disk; the second check code updating module is used for writing data into a data page of the disk and updating the second check code of the data page; the log record acquisition module is used for acquiring a page time stamp of the data page when the first check code is not matched with the second check code, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp; and the data updating module is used for updating the data of the data page according to the log record corresponding to the page time stamp.
In a third aspect, an embodiment of the present application further provides an electronic device, including: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the above-described methods.
In a fourth aspect, the present application also provides a computer-readable storage medium, where the readable storage medium stores program code executable by a processor, and a plurality of instructions in the program code, when executed by the processor, cause the processor to execute the above method.
According to the data storage method, the data storage device, the electronic equipment and the computer readable medium, when data exist in a memory and need to be written into a disk, a log file of the data and a corresponding first check code are generated; and writing the data into a data page of the disk, and updating the second check code of the data page, so that whether the data is partially written can be effectively detected according to whether the first check code and the second check code are matched. When the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp; and updating the data of the data page according to the log record corresponding to the page time stamp, so that when partial writing of the data is detected, the corresponding log record can be accurately found according to the time stamp to update the data page, thereby not only avoiding the partial writing of the data, but also ensuring the sequential writing of the data in a log mode.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram illustrating a memory storage hierarchy according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram illustrating a memory architecture provided in an embodiment of the present application;
FIG. 3 is a flow chart of a method of storing data according to an embodiment of the present application;
FIG. 4 is a flow chart of a method of storing data according to another embodiment of the present application;
FIG. 5 shows a flowchart of the method of S210 in FIG. 4;
FIG. 6 is a flow chart of a method of storing data according to yet another embodiment of the present application;
FIG. 7 is a flow chart illustrating a method for performing checkpointing according to one embodiment of the present application;
fig. 8 is a block diagram illustrating a data storage device according to an embodiment of the present application;
fig. 9 is a block diagram illustrating a data storage device according to another embodiment of the present application;
fig. 10 shows a block diagram of an electronic device provided in an embodiment of the present application;
fig. 11 illustrates a storage unit for storing or carrying program codes for implementing a data storage method according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.
With the widespread application of Storage Class Memory (SCM) in the market, the interest is increasing, and the new Memory has the advantages of both disk and Memory: load and store instructions of the CPU are supported, namely, the access speed of the memory can be matched by addressing with the granularity of bytes, so that the I/O unit of the SCM is generally one cache line size, namely 64B. The SCM has the same read speed as the memory and the write latency is about 4-5 times higher than the direct write memory. In contrast to RAM, SCM does not lose data after power is turned off. SCMs also present many challenges, for example SCMs have certain problems with sequential writes as well as partial writes.
Referring to fig. 1, as an example, fig. 1 shows a memory storage hierarchy, as shown in fig. 1, on one hand, an SCM memory is accessed to a memory bus, and on the other hand, a CPU directly addresses SCM storage space through a memory controller. Due to the long path from the CPU registers to SCM storage and the volatility of these storage media, memory data is lost after power down, resulting in partial writes of data. In addition, the control of the store buffer, L1 and L2 cache by the relevant software used by the target is small. Thus, the memory store instruction needs to be display declared and persisted to ensure coherency. For example, the Intel x86 processor family provides clflush, mfence, sfence, etc. instructions to write cached data to the SCM to ensure the persistence and ordering of the data. Future platforms are also prepared to support optimized clflush instructions, namely clflush and clwb. The clflush instruction writes a cache line back to memory, which is a synchronous instruction that does not require a memory fence (mfence) to serialize the instruction. sfence is used to ensure that all pending store instructions are serialized, i.e., to ensure that the store instruction before and after the instruction is executed in the order of sfence's division (mfence is valid for both load and store instructions). clflush is an asynchronous version of clflush; clwb writes the cache line back to memory, but does not flush, which is beneficial for later reads, and the instruction also does not guarantee write order.
The inventor finds in research that, based on the above-mentioned instruction, if the write order is guaranteed, a clwb instruction needs to be executed together with two sfence instructions, however, these instructions need to access the memory and serialize, which has a great influence on the performance and the overhead of the storage system.
Therefore, in order to solve the above problems, the inventors propose a data storage method, an apparatus, an electronic device, and a computer readable medium, which not only avoid out-of-order writing and partial writing of data in the SCM through a mechanism of checking codes, logs, and checkpoints, but also effectively reduce the impact on the performance and overhead of the storage system compared with the method that an order is guaranteed by using instructions such as mfence and clflush.
Referring to fig. 2, a memory architecture is shown, as shown in fig. 2, the architecture includes a Central Processing Unit (CPU), a cache memory (cache memory), a memory (RAM-main memory), and a disk. Fig. 1 shows only one CPU, and when the device is multi-core, multiple CPUs may be included, and each CPU is connected to the memory through its own cache. Each CPU executes one thread and multiple CPUs can perform tasks simultaneously, i.e., concurrently. Each CPU has its own register on which the CPU performs some operations, and the speed at which operations are performed is much greater than in main memory. Generally, when a CPU needs to read a memory, part of data is read and placed in a cache of the CPU, or even part of data in the cache of the CPU is read into a register, and then the data is operated in the register. When the CPU needs to store something in the cache layer, the contents stored in the cache are usually flushed back to the memory. The CPU cache may write data locally to its memory at a time, or may refresh its memory locally at a time. When the CPU needs to write data into the disk again, the CPU needs to pass through the memory.
Referring to fig. 3, fig. 3 is a schematic flowchart illustrating a data storage method according to an embodiment of the present application, where the method may be applied to the storage class storage, and the storage class storage may include a memory and a magnetic disk, where an execution subject of the method may be a controller in the storage class storage, and as shown in fig. 3, the method may include: s110 to S140.
S110: when data in the memory need to be written into the disk, a log file of the data and a corresponding first check code are generated.
In some embodiments, before writing the data in the memory of the storage class memory into the disk, the controller of the storage class memory may map an event that the data is written into the disk into an event code, then record the occurrence time of the event, organize the occurrence time of the event and the time code into a log record of the event, and then store the log record into a log file, thereby generating a log file of the data.
In addition, when a log file of the data is generated, a first check code may be generated based on the data as a check code corresponding to the log file.
Alternatively, the first check code may be a Checksum. Wherein a checksum is typically used to ensure data integrity and accuracy. Alternatively, the magnetic Disk may be replaced by another storage medium, such as a Solid State Disk (SSD), and the like, which is not limited herein.
S120: and writing the data into the data page of the disk, and updating the second check code of the data page.
When the data is written into the data page of the disk, the data corresponding to the data page is updated, and the second check code of the data page is updated based on the data written into the disk. In one embodiment, the second check code is of the same type as the first check code, and is also a checksum. Where a data page is a structure containing user data that has been added to a table of a database, as an example, a data page may be treated as an array containing 1024 64-bit integers in memory, and for the structure of a data page, a file header may be included generally: basic information for describing a page, such as page number, page type, upper and lower page information, etc., a page header: the state value for storing records in the data page, such as the number of slots in the page directory, the number of records in the page, etc. user records user data: for storing the records, for better management of the records.
In some embodiments, the second parity code may be saved to a page header of the data page after the update.
S130: and when the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp.
When it is necessary to determine whether the data is completely and accurately written into the disk, the data page may be read from the disk into the memory, and the check code of the read data page may be calculated to obtain the second check code. The first check code corresponds to a checksum value, the second check code also corresponds to a checksum value, and then the checksum value of the first check code and the checksum value of the second check code may be compared. If the checksum value of the first check code is consistent with the checksum value of the second check code, it is indicated that the first check code is matched with the second check code, which may also indicate that the data page is not damaged, the data in the data page is complete, and the data page may be marked as a normal data page.
If the checksum value of the first check code is inconsistent with the checksum value of the second check code, it indicates that the data Page is damaged, and it is necessary to acquire the corresponding data Page from the backup of the storage class memory and acquire a Page Time Stamp (PTS) from the corresponding data Page. The log record corresponding to the page timestamp may then be found from the log file based on the page timestamp. Wherein the page time stamp may be an update time recorded at the time of the data page update.
S140: and updating the data of the data page according to the log record corresponding to the page time stamp.
And then starting from the log record corresponding to the page timestamp, updating the data page according to the log record until the current latest data version of the data page so as to update the data of the data page, wherein the data in the updated data page is complete data. In this embodiment, when data in the memory needs to be written into the disk, a log file of the data and a corresponding first check code are generated; and writing data into a data page of the disk, updating the second check code of the data page, and determining whether the first check code and the second check code are matched according to whether the first check code and the second check code are matched, wherein when the first check code and the second check code are consistent, the first check code and the second check code can be determined to be matched, and whether the data is partially written or not can be effectively detected. When the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp; according to the log record corresponding to the page time stamp, the data of the data page is updated, so that when partial writing of the data is detected, the corresponding log record can be accurately found according to the time stamp to update the data page, the situation that the partial writing of the data occurs is avoided, and due to the fact that the event of writing the data into the disk is recorded in a log mode, the writing operation can be truly executed on the SCM according to the original sequence, and the sequential writing of the data is guaranteed.
Referring to fig. 4, fig. 4 is a schematic flowchart illustrating a data storage method according to an embodiment of the present application, where the method includes: s210 to S270
S210: when data in the memory need to be written into the disk, a log file of the data and a corresponding first check code are generated.
In some embodiments, as shown in fig. 5, S210 may include: s211 to S212.
S211: a log record of the data is constructed in a buffer of the memory.
In some embodiments, when constructing a log record, the log record may be made to include a log tag and a log block, the log block being used to store the contents of a buffer in which the transaction was committed to the log area, and the log tag being used to describe information about the log record. For example, the journal tag includes a disk block number corresponding to the journal record, a journal record valid flag, a metadata flag bit, and the like.
Considering that data of a data page may not be updated in place when the data is stored, in some embodiments, when a Log record of the data is constructed in a buffer of the memory, a Log Time Stamp (LTS) corresponding to the Log record is generated, where the Log time stamp may indicate a construction time of the Log record at the time of construction, or may indicate that a Log record exists, is complete, and is verifiable before a certain time. Specifically, each time a log record is constructed, a log timestamp is generated correspondingly, and thus, each log record corresponds to a log timestamp. In the embodiment, each log record corresponds to one log timestamp, so that whether the data of the data page is not updated in place can be conveniently determined according to the log timestamps.
S212: and generating a log file and a corresponding first check code based on the log record.
S220: and writing the data into the data page of the disk, and updating the second check code of the data page.
The specific implementation of S220 can refer to S120, and therefore is not described herein.
S230: and judging whether the first check code is matched with the second check code.
S240: and when the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp.
The specific implementation of S240 can refer to S130, and therefore is not described herein.
S250: and updating the data of the data page according to the log record corresponding to the page time stamp.
The specific implementation of S250 can refer to S140, and therefore is not described herein.
In view of the potential for data page corruption during data storage, in some embodiments, a page timestamp corresponding to a data page may be generated when data for the data page is updated based on a log record corresponding to the page timestamp. Specifically, a data page may correspond to a page timestamp, and each update to the data page triggers an update to the page timestamp. In the embodiment, the page time stamp corresponding to the data page is generated when the data page is updated, so that the corresponding data page can be conveniently found out from the backup according to the page time stamp when the page is damaged.
S260: and when the first check code is matched with the second check code, judging whether the log timestamp is equal to the page timestamp or not.
When the first check code is matched with the second check code, it indicates that the data page is not damaged, and at this time, it needs to further determine whether the timestamp of the current data page is the latest to ensure whether the data of the data page is updated normally. It can therefore be determined whether the timestamp of the current data page is up-to-date by determining whether the log timestamp is equal to the page timestamp.
S270: when the log timestamp is equal to the page timestamp, then it is determined that the data update of the data page is normal.
When the log timestamp is equal to the page timestamp, it is determined that the timestamp of the current data page is the latest, and it may also be determined that the data update of the data page is normal, and at this time, the data page may be directly used. Alternatively, the data page may be a multiple of 512 bytes, so that the storage system with the SCM can conveniently and effectively organize the data page with the memory unit of the storage system and the SCM.
In this embodiment, through when first check code and second check code match, judge whether the log timestamp equals the page timestamp, thereby whether the update condition of the data of data page is unanimous with the log record construction condition can be effectively confirmed, when the log timestamp equals the page timestamp, then can confirm whether the data of preceding data page is synchronous with the log record that corresponds, and then confirm that the timestamp of current data page is newest, so the data page at this moment can directly use, thereby the data normal update of data page has been guaranteed.
Referring to fig. 6, fig. 6 is a schematic flow chart illustrating another data storage method according to an embodiment of the present application, where the method includes: S310-S390.
S310: when data in the memory need to be written into the disk, a log file of the data and a corresponding first check code are generated.
S320: and writing the data into the data page of the disk, and updating the second check code of the data page.
S330: and judging whether the first check code is matched with the second check code.
S340: and when the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp.
S350: and updating the data of the data page according to the log record corresponding to the page time stamp.
And S360, judging whether the log timestamp is equal to the page timestamp or not when the first check code is matched with the second check code.
S370: and when the log timestamp is equal to the page timestamp, determining that the data updating of the data page is normal.
The specific implementation of S310-S370 can refer to S210-S270, and therefore is not described herein.
S380: and when the log timestamp is not equal to the page timestamp, finding the checkpoint closest to the data page from a plurality of preset checkpoints to obtain the log record corresponding to the checkpoint.
As an example, each preset check point corresponds to a time, and then the check point closest to the time of the page timestamp can be found according to the page timestamp of the data page and the time of each preset check point.
Wherein checkpoints are generally used for database recovery, a new type of record is usually added to the log file, checkpoint records (checkpoint), a restart file is added, and the recovery subsystem dynamically maintains the log during logging into the log file. This can take a significant amount of time due to the need to check all log records when using logging techniques for database recovery. If the check point is used for recovery, the data after the latest check point can be directly recovered, and the time can be effectively saved.
When the log timestamp is not equal to the page timestamp, which is usually because the data update of the data page is delayed or the data is not updated in place and thus is not synchronized with the log timestamp, the log timestamp is usually larger than the page timestamp of the data page. The data of the data page needs to be recovered to resynchronize the page timestamp of the data page with the log timestamp of the log record.
In some embodiments, data recovery of a data page may be performed by a check point written in advance in a log file. Specifically, the checkpoint closest to the data page, that is, the current latest checkpoint, may be found from a plurality of preset checkpoints, and then the log file after the checkpoint is scanned from the latest checkpoint to obtain the log record corresponding to the checkpoint from the log file.
S390: and updating the data of the data page according to the log record corresponding to the check point.
And updating the data of the data page according to the log record corresponding to the checkpoint so as to recover the data of the data page after the latest checkpoint. Therefore, the page time stamp of the updated data page can be kept consistent with the latest log time stamp, and the data of the data page can be normally updated.
In some embodiments, before finding out the checkpoint closest to the data page from the plurality of preset checkpoints, as shown in fig. 7, the method of performing the checkpoint operation may be that the preset checkpoint may be written into the log file first, and a specific writing process thereof may include the following steps:
s301: and writing a plurality of preset check points in the log file.
In some embodiments, when multiple pre-defined checkpoints are written to the log file, the checkpointing operation may be performed at intervals, so that one checkpoint is obtained at intervals in the log file. In the embodiment, the checkpoint operation is performed once at intervals, so that when the checkpoint is subsequently used for data recovery, a proper checkpoint can be conveniently found, and the checkpoint closest to the data page or the checkpoint too far from the data page cannot be found.
It should be noted that the purpose of performing the checkpoint operation is to write a checkpoint in a corresponding log record in the log file when the modified data page is written back to the disk, so that the subsequent database can be recovered based on the checkpoint. Optionally, when the log record writes a checkpoint, the modified data page corresponding to the checkpoint may be deleted.
In some embodiments, a checkpoint operation may be performed each time a modified page of data is written back to disk or other storage medium (e.g., an SSD). In the embodiment, the checkpoint operation is performed each time when the modified data page is written back to the disk, so that the data page corresponding to the checkpoint obtained after the checkpoint operation is performed each time is ensured to be changed, and the validity of the checkpoint is ensured.
S302; and acquiring the address written into the log file by each preset check point.
S303: and writing a restart file at the address, wherein the restart file is used for returning the log file to the log record corresponding to the preset check point.
In the embodiment, the log file can be maintained conveniently by performing checkpoint operation during data storage.
In the present embodiment, data recovery of data of a data page is performed by using a check point when it is determined that the log timestamp and the data page timestamp do not coincide. Therefore, the page time stamp of the updated data page can be kept consistent with the latest log time stamp, and the data of the data page can be normally updated.
Referring to fig. 8, a block diagram of a data storage device 400 according to an embodiment of the present application is shown, where the data storage device 400 may include: the system comprises a first check code generation module 410, a second check code updating module 420, a log record acquisition module 430 and a data updating module 440.
The first check code generating module 410 is configured to generate a log file of data and a corresponding first check code when data in the memory needs to be written to a disk.
The second parity update module 420 is configured to write data into a data page of the disk, and update the second parity of the data page.
The log record obtaining module 430 is configured to obtain a page timestamp of the data page when the first check code is not matched with the second check code, and obtain a log record corresponding to the page timestamp from the log file according to the page timestamp.
The data updating module 440 is configured to update the data of the data page according to the log record corresponding to the page timestamp.
Referring to fig. 9, a block diagram of another data storage device according to an embodiment of the present application is shown, where the data storage device may further include, on the basis of the storage device 400 in fig. 8:
and a timestamp obtaining module 450, configured to obtain a page timestamp of the data page and a log timestamp of the log record when the first check code matches the second check code.
A timestamp determination module 460, configured to determine whether the log timestamp is equal to the page timestamp.
An update result determining module 470, configured to determine that the data update of the data page is normal when the log timestamp is equal to the page timestamp.
Further, the data update module 440 is further configured to: the check point searching module is used for finding out a check point closest to the data page from a plurality of preset check points to obtain a log record corresponding to the check point when the log timestamp is not equal to the page timestamp; and updating the data of the data page according to the log record corresponding to the check point.
Further, the data storage apparatus 400 may further include: the system comprises a check point setting module, a log file processing module and a checking point setting module, wherein the check point setting module is used for writing a plurality of preset check points in the log file; acquiring an address written into a log file by each preset check point; and writing a restart file at the address, wherein the restart file is used for returning the log file to the log record corresponding to the preset check point.
Further, the data storage apparatus 400 further includes: and the page time stamp generating module is used for generating a page time stamp corresponding to the data page when the data of the data page is updated based on the log record.
Further, the first check code generating module 410 is configured to construct a log record of data in a buffer of the memory; and generating a log file and a corresponding first check code based on the log record.
Further, the data storage apparatus 400 further includes: and the log timestamp generation module is used for generating a log timestamp corresponding to the log record when the log record of the data is constructed in the buffer area of the memory.
Further, the data page is a data page of a multiple of 512 bytes.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, the coupling between the modules may be electrical, mechanical or other type of coupling.
In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.
Referring to fig. 10, a block diagram of an electronic device according to an embodiment of the present application is shown. The electronic device 500 may be a smart phone, a tablet computer, an electronic book, or other electronic devices capable of running an application. The electronic device 500 in the present application may include one or more of the following components: a processor 510, a memory 520, and one or more applications, wherein the one or more applications may be stored in the memory 520 and configured to be executed by the one or more processors 510, the one or more programs configured to perform a method as described in the aforementioned method embodiments.
The Memory 520 may include a Random Access Memory (RAM) or a Read-Only Memory (Read-Only Memory). The memory 520 may be used to store instructions, programs, code sets, or instruction sets. The memory 520 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, and the like. The data storage area may also store data created during use by the electronic device 500 (e.g., phone books, audio-visual data, chat log data), and so forth.
Referring to fig. 11, a block diagram of a computer-readable storage medium according to an embodiment of the present application is shown. The computer-readable medium 600 has stored therein a program code that can be called by a processor to execute the method described in the above-described method embodiments.
The computer-readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 600 includes a non-volatile computer-readable storage medium. The computer readable storage medium 600 has storage space for program code 610 for performing any of the method steps of the method described above. The program code can be read from or written to one or more computer program products. The program code 610 may be compressed, for example, in a suitable form.
In summary, the above is provided. According to the data storage method, the data storage device, the electronic equipment and the computer readable medium, when data exist in a memory and need to be written into a disk, a log file of the data and a corresponding first check code are generated; and writing the data into a data page of the disk, updating the second check code of the data page, and effectively detecting whether the data is partially written according to whether the first check code is matched with the second check code. When the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp; according to the log record corresponding to the page time stamp, the data of the data page is updated, so that when partial writing of the data is detected, the corresponding log record can be accurately found according to the time stamp to update the data page, the situation that the partial writing of the data occurs is avoided, and due to the fact that the event of writing the data into the disk is recorded in a log mode, the writing operation can be truly executed on the SCM according to the original sequence, and the sequential writing of the data is guaranteed. In addition, by performing the checkpoint operation, it can be ensured that the data of the data page can be accurately replied and normally updated. Moreover, the data is written on the SCM in a mode of combining a log, a check code and a check point, so that the frequent use of instructions such as CLWB or MFENCE is not needed, the problem that the instructions need to be executed in a machine serialization mode, a cache cannot be well utilized, and great performance loss is brought to a system is avoided. While enabling the use of persistent storage features.
Therefore, data suitable for storage by the SCM can be moved to a corresponding location, and the selection method is based on the characteristics of the SCM, so that the service life of the equipment is delayed due to the performance of the whole system. On the other hand, the invention can save the memory by moving a part of data suitable for the SCM to the SCM.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (11)
1. A data storage method is applied to a storage class memory, wherein the storage class memory comprises a memory and a magnetic disk, and comprises the following steps:
when data in the memory needs to be written into the disk, generating a log file of the data and a corresponding first check code;
writing the data into a data page of the disk, and updating a second check code of the data page;
when the first check code is not matched with the second check code, acquiring a page time stamp of the data page, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp;
and updating the data of the data page according to the log record corresponding to the page time stamp.
2. The method of claim 1, further comprising:
when the first check code is matched with the second check code, acquiring a page time stamp of the data page and a log time stamp of the log record;
judging whether the log timestamp is equal to the page timestamp;
and when the log timestamp is equal to the page timestamp, determining that the data updating of the data page is normal.
3. The method of claim 2, further comprising:
when the log timestamp is not equal to the page timestamp, finding a check point closest to the data page from a plurality of preset check points to obtain a log record corresponding to the check point;
and updating the data of the data page according to the log record corresponding to the check point.
4. The method of claim 3, further comprising, before the updating the data of the data page according to the log record corresponding to the page timestamp:
writing a plurality of preset check points in the log file;
acquiring an address written into the log file by each preset check point;
and writing a restart file in the address, wherein the restart file is used for returning the log file to the log record corresponding to the preset check point.
5. The method of claim 1, further comprising:
generating a page timestamp corresponding to the data page when the data of the data page is updated based on the log record.
6. The method of claim 1, wherein generating the log file of the data and the corresponding first check code comprises:
constructing a log record of the data in a buffer area of the memory;
and generating the log file and a corresponding first check code based on the log record.
7. The method of claim 6, further comprising:
and when the log record of the data is constructed in the buffer area of the memory, generating a log time stamp corresponding to the log record.
8. The method of any of claims 1-7, wherein the data page is a multiple of 512 bytes of data page.
9. A data storage device, comprising:
the first check code generation module is used for generating a log file of the data and a corresponding first check code when the data in the memory needs to be written into the disk;
the second check code updating module is used for writing the data into a data page of the disk and updating the second check code of the data page;
the log record acquisition module is used for acquiring a page time stamp of the data page when the first check code is not matched with the second check code, and acquiring a log record corresponding to the page time stamp from the log file according to the page time stamp;
and the data updating module is used for updating the data of the data page according to the log record corresponding to the page time stamp.
10. An electronic device, comprising:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of claims 1-8.
11. A computer-readable storage medium storing program code executable by a processor, wherein a plurality of instructions in the program code, when executed by the processor, cause the processor to perform the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911088468.XA CN110928890B (en) | 2019-11-08 | 2019-11-08 | Data storage method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911088468.XA CN110928890B (en) | 2019-11-08 | 2019-11-08 | Data storage method and device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110928890A true CN110928890A (en) | 2020-03-27 |
| CN110928890B CN110928890B (en) | 2023-01-24 |
Family
ID=69852593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911088468.XA Active CN110928890B (en) | 2019-11-08 | 2019-11-08 | Data storage method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928890B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666046A (en) * | 2020-05-20 | 2020-09-15 | 西安奥卡云数据科技有限公司 | Data storage method, device and equipment |
| CN112631823A (en) * | 2020-12-30 | 2021-04-09 | 锐捷网络股份有限公司 | File processing method and device |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6779001B1 (en) * | 1999-09-29 | 2004-08-17 | Kabushiki Kaisha Toshiba | Transactional file system for realizing atomic update of plural files by transactions |
| US20090070304A1 (en) * | 2004-02-11 | 2009-03-12 | International Business Machines Corporation | Low-overhead built-in timestamp column for relational database systems |
| CN102436490A (en) * | 2010-10-28 | 2012-05-02 | 微软公司 | Versatile in-memory database recovery |
| CN102609337A (en) * | 2012-01-19 | 2012-07-25 | 北京神州数码思特奇信息技术股份有限公司 | Rapid data recovery method for memory database |
| CN102819493A (en) * | 2011-11-30 | 2012-12-12 | 凯迈（洛阳）环测有限公司 | Storage and query method based on non-volatile memory chip |
| CN103577121A (en) * | 2013-11-05 | 2014-02-12 | 中船重工（武汉）凌久电子有限责任公司 | High-reliability linear file access method based on nand flash |
| CN103617009A (en) * | 2013-12-10 | 2014-03-05 | 北京奇虎科技有限公司 | Method and device for writing data to disk through cache during starting up |
| US20140122959A1 (en) * | 2012-10-31 | 2014-05-01 | Lsi Corporation | Load Balanced Decoding of Low-Density Parity-Check Codes |
| CN103778030A (en) * | 2013-12-30 | 2014-05-07 | 上海晨思电子科技有限公司 | Write-in method, error tracking method and processor for blog subsystem |
| CN103942252A (en) * | 2014-03-17 | 2014-07-23 | 华为技术有限公司 | Method and system for recovering data |
| CN104182293A (en) * | 2013-05-22 | 2014-12-03 | 群联电子股份有限公司 | Data writing method, memory storage device and memory controller |
| CN104951474A (en) * | 2014-03-31 | 2015-09-30 | 阿里巴巴集团控股有限公司 | Method and device for acquiring MySQL binlog incremental logs |
| CN106788468A (en) * | 2016-11-28 | 2017-05-31 | 北京三快在线科技有限公司 | A kind of correcting and eleting codes update method and device, electronic equipment |
| CN109617648A (en) * | 2018-10-29 | 2019-04-12 | 青岛民航凯亚系统集成有限公司 | A kind of variable time sliding window calculation method |
| WO2019171133A1 (en) * | 2018-03-07 | 2019-09-12 | Pratik Sharma | Grouping dependent database transactions for recovery |
| CN110309013A (en) * | 2018-03-20 | 2019-10-08 | 慧荣科技股份有限公司 | The method of access flash memory module and relevant flash controller and electronic device |
-
2019
- 2019-11-08 CN CN201911088468.XA patent/CN110928890B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6779001B1 (en) * | 1999-09-29 | 2004-08-17 | Kabushiki Kaisha Toshiba | Transactional file system for realizing atomic update of plural files by transactions |
| US20090070304A1 (en) * | 2004-02-11 | 2009-03-12 | International Business Machines Corporation | Low-overhead built-in timestamp column for relational database systems |
| CN102436490A (en) * | 2010-10-28 | 2012-05-02 | 微软公司 | Versatile in-memory database recovery |
| CN102819493A (en) * | 2011-11-30 | 2012-12-12 | 凯迈（洛阳）环测有限公司 | Storage and query method based on non-volatile memory chip |
| CN102609337A (en) * | 2012-01-19 | 2012-07-25 | 北京神州数码思特奇信息技术股份有限公司 | Rapid data recovery method for memory database |
| US20140122959A1 (en) * | 2012-10-31 | 2014-05-01 | Lsi Corporation | Load Balanced Decoding of Low-Density Parity-Check Codes |
| CN104182293A (en) * | 2013-05-22 | 2014-12-03 | 群联电子股份有限公司 | Data writing method, memory storage device and memory controller |
| CN103577121A (en) * | 2013-11-05 | 2014-02-12 | 中船重工（武汉）凌久电子有限责任公司 | High-reliability linear file access method based on nand flash |
| CN103617009A (en) * | 2013-12-10 | 2014-03-05 | 北京奇虎科技有限公司 | Method and device for writing data to disk through cache during starting up |
| CN103778030A (en) * | 2013-12-30 | 2014-05-07 | 上海晨思电子科技有限公司 | Write-in method, error tracking method and processor for blog subsystem |
| CN103942252A (en) * | 2014-03-17 | 2014-07-23 | 华为技术有限公司 | Method and system for recovering data |
| CN104951474A (en) * | 2014-03-31 | 2015-09-30 | 阿里巴巴集团控股有限公司 | Method and device for acquiring MySQL binlog incremental logs |
| CN106788468A (en) * | 2016-11-28 | 2017-05-31 | 北京三快在线科技有限公司 | A kind of correcting and eleting codes update method and device, electronic equipment |
| WO2019171133A1 (en) * | 2018-03-07 | 2019-09-12 | Pratik Sharma | Grouping dependent database transactions for recovery |
| CN110309013A (en) * | 2018-03-20 | 2019-10-08 | 慧荣科技股份有限公司 | The method of access flash memory module and relevant flash controller and electronic device |
| CN109617648A (en) * | 2018-10-29 | 2019-04-12 | 青岛民航凯亚系统集成有限公司 | A kind of variable time sliding window calculation method |
Non-Patent Citations (4)
| Title |
|---|
| J. JEONG 等: "Efficient Hardware-Assisted Logging with Asynchronous and Direct-Update for Persistent Memory：IEEE， log update data store", 《2018 51ST ANNUAL IEEE/ACM INTERNATIONAL SYMPOSIUM ON MICROARCHITECTURE》 * |
| 人邮异步社区: "系统可能在任何两次写入之间崩溃或断电，崩溃后，如何更新磁盘？", 《HTTPS://BLOG.CSDN.NET/EPUBIT17/ARTICLE/DETAILS/99277412》 * |
| 刘烊: "嵌入式系统中数据存储保护的关键技术研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 刘钢: "分布式存储系统中基于纠错码的容错机制研究", 《中国博士学位论文全文数据库信息科技辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666046A (en) * | 2020-05-20 | 2020-09-15 | 西安奥卡云数据科技有限公司 | Data storage method, device and equipment |
| CN111666046B (en) * | 2020-05-20 | 2023-07-25 | 西安奥卡云数据科技有限公司 | Data storage method, device and equipment |
| CN112631823A (en) * | 2020-12-30 | 2021-04-09 | 锐捷网络股份有限公司 | File processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110928890B (en) | 2023-01-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10983955B2 (en) | Data unit cloning in memory-based file systems | |
| US10176190B2 (en) | Data integrity and loss resistance in high performance and high capacity storage deduplication | |
| CN108431783B (en) | Access request processing method and device and computer system | |
| US8954673B1 (en) | Using a conditional read request and a hash to determine synchronization of data in a cache at a host with data in storage array | |
| US9632932B1 (en) | Backup-power-free cache memory system | |
| US12093140B2 (en) | Data recovery method, apparatus, and solid state drive | |
| US10445211B2 (en) | Logging trace data for program code execution at an instruction level | |
| US11042469B2 (en) | Logging trace data for program code execution at an instruction level | |
| WO2015012871A1 (en) | Methods and apparatus for supporting persistent memory | |
| CN110196818B (en) | Data caching method, caching device and storage system | |
| US10489289B1 (en) | Physical media aware spacially coupled journaling and trim | |
| US9003228B2 (en) | Consistency of data in persistent memory | |
| US8370587B2 (en) | Memory system storing updated status information and updated address translation information and managing method therefor | |
| US20180300083A1 (en) | Write-ahead logging through a plurality of logging buffers using nvm | |
| Kim et al. | Reducing excessive journaling overhead with small-sized NVRAM for mobile devices | |
| US20130246716A1 (en) | Memory system and data writing method | |
| CN111125040B (en) | Method, device and storage medium for managing redo log | |
| CN108431784B (en) | Access request processing method and device and computer system | |
| US12061557B2 (en) | Method for storing L2P table, system, device, and medium | |
| CN113076220B (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN110928890B (en) | Data storage method and device, electronic equipment and computer readable storage medium | |
| CN112799595A (en) | Data processing method, device and storage medium | |
| US11886728B2 (en) | Undo capability for memory devices | |
| CN110968530B (en) | Key value storage system based on nonvolatile memory and memory access method | |
| US11237925B2 (en) | Systems and methods for implementing persistent data structures on an asymmetric non-volatile memory architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20200327 Assignee: GUANGZHOU CUBESILI INFORMATION TECHNOLOGY Co.,Ltd. Assignor: GUANGZHOU HUADUO NETWORK TECHNOLOGY Co.,Ltd. Contract record no.: X2021440000031 Denomination of invention: Data storage method, device, electronic device and computer readable medium License type: Common License Record date: 20210125 |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |