CN106682193B - Data persistence storage method and device based on cache - Google Patents
Data persistence storage method and device based on cache Download PDFInfo
- Publication number
- CN106682193B CN106682193B CN201611249423.2A CN201611249423A CN106682193B CN 106682193 B CN106682193 B CN 106682193B CN 201611249423 A CN201611249423 A CN 201611249423A CN 106682193 B CN106682193 B CN 106682193B
- Authority
- CN
- China
- Prior art keywords
- data
- cache
- database
- time stamp
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1032—Reliability improvement, data loss prevention, degraded operation etc
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/16—General purpose computing application
- G06F2212/163—Server or database system
Abstract
The invention discloses a data persistence storage method and device based on cache, which can at least solve the technical problems that the existing cache mechanism has poor disaster tolerance and can not prevent data in a database from making mistakes when the cache fails. The method comprises the following steps: when data are written into the cache, corresponding timestamps are respectively set for each data in the cache according to the data writing time; when each data in the cache is stored in the database in a persistent mode, respectively acquiring a time stamp of the data in the cache and a time stamp of a data record corresponding to the data in the database aiming at each data in the cache; when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database, deleting the data in the cache; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a data persistence storage method and device based on cache.
Background
The cache is a buffer area for data exchange, and the cache has a high read-write speed, so that the efficiency of data reading or writing can be improved by using the cache. For example, when data is read, the required data is searched from the cache, and if the required data is searched, the data is directly read from the cache; when data is written, the data to be written is written into the cache, and then the data in the cache is stored into the database in a persistent mode. Therefore, the cache mechanism can bring convenience to the read-write process, and therefore the cache mechanism is more and more widely applied.
However, the inventor finds that the existing caching mechanism has at least the following problems in the process of implementing the invention: when the cache is normal, the data stored in the cache can be orderly stored in the database according to the preset rule. However, once a cache fails, the data stored in the cache may be stored out of order or even incorrectly in the database when the failure is recovered. For example, suppose that a part of data in the cache fails temporarily and cannot be read, and therefore, when the data in the cache is stored to the database persistently, the part of data is missed; after a period of time, the part of data in the cache returns to normal, and the part of data is stored in the database in the process of storing the data in the cache to the database in a persistent mode next time. However, in practical situations, the portion of data is likely to have become out-of-date, erroneous data, thereby causing errors in the data in the database. Therefore, the existing cache mechanism has poor disaster tolerance and cannot prevent data in the database from being mistaken when the cache fails.
Disclosure of Invention
In view of the above, the present invention has been made to provide a cache-based data persistence storage method and apparatus that overcomes or at least partially solves the above-mentioned problems.
According to one aspect of the invention, a cache-based data persistent storage method is provided, which comprises the following steps: when data are written into the cache, corresponding timestamps are respectively set for each data in the cache according to the data writing time; when each data in the cache is stored in a database in a persistent mode, respectively acquiring a timestamp of the data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each data in the cache; when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database, deleting the data in the cache; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
According to another aspect of the present invention, there is provided a cache-based data persistent storage apparatus, including: the setting module is suitable for setting corresponding timestamps for each data in the cache according to the data writing time when the data are written into the cache; the acquisition module is suitable for respectively acquiring a timestamp of each piece of data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each piece of data in the cache when each piece of data in the cache is persistently stored in the database; the processing module is suitable for deleting the data in the cache when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
In the method and the device for the data persistence storage based on the cache, when data are written into the cache, corresponding timestamps are respectively set for each data in the cache according to the data writing time; then when each data in the cache is stored in the database in a persistent mode, respectively acquiring a timestamp of the data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each data in the cache; finally, when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database, deleting the data in the cache; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode. Therefore, the corresponding time stamps are respectively set for each data in the cache, and correspondingly, when certain data in the cache is stored in the database in a persistent mode, whether the data in the cache is expired invalid data is judged by comparing the time stamp of the data in the cache with the time stamp of the data record corresponding to the data in the database, so that the reliability in the data persistence process is improved, and the data error phenomenon caused by cache faults is effectively prevented.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a schematic flowchart illustrating a cache-based data persistent storage method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart illustrating a cache-based data persistent storage method according to a second embodiment of the present invention;
fig. 3 shows a block diagram of a cache-based data persistent storage device according to a third embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The embodiment of the invention provides a data persistence storage method and device based on a cache, which can at least solve the technical problems that the existing cache mechanism has poor disaster tolerance and can not prevent data in a database from making mistakes when the cache fails.
Example one
Fig. 1 shows a flowchart of a cache-based data persistent storage method according to an embodiment of the present invention, where the method includes:
step S110: and when data are written into the cache, respectively setting corresponding time stamps for each data in the cache according to the data writing time.
Wherein the time stamp is used to indicate the time when the data is written to the buffer. The specific format and precision of the time stamp can be flexibly set by those skilled in the art, and the present invention is not limited thereto. For example, if a data is written into the cache at 10 am 01 on 12/13/2016, the corresponding timestamp of the data is 201612131001. In addition, when the data is stored in the cache, the data and the corresponding timestamp thereof may be stored in association as one data group, or the timestamp may be stored as one attribute of the data itself, and the specific storage form is not limited in the present invention.
Step S120: when each piece of data in the cache is stored in the database in a persistent mode, the time stamp of the data in the cache and the time stamp of the data record corresponding to the data in the database are obtained respectively aiming at each piece of data in the cache.
The time for storing each data into the database in a persistent manner can be flexibly set by a person skilled in the art, which is not limited by the present invention. During persistent storage, firstly, a timestamp of the data in a cache is acquired; then, a data record corresponding to the data is searched in the database, and a time stamp of the data record corresponding to the data is obtained. The data records corresponding to the data may be stored in various forms.
Step S130: when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database, deleting the data in the cache; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
Specifically, the time stamp is used to indicate the writing time of the data, and may be represented in a numerical form. In the invention, the sequence of time is represented by the size of the time stamp: the smaller the timestamp of the data before the time, the larger the timestamp of the data after the time. It can be seen that comparing the sizes of the timestamps is actually determining the time sequence. For example, suppose that the write time of data one is 2016, 12, 13, 10 am, 01 minutes, and the timestamp is 201612131001; the writing time of the data two is 2016, 12 and 14 days, 10 am and 01 minutes, the time stamp is 201612141001, and the writing time of the data one is earlier than that of the data two, so that the time stamp of the data one is smaller than that of the data two.
Accordingly, when the timestamp of the data in the cache is smaller than the timestamp of the data record corresponding to the data in the database, it indicates that the write time of the data in the cache is earlier than the write time of the corresponding data record in the database, that is: the data in the cache is invalid data which has expired, so that the data in the cache is deleted to prevent a data error phenomenon. When the timestamp of the data in the cache is not less than the timestamp of the data record corresponding to the data in the database, it is indicated that the writing time of the data in the cache is later than the writing time of the corresponding data record in the database, that is: the data in the cache is valid data which is not expired, so that the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
Therefore, in the data persistence storage method based on the cache, when data is written into the cache, corresponding timestamps are respectively set for each data in the cache according to the data writing time; then when each data in the cache is stored in the database in a persistent mode, respectively acquiring a timestamp of the data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each data in the cache; finally, when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database, deleting the data in the cache; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode. Therefore, the method in the embodiment of the invention respectively sets the corresponding time stamps for each data in the cache, and correspondingly, when a certain data in the cache is stored in the database in a persistent manner, whether the data in the cache is expired invalid data is judged by comparing the time stamp of the data in the cache with the time stamp of the data record corresponding to the data in the database, so that the reliability in the data persistence process is improved, and the data error phenomenon caused by cache failure is effectively prevented.
Example two
Fig. 2 is a schematic flowchart illustrating a cache-based data persistent storage method according to a second embodiment of the present invention, and as shown in fig. 2, the method includes:
step S210: and when data are written into the cache, respectively setting corresponding time stamps for each data in the cache according to the data writing time.
Wherein the time stamp is used to indicate the time when the data is written to the buffer. The specific format and precision of the time stamp can be flexibly set by those skilled in the art, and the present invention is not limited thereto. For example, if a data is written into the cache at 10 am 01 on 12/13/2016, the corresponding timestamp of the data is 201612131001.
In addition, when storing data into the cache, various storage methods can be adopted. In the first storage method, the data and the corresponding time stamp may be stored in association as one data group. In addition, when the associative storage is performed, two types of operations are actually involved, the first type of operation is data writing, the second type of operation is a time stamp corresponding to the data writing, in order to prevent an error caused by successful execution of one type of operation and failed execution of the other type of operation, the first type of operation and the second type of operation can be executed together as a transaction, the transaction is submitted only when the first type of operation and the second type of operation are all successfully executed, otherwise, the transaction is rolled back to perform the data writing operation again, and therefore the accuracy of each data and the time stamp thereof in the cache is ensured. In the second storage method, the time stamp may be stored as an attribute of the data itself, or the time stamp may be stored as a suffix or a prefix of the data. The present invention is not limited to a specific storage form.
In addition, in the present embodiment, the writing of data includes various types of write operations such as direct writing and modification.
Step S220: when each piece of data in the cache is stored in the database in a persistent mode, the time stamp of the data in the cache and the time stamp of the data record corresponding to the data in the database are obtained respectively aiming at each piece of data in the cache.
In this embodiment, the above operation of persistently storing each data in the cache to the database may be performed when a preset persistent condition is satisfied, and a specific persistent condition may be flexibly set by a person skilled in the art according to a requirement. For example, in one implementation of this embodiment, the persistence condition is: the written data in the cache reaches a preset proportion. As another example, in another implementation, the persistence condition is: when the preset persistence time is reached, or the time from the last persistence reaches the preset time interval. The specific persistence time and the predetermined time interval can be flexibly set by those skilled in the art.
Specifically, when storing persistently, first, a timestamp of data to be persisted in a cache is to be acquired; then, a data record corresponding to the data is searched in the database, and a time stamp of the data record corresponding to the data is obtained. In addition, because the same data is often written into the cache for multiple times in sequence, in this step, the timestamps corresponding to the data in the cache at each time of writing can be respectively obtained, and the maximum value (the timestamp indicates that the writing time is latest) in the timestamps corresponding to each time of writing is determined as the timestamp of the data in the cache; and respectively acquiring the time stamps of the data records corresponding to the data in the database, and determining the maximum value in the time stamps of the data records as the time stamp of the data record corresponding to the data in the database. Therefore, the time of the data when the data is written in the cache last can be determined by determining the data with the maximum timestamp, the time of the corresponding data record when the data is written in the database last can be determined by determining the data record with the maximum timestamp, and whether the data in the cache is effective or not can be judged quickly and accurately. In addition, because data is generally uniquely identified by a key value, data with the same key value is considered as the same data, and correspondingly, in the above process, a corresponding timestamp of each writing of the data is determined according to the key value of the data in the cache; and searching each data record corresponding to the data in the database according to the key value of the data in the cache, and determining the time stamp of each data record.
Step S230: judging whether the timestamp of the data in the cache is smaller than the timestamp of the data record corresponding to the data in the database, if so, executing the step S240; if the determination result is negative, step S250 is executed.
Specifically, the time stamp is used to indicate the writing time of the data, and may be represented in a numerical form. In the invention, the sequence of time is represented by the size of the time stamp: the smaller the timestamp of the data before the time, the larger the timestamp of the data after the time. It can be seen that comparing the sizes of the timestamps is actually determining the time sequence. For example, suppose that the write time of data one is 2016, 12, 13, 10 am, 01 minutes, and the timestamp is 201612131001; the writing time of the data two is 2016, 12 and 14 days, 10 am and 01 minutes, the time stamp is 201612141001, and the writing time of the data one is earlier than that of the data two, so that the time stamp of the data one is smaller than that of the data two.
Step S240: the data in the cache is deleted.
Specifically, when the timestamp of the data in the cache is smaller than the timestamp of the data record corresponding to the data in the database, it indicates that the write time of the data in the cache is earlier than the write time of the corresponding data record in the database, that is: the data in the cache is invalid data which has expired, so that the data in the cache is deleted to prevent a data error phenomenon.
In addition, in this step, when it is determined that the timestamp of the data in the cache is smaller than the timestamp of the data record corresponding to the data in the database, the following operations may be further included: and storing the data in the database and the corresponding time stamp in a cache. Because the cache has many advantages of high access speed and the like, when the data in the cache is judged to be the expired invalid data, the data in the database and the corresponding timestamp are further stored in the cache, so that the effect of improving the access speed is realized. In addition, the data lost during the cache failure can be recovered by saving the data in the database and the corresponding timestamp thereof in the cache. Specifically, the data in the database actually refers to the data record in the database corresponding to the data. In practical cases, the storage form of the data in the cache may be the same as or different from the storage form of the data record corresponding to the data in the database. For example, the data in the cache may be stored in a Word document, and the data record in the database corresponding to the data may be stored in an Excel table.
Step S250: and persistently storing the data in the cache and the corresponding time stamp into a database.
Specifically, when the timestamp of the data in the cache is not less than the timestamp of the data record corresponding to the data in the database, it is indicated that the write time of the data in the cache is later than the write time of the corresponding data record in the database, that is: the data in the cache is valid data which is not expired, so that the data in the cache and the corresponding time stamp are stored in the database in a persistent mode. By judging the time stamp, the correctness of the data stored in the database can be improved.
In addition, in order to further improve the disaster recovery performance of the cache system, in the execution process of the above step S210 to step S250, the following steps may be further included: when a cache failure is detected, data generated during the cache failure and a timestamp corresponding to the data are stored in a database in a persistent mode. The present invention does not limit the specific execution time of the step. For example, this step may be executed after step S240, so that this step is triggered when the cache is found to have expired error data; or automatically detecting whether the cache fails or not at preset time intervals, and triggering the step when the detection result is yes. Through the steps, the data generated during the cache failure period and the corresponding time stamp can be directly stored in the database in a persistent mode, so that the data loss is prevented, and the reliable operation of the whole cache system is guaranteed. In addition, on this basis, when the persistent storage operation is executed after the cache failure is recovered, the timestamps of the cache data are further compared, so that the write time of the part of data which is directly persistent without passing through the cache in the database can be judged, and when the timestamp of one data in the cache is determined to be smaller than the timestamp of the data record corresponding to the data in the database, the data in the database and the timestamp corresponding to the data in the database are stored in the cache, so that the data during the cache failure can be completely recovered, and the disaster tolerance performance of the cache system is improved.
In addition, various modifications and alterations to the above-described embodiments may be made by those skilled in the art. For example, each step in the above embodiments may be split into more steps, or combined into fewer steps, and the execution order between the steps may also be adjusted, or even a part of the steps may be deleted or some steps may be added, and in short, the present invention does not limit the specific implementation details.
In addition, the inventor finds that in the process of implementing the present invention, most of the common caches are implemented by non-persistent memories or other storage devices, and the non-persistent caches have the characteristic of being powered off and then being dropped, that is: after a power failure or malfunction, the data stored thereon will be subsequently emptied, resulting in loss of the buffered data. Moreover, the non-persistent cache often has the limitation of smaller capacity and the like. To solve the above problem, in the present embodiment, a persistent cache implementation may be utilized. For example, persistent caches include, but are not limited to: SSDs, solid state drives, and/or mechanical hard drives, etc. These persistent caches often have the advantages of fast read/write speed and large storage capacity. Moreover, in this embodiment, a timestamp is set for each data, so that even if the persistent cache is in an abnormal condition such as power failure or failure, the data stored thereon is still retained, and whether each data is still valid can be determined according to the timestamp of each data, thereby improving the performance of the cache system.
EXAMPLE III
Fig. 3 shows a block diagram of a cache-based data persistent storage device according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes: a setting module 31, an obtaining module 32, a processing module 33, a time storage module 34 and a storage processing module 35.
The setting module 31 is adapted to set a corresponding timestamp for each data in the cache according to the data writing time when the data is written into the cache.
Wherein the time stamp is used to indicate the time when the data is written to the buffer. The specific format and precision of the time stamp can be flexibly set by those skilled in the art, and the present invention is not limited thereto. For example, if a data is written into the cache at 10 am 01 on 12/13/2016, the corresponding timestamp of the data is 201612131001. In addition, when the setting module 31 sets corresponding timestamps for each data in the cache according to the data writing time, a plurality of storage modes can be set. In the first storage method, the setting module 31 may store the data and the corresponding time stamp in association as a data group. In addition, when the associative storage is performed, two types of operations are actually involved, the first type of operation is data writing, the second type of operation is a time stamp corresponding to the data writing, in order to prevent an error caused by successful execution of one type of operation and failed execution of the other type of operation, the first type of operation and the second type of operation can be executed together as a transaction, the transaction is submitted only when the first type of operation and the second type of operation are all successfully executed, otherwise, the transaction is rolled back to perform the data writing operation again, and therefore the accuracy of each data and the time stamp thereof in the cache is ensured. In the second storage method, the setting module 31 may store the timestamp as an attribute of the data itself, or store the timestamp as a suffix or a prefix of the data. The present invention is not limited to a specific storage form. In addition, in the present embodiment, the writing of data includes various types of write operations such as direct writing and modification.
The obtaining module 32 is adapted to, when storing each piece of data in the cache to the database persistently, obtain, for each piece of data in the cache, a timestamp of the piece of data in the cache and a timestamp of a data record corresponding to the piece of data in the database, respectively.
In this embodiment, the obtaining module 32 may perform the above operation of persistently storing each data in the cache to the database when a preset persistent condition is met, and a specific persistent condition may be flexibly set by a person skilled in the art according to a requirement. For example, in one implementation of this embodiment, the persistence condition is: the written data in the cache reaches a preset proportion. As another example, in another implementation, the persistence condition is: when the preset persistence time is reached, or the time from the last persistence reaches the preset time interval. The specific persistence time and the predetermined time interval can be flexibly set by those skilled in the art.
Specifically, in the persistent storage, first, the obtaining module 32 needs to obtain a timestamp of data to be persisted in the cache; then, a data record corresponding to the data is searched in the database, and a time stamp of the data record corresponding to the data is obtained. In addition, because the same data often performs multiple write operations in the cache, the obtaining module 32 may obtain the timestamps corresponding to the data in the cache at each write time, and determine the maximum value (the maximum timestamp indicates the latest write time) in the timestamps corresponding to each write time as the timestamp of the data in the cache; and respectively acquiring the time stamps of the data records corresponding to the data in the database, and determining the maximum value in the time stamps of the data records as the time stamp of the data record corresponding to the data in the database. Therefore, the time of the data when the data is written last in the cache can be determined by determining the data with the largest timestamp through the obtaining module 32, the time of the corresponding data record when the data is written last in the database can be determined by determining the data record with the largest timestamp through the obtaining module 32, and whether the data in the cache is valid can be judged quickly and accurately. In addition, because data is generally uniquely identified by a key value, data with the same key value is considered as the same data, and correspondingly, a corresponding timestamp of each writing of the data is determined according to the key value of the data in the cache; and searching each data record corresponding to the data in the database according to the key value of the data in the cache, and determining the time stamp of each data record.
The processing module 33 is adapted to delete the data in the cache when it is determined that the timestamp of the data in the cache is less than the timestamp of the data record in the database corresponding to the data; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
Specifically, the time stamp is used to indicate the writing time of the data, and may be represented in a numerical form. In the invention, the sequence of time is represented by the size of the time stamp: the smaller the timestamp of the data before the time, the larger the timestamp of the data after the time. Thus, the processing module 33 compares the sizes of the time stamps to determine the time sequence. For example, suppose that the write time of data one is 2016, 12, 13, 10 am, 01 minutes, and the timestamp is 201612131001; the writing time of the data two is 2016, 12 and 14 days, 10 am and 01 minutes, the time stamp is 201612141001, and the writing time of the data one is earlier than that of the data two, so that the time stamp of the data one is smaller than that of the data two. When the processing module 33 determines that the timestamp of the data in the cache is smaller than the timestamp of the data record corresponding to the data in the database, it indicates that the write time of the data in the cache is earlier than the write time of the corresponding data record in the database, that is: the data in the cache is invalid data which has expired, so that the data in the cache is deleted to prevent a data error phenomenon. When the timestamp of the data in the cache is not less than the timestamp of the data record corresponding to the data in the database, it is indicated that the writing time of the data in the cache is later than the writing time of the corresponding data record in the database, that is: the data in the cache is valid data that has not expired, so the processing module 33 persistently stores the data in the cache and the corresponding timestamp in the database. By judging the time stamp, the correctness of the data stored in the database can be improved.
The time storage module 34 is adapted to store the data in the database and the corresponding time stamp in the cache.
When the processing module 33 determines that the timestamp of the data in the cache is smaller than the timestamp of the data record corresponding to the data in the database, the time storage module 34 may further store the data in the database and the timestamp corresponding to the data in the cache. Since the cache has many advantages such as fast access speed, when the processing module 33 determines that the data in the cache is the expired invalid data, the time storage module 34 further stores the data in the database and the corresponding timestamp thereof in the cache, so as to achieve the effect of increasing the access speed. In addition, the time storage module 34 may also recover data lost during a cache failure by saving the data in the database and its corresponding timestamp to the cache. Specifically, the data in the database actually refers to the data record in the database corresponding to the data. In practical cases, the storage form of the data in the cache may be the same as or different from the storage form of the data record corresponding to the data in the database. For example, the data in the cache may be stored in a Word document, and the data record in the database corresponding to the data may be stored in an Excel table.
The storage handling module 35 is adapted to, when a cache failure is detected, persistently store data generated during the cache failure and its corresponding time stamp in the database.
Specifically, the storage processing module 35 is configured to, when a cache failure is detected, persistently store data generated during the cache failure and a timestamp corresponding to the data into the database, so as to achieve a purpose of improving the disaster recovery performance of the cache system. Here, the execution timing of the storage processing module 35 may trigger the storage processing module 35 to execute the corresponding persistent storage operation when finding that there is expired error data in the cache after deleting the data in the cache; or, it may also be that whether the cache fails is automatically detected every preset time interval, and when the detection result is yes, the storage processing module 35 is triggered to execute the corresponding persistent storage operation, and the specific execution time of the storage processing module 35 is not limited in the present invention. Therefore, the data generated during the cache failure and the corresponding timestamp thereof can be directly stored into the database in a persistent manner through the storage processing module 35, so that the data loss is prevented, and the reliable operation of the whole cache system is guaranteed.
The specific structure and the working principle of each module may refer to the description of the corresponding step in the method embodiment, and are not described herein again.
Therefore, in the data persistence storage device based on the cache provided by the invention, when data is written into the cache through the setting module 31, corresponding timestamps are respectively set for each data in the cache according to the data writing time; then, when each data in the cache is stored in the database persistently through the obtaining module 32, the time stamp of the data in the cache and the time stamp of the data record corresponding to the data in the database are respectively obtained for each data in the cache; when the processing module 33 determines that the timestamp of the data in the cache is smaller than the timestamp of the data record corresponding to the data in the database, the data in the cache is deleted; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode. Meanwhile, the data and the corresponding timestamp in the database are further saved in the cache by the time storage module 34, and when a cache failure is detected, the data and the corresponding timestamp generated during the cache failure are stored in the database in a persistent manner by the storage processing module 35. Therefore, the method in the embodiment of the invention respectively sets the corresponding time stamps for each data in the cache, correspondingly, when a certain data in the cache is stored in the database in a persistent manner, whether the data in the cache is expired invalid data is judged by comparing the time stamp of the data in the cache with the time stamp of the data record corresponding to the data in the database, so that the reliability in the data persistence process is improved, the data error phenomenon caused by cache failure is effectively prevented, and even if the persistent cache has abnormal conditions such as power failure or failure, the data stored in the persistent cache still remains, and whether each data is still valid can be judged according to the time stamp of each data, so that the performance of the cache system is improved.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a message queue consuming device according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
The invention discloses: a1, a cache-based data persistence storage method, comprising:
when data are written into the cache, corresponding timestamps are respectively set for each data in the cache according to the data writing time;
when each data in the cache is stored in a database in a persistent mode, respectively acquiring a timestamp of the data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each data in the cache;
when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database, deleting the data in the cache; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
A2, the method of A1, wherein when it is determined that the timestamp of the data in the cache is less than the timestamp of the data record corresponding to the data in the database, the method further comprises: and storing the data in the database and the corresponding time stamp into the cache.
A3, the method of A1 or A2, further comprising: when the cache failure is detected, persistently storing data generated during the cache failure and a timestamp corresponding to the data into the database.
A4, the method according to any one of a1-A3, wherein the step of respectively obtaining the timestamp of the data in the cache and the timestamp of the data record corresponding to the data in the database specifically includes:
respectively acquiring corresponding time stamps of the data in the cache when the data are written in each time, and determining the maximum value of the corresponding time stamps when the data are written in each time as the time stamp of the data in the cache;
and respectively acquiring the time stamps of the data records corresponding to the data in the database, and determining the maximum value in the time stamps of the data records as the time stamp of the data record corresponding to the data in the database.
A5, the method according to a4, wherein the step of respectively obtaining the corresponding time stamp of each write of the data in the cache specifically includes: determining a corresponding time stamp of each writing of the data according to the key value of the data in the cache;
the step of respectively obtaining the time stamp of each data record corresponding to the data in the database specifically includes: and searching each data record corresponding to the data in the database according to the key value of the data in the cache, and determining the time stamp of each data record.
A6, the method according to any of A1-A5, wherein the cache is a persistent cache, wherein,
the persistent cache includes: solid state disks and/or mechanical hard disks.
The invention also discloses: b7, a cache-based data persistence storage device, comprising:
the setting module is suitable for setting corresponding timestamps for each data in the cache according to the data writing time when the data are written into the cache;
the acquisition module is suitable for respectively acquiring a timestamp of each piece of data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each piece of data in the cache when each piece of data in the cache is persistently stored in the database;
the processing module is suitable for deleting the data in the cache when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
B8, the apparatus according to B7, wherein the apparatus further comprises: and the time storage module is suitable for storing the data in the database and the corresponding time stamp into the cache.
B9, the apparatus according to B7 or B8, wherein the apparatus further comprises: and the storage processing module is suitable for storing the data generated during the cache failure and the corresponding time stamp into the database in a persistent mode when the cache failure is detected.
B10, the apparatus according to any one of B7-B9, wherein the obtaining module is further configured to:
respectively acquiring corresponding time stamps of the data in the cache when the data are written in each time, and determining the maximum value of the corresponding time stamps when the data are written in each time as the time stamp of the data in the cache;
and respectively acquiring the time stamps of the data records corresponding to the data in the database, and determining the maximum value in the time stamps of the data records as the time stamp of the data record corresponding to the data in the database.
B11, the device according to B10, wherein the obtaining module is specifically configured to: determining a corresponding time stamp of each writing of the data according to the key value of the data in the cache;
the acquisition module is specifically configured to: and searching each data record corresponding to the data in the database according to the key value of the data in the cache, and determining the time stamp of each data record.
B12, the apparatus according to any one of B7-B11, wherein the cache is a persistent cache, wherein the persistent cache comprises: solid state disks and/or mechanical hard disks.
Claims (12)
1. A cache-based data persistence storage method, comprising:
when data are written into the cache, corresponding timestamps are respectively set for each data in the cache according to the data writing time; the method comprises the steps that written data are first-class operations, a timestamp corresponding to the written data is second-class operations, and the first-class operations and the second-class operations are jointly used as a transaction to be executed; when the first type of operation and the second type of operation are all executed successfully, the transaction is submitted; otherwise, roll back the transaction;
when each data in the cache is stored in a database in a persistent mode, respectively acquiring a timestamp of the data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each data in the cache;
when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database, deleting the data in the cache; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
2. The method of claim 1, wherein when it is determined that the timestamp of the data in the cache is less than the timestamp of the data record in the database corresponding to the data, the method further comprises: and storing the data in the database and the corresponding time stamp into the cache.
3. The method of claim 1 or 2, further comprising: when the cache failure is detected, persistently storing data generated during the cache failure and a timestamp corresponding to the data into the database.
4. The method according to claim 3, wherein the step of respectively obtaining the timestamp of the data in the cache and the timestamp of the data record corresponding to the data in the database specifically comprises:
respectively acquiring corresponding time stamps of the data in the cache when the data are written in each time, and determining the maximum value of the corresponding time stamps when the data are written in each time as the time stamp of the data in the cache;
and respectively acquiring the time stamps of the data records corresponding to the data in the database, and determining the maximum value in the time stamps of the data records as the time stamp of the data record corresponding to the data in the database.
5. The method according to claim 4, wherein the step of respectively obtaining the corresponding time stamp of each writing of the data in the cache specifically comprises: determining a corresponding time stamp of each writing of the data according to the key value of the data in the cache;
the step of respectively obtaining the time stamp of each data record corresponding to the data in the database specifically includes: and searching each data record corresponding to the data in the database according to the key value of the data in the cache, and determining the time stamp of each data record.
6. The method of claim 5, wherein the cache is a persistent cache, wherein the persistent cache comprises: solid state disks and/or mechanical hard disks.
7. A cache-based data persistence store, comprising:
the setting module is suitable for setting corresponding timestamps for each data in the cache according to the data writing time when the data are written into the cache; the method comprises the steps that written data are first-class operations, a timestamp corresponding to the written data is second-class operations, and the first-class operations and the second-class operations are jointly used as a transaction to be executed; when the first type of operation and the second type of operation are all executed successfully, the transaction is submitted; otherwise, roll back the transaction;
the acquisition module is suitable for respectively acquiring a timestamp of each piece of data in the cache and a timestamp of a data record corresponding to the data in the database aiming at each piece of data in the cache when each piece of data in the cache is persistently stored in the database;
the processing module is suitable for deleting the data in the cache when the time stamp of the data in the cache is determined to be smaller than the time stamp of the data record corresponding to the data in the database; otherwise, the data in the cache and the corresponding time stamp are stored in the database in a persistent mode.
8. The apparatus of claim 7, wherein the apparatus further comprises: and the time storage module is suitable for storing the data in the database and the corresponding time stamp into the cache.
9. The apparatus of claim 7 or 8, wherein the apparatus further comprises: and the storage processing module is suitable for storing the data generated during the cache failure and the corresponding time stamp into the database in a persistent mode when the cache failure is detected.
10. The apparatus of claim 9, wherein the means for obtaining is further configured to:
respectively acquiring corresponding time stamps of the data in the cache when the data are written in each time, and determining the maximum value of the corresponding time stamps when the data are written in each time as the time stamp of the data in the cache;
and respectively acquiring the time stamps of the data records corresponding to the data in the database, and determining the maximum value in the time stamps of the data records as the time stamp of the data record corresponding to the data in the database.
11. The apparatus according to claim 10, wherein the obtaining module is specifically configured to: determining a corresponding time stamp of each writing of the data according to the key value of the data in the cache;
the acquisition module is specifically configured to: and searching each data record corresponding to the data in the database according to the key value of the data in the cache, and determining the time stamp of each data record.
12. The apparatus of claim 11, wherein the cache is a persistent cache, wherein the persistent cache comprises: solid state disks and/or mechanical hard disks.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611249423.2A CN106682193B (en) | 2016-12-29 | 2016-12-29 | Data persistence storage method and device based on cache |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611249423.2A CN106682193B (en) | 2016-12-29 | 2016-12-29 | Data persistence storage method and device based on cache |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106682193A CN106682193A (en) | 2017-05-17 |
| CN106682193B true CN106682193B (en) | 2021-01-08 |
Family
ID=58873390
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611249423.2A Active CN106682193B (en) | 2016-12-29 | 2016-12-29 | Data persistence storage method and device based on cache |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106682193B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107589907B (en) * | 2017-08-10 | 2019-12-13 | 深圳壹账通智能科技有限公司 | Data processing method, electronic device and computer readable storage medium |
| CN109032505A (en) * | 2018-06-26 | 2018-12-18 | 深圳忆联信息系统有限公司 | Data read-write method, device, computer equipment and storage medium with timeliness |
| CN109408496A (en) * | 2018-09-18 | 2019-03-01 | 沈文策 | A kind of method and device reducing data redundancy |
| CN109815287A (en) * | 2019-01-30 | 2019-05-28 | 网易（杭州）网络有限公司 | A kind of data base access system |
| CN109902130A (en) * | 2019-01-31 | 2019-06-18 | 北京明略软件系统有限公司 | A kind of date storage method, data query method and apparatus, storage medium |
| CN110347545A (en) * | 2019-05-21 | 2019-10-18 | 深圳壹账通智能科技有限公司 | A kind of test method and device of business platform cache policy |
| CN111708783A (en) * | 2020-06-18 | 2020-09-25 | 北京金山云网络技术有限公司 | Data storage and data recovery method and device and electronic equipment |
| CN112434066A (en) * | 2020-11-24 | 2021-03-02 | 深圳市前海迅智系统科技有限公司 | Multi-channel high-frequency real-time data aggregation method and terminal |
| CN113778723B (en) * | 2021-11-11 | 2022-02-22 | 中汽数据(天津)有限公司 | Data playback method, electronic device and readable storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101257485A (en) * | 2007-03-02 | 2008-09-03 | 华为技术有限公司 | Web applied system and method |
| CN100566234C (en) * | 2007-07-16 | 2009-12-02 | 腾讯科技（深圳）有限公司 | Method of data synchronization and processing method, client device, server |
| CN102024032A (en) * | 2010-11-29 | 2011-04-20 | 广州明朝网络科技有限公司 | Distributed data caching and persisting method and system based on Erlang |
| US8935481B2 (en) * | 2012-10-19 | 2015-01-13 | Oracle International Corporation | Apparatus system and method for providing raw data in a level-two cache |
| CN104599032A (en) * | 2014-11-28 | 2015-05-06 | 国家电网公司 | Distributed memory power grid construction method and system for resource management |
| CN105741070B (en) * | 2016-04-18 | 2019-03-12 | 深圳马可孛罗科技有限公司 | A kind of method of the quick difference update storage of the real-time freight space freight rate result data of magnanimity |
-
2016
- 2016-12-29 CN CN201611249423.2A patent/CN106682193B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN106682193A (en) | 2017-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106682193B (en) | Data persistence storage method and device based on cache | |
| US9009428B2 (en) | Data store page recovery | |
| US6629198B2 (en) | Data storage system and method employing a write-ahead hash log | |
| US8954673B1 (en) | Using a conditional read request and a hash to determine synchronization of data in a cache at a host with data in storage array | |
| CN106910528B (en) | Optimization method and device for data inspection of solid state disk | |
| US11113199B2 (en) | Low-overhead index for a flash cache | |
| CN108431783B (en) | Access request processing method and device and computer system | |
| MX2014002960A (en) | Nonvolatile media dirty region tracking. | |
| US9003228B2 (en) | Consistency of data in persistent memory | |
| US10949289B1 (en) | System and method for maintaining data integrity of data on a storage device | |
| EP3438845A1 (en) | Data updating method and device for a distributed database system | |
| JP2017079053A (en) | Methods and systems for improving storage journaling | |
| EP2336901A2 (en) | Online access to database snapshots | |
| US20140351628A1 (en) | Information processing device, control circuit, computer-readable recording medium for control program, and control method | |
| JP2016224921A (en) | Database rollback using WAL | |
| CN109753378A (en) | A kind of partition method of memory failure, device, system and readable storage medium storing program for executing | |
| EP3301577B1 (en) | Fault tolerance in shared memory | |
| CN106909514B (en) | Method and device for positioning snapshot disk address | |
| CN110928890B (en) | Data storage method and device, electronic equipment and computer readable storage medium | |
| WO2018076954A1 (en) | Data storage method, apparatus and system | |
| US20150249708A1 (en) | System and method for asynchronous replication of a storage in a computing environment | |
| CN109271277B (en) | Access method, device and system after database downtime | |
| US20110113010A1 (en) | Synchronizing an auxiliary data system with a primary data system | |
| CN117453632B (en) | Data storage method and device | |
| CN111048141B (en) | Error processing method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |