CN106682193A - Device and method for data persistent storage on basis of cache - Google Patents
Device and method for data persistent storage on basis of cache Download PDFInfo
- Publication number
- CN106682193A CN106682193A CN201611249423.2A CN201611249423A CN106682193A CN 106682193 A CN106682193 A CN 106682193A CN 201611249423 A CN201611249423 A CN 201611249423A CN 106682193 A CN106682193 A CN 106682193A
- Authority
- CN
- China
- Prior art keywords
- data
- caching
- timestamp
- base
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1032—Reliability improvement, data loss prevention, degraded operation etc
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/16—General purpose computing application
- G06F2212/163—Server or database system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a device and a method for data persistent storage on the basis of cache and at least solves the technical problem of failure in prevention of data errors in a database in cache failure due to poor disaster tolerance of an existing cache mechanism. The method includes: when data are written into cache, respectively setting corresponding timestamps for data in cache according to data writing time; when the data in cache are persistently stored into the database, respectively acquiring the timestamp of each datum in cache and a timestamp, of a data record corresponding to the datum, in the database; if the timestamps of the data in cache are smaller than those of the data records corresponding to the data, deleting the data in cache; otherwise, persistently storing the data in cache and the corresponding timestamps into the database.
Description
Technical field
The present invention relates to communication technical field, and in particular to a kind of data persistence storage method and dress based on caching
Put.
Background technology
Caching is exactly the relief area of data exchange, due to the read or write speed for caching it is very fast, therefore, can be lifted using caching
Digital independent or the efficiency of write.For example, when reading the data, first desired data is searched from caching, it is direct if finding
It is read out from caching;When the data is written, the data write caching being first written into, then again holds the data in caching
Longization is stored in data base.As can be seen here, can be offered convenience for read-write process by caching mechanism, therefore, caching mechanism
Obtain increasingly being widely applied.
But, inventor has found that at least there are the following problems for existing caching mechanism during the present invention is realized:When
When caching normal, the data of cache storage can be stored in sorted order in data base according to preset rules.But, once caching is sent out
Failure is given birth to, after fault recovery, the data of cache storage then out of order or even may be stored mistakenly in data base.Example
Such as, it is assumed that a part of data in caching temporarily break down and cannot read, therefore, the data persistence in caching is being deposited
When storing up data base, the partial data is missed;After a period of time, the partial data in caching recovers normal, then exist
During next time the data persistence in caching is stored to data base, the partial data will be stored in data base.
However, in a practical situation, the partial data is likely to have become expired wrong data, thus causes the number in data base
According to there is mistake.As can be seen here, existing caching mechanism disaster tolerance is poor, it is impossible to the number in preventing data base when failure is cached
According to error.
The content of the invention
In view of the above problems, it is proposed that the present invention so as to provide one kind overcome the problems referred to above or at least in part solve on
State the data persistence storage method based on caching and device of problem.
According to an aspect of the invention, there is provided a kind of data persistence storage method based on caching, including:Xiang Huan
Deposit it is middle write data when, according to Data writing time be caching in each data be respectively provided with corresponding timestamp;Will be described
When each data persistence storage in caching is to data base, for each data in the caching, obtain described slow respectively
Deposit the timestamp of the data and the timestamp of data record corresponding with the data in the data base;When it is determined that described slow
The timestamp for depositing the data is described slow less than during the timestamp of data record corresponding with the data, deleting in the data base
The data in depositing;Otherwise, by the data in the caching and its corresponding timestamp persistent storage to the data base
In.
According to another aspect of the present invention, there is provided it is a kind of based on caching data persistence storage device, including:If
Put module, be suitable to in caching write data when, according to Data writing time be caching in each data be respectively provided with correspondence
Timestamp;Acquisition module, when being suitable to for each data persistence storage in the caching to arrive data base, for the caching
In each data, number corresponding with the data in the timestamp of the data and the data base is obtained in the caching respectively
According to the timestamp of record;Processing module, in being suitable to the timestamp of the data in determine the caching less than the data base with
During the timestamp of the corresponding data record of the data, the data in the caching are deleted;Otherwise, by the number in the caching
According to and its corresponding timestamp persistent storage in the data base.
In a kind of data persistence storage method and device based on caching that the present invention is provided, write in caching first
When entering data, according to Data writing time be caching in each data be respectively provided with corresponding timestamp;Then in caching
The storage of each data persistence to data base when, for each data in caching, obtain respectively the data in caching when
Between stamp and data base in data record corresponding with the data timestamp;Finally, when it is determined that caching in the data time
Stamp is less than the data in data base during the timestamp of data record corresponding with the data, in deletion caching;Otherwise, will cache
In the data and its corresponding timestamp persistent storage in data base.As can be seen here, the present invention is each in caching
Data are respectively provided with corresponding timestamp, correspondingly, when by certain the data persistence storage in caching to data base, pass through
The mode that the timestamp of the data in caching is compared with the timestamp of data record corresponding with the data in data base,
Judge cache in data whether be invalid data out of date, so as to improve data persistence during reliability, have
Effect is prevented because caching the caused error in data phenomenon of failure institute.
Described above is only the general introduction of technical solution of the present invention, in order to better understand the technological means of the present invention,
And can be practiced according to the content of description, and in order to allow the above and other objects of the present invention, feature and advantage can
Become apparent, below especially exemplified by the specific embodiment of the present invention.
Description of the drawings
By the detailed description for reading hereafter preferred implementation, various other advantages and benefit is common for this area
Technical staff will be clear from understanding.Accompanying drawing is only used for illustrating the purpose of preferred implementation, and is not considered as to the present invention
Restriction.And in whole accompanying drawing, it is denoted by the same reference numerals identical part.In the accompanying drawings:
Fig. 1 shows that a kind of flow process of data persistence storage method based on caching that the embodiment of the present invention one is provided is shown
It is intended to;
Fig. 2 shows that a kind of flow process of data persistence storage method based on caching that the embodiment of the present invention two is provided is shown
It is intended to;
Fig. 3 shows a kind of structural frames of data persistence storage device based on caching that the embodiment of the present invention three is provided
Figure.
Specific embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing the disclosure in accompanying drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure and should not be by embodiments set forth here
Limited.On the contrary, there is provided these embodiments are able to be best understood from the disclosure, and can be by the scope of the present disclosure
Complete conveys to those skilled in the art.
A kind of data persistence storage method based on caching and device are embodiments provided, at least be can solve the problem that
Existing caching mechanism disaster tolerance is poor, it is impossible to the technical problem of the corrupt data in data base is prevented when failure is cached.
Embodiment one
Fig. 1 shows that a kind of flow process of data persistence storage method based on caching that the embodiment of the present invention one is provided is shown
It is intended to, the method includes:
Step S110:To in caching write data when, according to Data writing time be caching in each data set respectively
Put corresponding timestamp.
Wherein, timestamp is used for time when representing that data write is cached.The concrete form of timestamp and precision can be by these
Art personnel flexibly arranged, and the present invention is not limited this.For example, it is assumed that a data are in the morning 10 on December 13rd, 2016
01 minute write caching of point, then the corresponding timestamp of the data is 201612131001.In addition, in the buffer data storage when, can
So that data and its corresponding timestamp are associated into storage as a data set, it is also possible to using timestamp as data itself
An attribute stored, the present invention concrete storage form is not limited.
Step S120:During by each data persistence storage in caching to data base, for each data in caching,
The timestamp of the timestamp of the data and data record corresponding with the data in data base in caching is obtained respectively.
Wherein, each data persistence storage to the opportunity in data base can flexibly be arranged by those skilled in the art,
The present invention is not limited this.In persistent storage, first, the timestamp of the data in caching is obtained;Then, counting
According to lookup data record corresponding with the data in storehouse, and obtain the timestamp of data record corresponding with the data.Wherein, with
The corresponding data record of the data can be stored by various forms.
Step S130:When it is determined that the timestamp of the data is less than data record corresponding with the data in data base in caching
Timestamp when, delete caching in the data;Otherwise, by the data in caching and its corresponding timestamp persistent storage
To in data base.
Specifically, timestamp is used to represent write time of data, can be indicated by numeric form.In the present invention
In, the priority of time is represented by the size of timestamp:The timestamp of time more forward data is less, and the time is more rearward
Data timestamp it is bigger.As can be seen here, the size for comparing timestamp is actually the priority of determination time.For example, it is false
If the write time of data one is 01 minute morning 10 point of on December 13rd, 2016, timestamp is 201612131001;Data two
Write time be 01 minute morning 10 point of on December 14th, 2016, timestamp is 201612141001, due to data one write when
Between earlier than data two write time, then the timestamp of data one less than data two timestamp.
Correspondingly, when the timestamp of the data in caching is less than the time of data record corresponding with the data in data base
During stamp, illustrate cache in the data write time earlier than corresponding data record in data base write time, i.e.,:Caching
In the data be invalid data out of date, thus delete caching in the data, to prevent error in data phenomenon.When
During the timestamp of the data in data base not less than during the timestamp of data record corresponding with the data, illustrating to cache in caching
Write time of the data be later than write time of corresponding data record in data base, i.e.,:The data in caching are not for
Expired valid data, thus by the data in caching and its corresponding timestamp persistent storage in data base.
As can be seen here, in a kind of data persistence storage method based on caching that the present invention is provided, first to caching
It is middle write data when, according to Data writing time be caching in each data be respectively provided with corresponding timestamp;Then will be slow
When each data persistence storage in depositing is to data base, for each data in caching, the data in caching are obtained respectively
Timestamp and data base in data record corresponding with the data timestamp;Finally, when it is determined that the data in caching
Timestamp is less than the data in data base during the timestamp of data record corresponding with the data, in deletion caching;Otherwise, will
Data and its corresponding timestamp persistent storage in caching are in data base.Therefore, the side in the present invention in embodiment
Method is that each data in caching are respectively provided with corresponding timestamp, correspondingly, certain data persistence in caching is being deposited
When storing up data base, by by the time of the timestamp of the data in caching and data record corresponding with the data in data base
The mode that stamp is compared, judges whether the data in caching are invalid data out of date, so as to improve data persistence
During reliability, error in data phenomenon caused by effectively prevent because caching failure institute.
Embodiment two
Fig. 2 shows that a kind of flow process of data persistence storage method based on caching that the embodiment of the present invention two is provided is shown
It is intended to, as shown in Fig. 2 the method includes:
Step S210:To in caching write data when, according to Data writing time be caching in each data set respectively
Put corresponding timestamp.
Wherein, timestamp is used for time when representing that data write is cached.The concrete form of timestamp and precision can be by these
Art personnel flexibly arranged, and the present invention is not limited this.For example, it is assumed that a data are in the morning 10 on December 13rd, 2016
01 minute write caching of point, then the corresponding timestamp of the data is 201612131001.
In addition, when to data storage in caching, various storage modes can be adopted.In the first storage mode, can
So that data and its corresponding timestamp are associated into storage as a data set.In addition, in associated storage, actually relating to
And two generic operations, the first generic operation is write data, and the second generic operation is the corresponding timestamp of write data, in order to prevent because of it
In a generic operation run succeeded, another generic operation performs a mistake caused by failure institute, can be by the first generic operation and Equations of The Second Kind
Operation is performed collectively as an affairs, and only when the first generic operation and the second generic operation all run succeeded, just submitting to should
Affairs, otherwise, then rollback affairs, to re-execute data write operation, so as to ensure that caching in each data and at that time
Between stab accuracy.In second storage mode, it is also possible to stored timestamp as an attribute of data itself,
Can also be stored timestamp as the suffix or prefix of data.The present invention is not limited specific storage form.
In addition, in the present embodiment, various types of write operations such as the write of data includes writing direct and change.
Step S220:During by each data persistence storage in caching to data base, for each data in caching,
The timestamp of the timestamp of the data and data record corresponding with the data in data base in caching is obtained respectively.
In the present embodiment, can carry out when default persistence condition meets above-mentioned by each data in caching
To the operation of data base, specific persistence condition can flexibly set according to demand persistent storage by those skilled in the art.
For example, in a kind of implementation of the present embodiment, the persistence condition is:In caching written into data reach default ratio
Example.And for example, in another kind of implementation, the persistence condition is:When reaching the predetermined persistence time, or distance is last
When the time of persistence reaches prefixed time interval.Specific persistence time and prefixed time interval can be by art technologies
Personnel flexibly set.
Specifically, in persistent storage, first, the timestamp of the data that persistence is treated in caching is obtained;Then, exist
Data record corresponding with the data is searched in data base, and obtains the timestamp of data record corresponding with the data.In addition,
Because same data often successively perform in the buffer multiple write operations, therefore, in this step, can respectively obtain slow
Corresponding timestamp during each write of the data in depositing, by the maximum (timestamp in corresponding timestamp during each write
The maximum explanation write time is the latest) it is defined as the timestamp of the data in caching;Obtain respectively corresponding with the data in data base
Pieces of data record timestamp, by pieces of data record timestamp in maximum be defined as in data base and the data
The timestamp of corresponding data record.As can be seen here, can determine the data slow by determining the maximum data of timestamp
Time when being ultimately written in depositing, by determining that the maximum data record of timestamp can determine corresponding data Ji Lu data
Time when being ultimately written in storehouse, and then can rapidly and accurately judge whether the data in caching are effective.Further, since data
Generally unique mark is carried out by key assignments, therefore, key assignments identical data are considered as same data, correspondingly, above-mentioned
During, corresponding timestamp when the key assignments of the data in caching determines each write of the data;According in caching
Pieces of data record corresponding with the data in the key assignments searching data storehouse of the data, and determine the time that pieces of data is recorded
Stamp.
Step S230:Whether the timestamp of the data is less than data note corresponding with the data in data base in judging to cache
The timestamp of record, if the determination result is YES, execution step S240;If judged result is no, execution step S250.
Specifically, timestamp is used to represent write time of data, can be indicated by numeric form.In the present invention
In, the priority of time is represented by the size of timestamp:The timestamp of time more forward data is less, and the time is more rearward
Data timestamp it is bigger.As can be seen here, the size for comparing timestamp is actually the priority of determination time.For example, it is false
If the write time of data one is 01 minute morning 10 point of on December 13rd, 2016, timestamp is 201612131001;Data two
Write time be 01 minute morning 10 point of on December 14th, 2016, timestamp is 201612141001, due to data one write when
Between earlier than data two write time, then the timestamp of data one less than data two timestamp.
Step S240:Delete the data in caching.
Specifically, when the timestamp of the data in caching is less than the time of data record corresponding with the data in data base
During stamp, illustrate cache in the data write time earlier than corresponding data record in data base write time, i.e.,:Caching
In the data be invalid data out of date, thus delete caching in the data, to prevent error in data phenomenon.
In addition, in this step, when it is determined that the timestamp of the data is corresponding with the data less than in data base in caching
During the timestamp of data record, following operation is can further include:By the data in data base and its corresponding timestamp
Preserve into caching.Many advantages are waited soon because caching possesses access speed, therefore, the data in judging to cache are for
During expired invalid data, further the data in data base and its corresponding timestamp are preserved into caching, to realize carrying
Rise the effect of access speed.In addition, can be with by the way that the data in data base and its corresponding timestamp are preserved into caching
The data lost during caching failure are enable to recover.Specifically, the data in data base actually refer in data base with
The corresponding data record of the data.In a practical situation, in the storage form of the data in caching and data base with the data pair
The storage form of the data record answered both can be the same or different.For example, the data in caching can pass through Word document
Stored, data record corresponding with the data can be stored by Excel forms in data base.
Step S250:By the data in caching and its corresponding timestamp persistent storage in data base.
Specifically, when the data in caching timestamp not less than in data base data record corresponding with the data when
Between when stabbing, the write time for illustrating the data in caching is later than the write time of corresponding data record in data base, i.e.,:It is slow
The data in depositing are not out of date valid data, thus by the data in caching and its corresponding timestamp persistent storage
To in data base.By the judgement of timestamp, the correctness of the data stored in data base can be lifted.
In addition, the disaster tolerance performance in order to further lift caching system, in the execution of above-mentioned steps S210 to step S250
During, can further include following steps:When detect caching failure when, by caching failure during produce data and
Its corresponding timestamp persistent storage is in data base.The present invention was not limited the concrete execution opportunity of the step.For example,
This step can be put into after step S240 and be performed, so as to trigger this when there is wrong data out of date in finding caching
Step;Whether failure can also be once cached every default time interval automatic detection, and be triggered when testing result is to be
This step.By this step, can be by the data produced during caching failure and its direct persistent storage of corresponding timestamp
To in data base, so as to prevent the loss of data, the reliability service for whole caching system provides guarantee.In addition, here
On the basis of, due to when performing persistent storage after caching fault recovery and operating, can further compare each it is data cached when
Between stab, it is thus possible to judge in data base through caching and direct write time of the partial data of persistence, and
When the timestamp of the data in determining caching is less than the timestamp of data record corresponding with the data in data base, will
The data and its corresponding timestamp are preserved into caching in data base so that the data during caching failure be able to it is completely extensive
It is multiple, improve the disaster tolerance performance of caching system.
In addition, those skilled in the art can also carry out various changes and deformation to above-described embodiment.For example, can will be upper
State each step in embodiment and split into more steps, or merge into less step, can also adjust each step it
Between execution sequence, it might even be possible to delete part steps or increase newly some steps, in a word, the present invention to concrete implementation details not
Limit.
In addition, inventor has found during the present invention is realized, common caching is mostly by the internal memory of non-persistentization
Or other storage devices are realized, the characteristic that non-persistentization caching falls with power-off, i.e.,:After power-off or failure, store thereon
Data will be emptied therewith, so as to cause data cached loss.And, non-persistentization caching often has the offices such as capacity is less than normal
It is sex-limited.In order to solve the above problems, in the present embodiment, it is possible to use persistence caching is realized.For example, persistence caching includes
But it is not limited to:SSD, solid state hard disc and/or mechanical hard disk etc..These persistences caching be often provided simultaneously with read or write speed it is fast and
The big advantage of memory capacity.And, it is that each data are provided with timestamp in the present embodiment, therefore, even if persistence caching
There are the abnormal conditions such as power-off or failure, the data for storing thereon still retain, and can sentence according to the timestamp of each data
Whether each data of breaking are still effective, so as to improve the performance of caching system.
Embodiment three
Fig. 3 shows a kind of structural frames of data persistence storage device based on caching that the embodiment of the present invention three is provided
Figure.As shown in figure 3, the device includes:Setup module 31, acquisition module 32, processing module 33, time memory module 34 and deposit
Storage processing module 35.
It is each data point in caching according to Data writing time when setup module 31 is suitable to write data in caching
Corresponding timestamp is not set.
Wherein, timestamp is used for time when representing that data write is cached.The concrete form of timestamp and precision can be by these
Art personnel flexibly arranged, and the present invention is not limited this.For example, it is assumed that a data are in the morning 10 on December 13rd, 2016
01 minute write caching of point, then the corresponding timestamp of the data is 201612131001.In addition, setup module 31 writes according to data
When time is that each data in caching are respectively provided with corresponding timestamp, various storage modes can be set.Deposit at the first
In storage mode, data and its corresponding timestamp can be associated storage by setup module 31 as a data set.In addition,
In associated storage, two generic operations are actually related to, the first generic operation is write data, and the second generic operation is write data correspondence
Timestamp, in order to prevent because a generic operation therein runs succeeded, another generic operation performs a mistake caused by failure institute, can be with
First generic operation and the second generic operation are performed collectively as an affairs, only when the first generic operation and the second generic operation whole
When running succeeded, the affairs are just submitted to, otherwise, then the rollback affairs, slow so as to ensure that to re-execute data write operation
The accuracy of each data and its timestamp in depositing.In second storage mode, setup module 31 can also make timestamp
An attribute for data itself is stored, and can also be stored timestamp as the suffix or prefix of data.This
It is bright that specific storage form is not limited.In addition, in the present embodiment, the write of data includes writing direct and changing
Various types of write operations.
When acquisition module 32 is suitable to for each data persistence storage in caching to arrive data base, for each in caching
Data, obtain respectively the timestamp of the timestamp of the data and data record corresponding with the data in data base in caching.
In the present embodiment, acquisition module 32 can carry out when default persistence condition meets it is above-mentioned will be in caching
The storage of each data persistence to data base operation, specific persistence condition can be by those skilled in the art according to demand
Flexibly setting.For example, in a kind of implementation of the present embodiment, the persistence condition is:In caching written into data reach
To preset ratio.And for example, in another kind of implementation, the persistence condition is:When reaching the predetermined persistence time, or away from
When reaching prefixed time interval from the time of last persistence.Specific persistence time and prefixed time interval can be by these
Art personnel flexibly set.
Specifically, in persistent storage, first, acquisition module 32 will obtain the time of the data that persistence is treated in caching
Stamp;Then, search corresponding with data data record in data base, and acquisition data record corresponding with the data when
Between stab.Further, since same data often successively perform in the buffer multiple write operations, therefore, acquisition module 32 can be with
Corresponding timestamp when obtaining each write of the data in caching respectively, by the maximum in corresponding timestamp during each write
Value (the timestamp maximum explanation write time is the latest) is defined as the timestamp of the data in caching;And, data base is obtained respectively
In corresponding with data pieces of data record timestamp, the maximum in the timestamp that pieces of data is recorded is defined as number
According to the timestamp of data record corresponding with the data in storehouse.As can be seen here, timestamp maximum is determined by acquisition module 32
Time when data can determine that the data is ultimately written in the buffer, the maximum number of timestamp is determined by acquisition module 32
Time when being ultimately written in corresponding data Ji Lu data base is can determine according to record, and then can rapidly and accurately be judged
Whether the data in caching are effective.Further, since data generally carry out unique mark by key assignments, therefore, key assignments identical number
According to being considered as same data, correspondingly, the correspondence when key assignments of the data in caching determines each write of the data
Timestamp;Pieces of data record corresponding with the data in the key assignments searching data storehouse of the data in caching, and really
Determine the timestamp of pieces of data record.
Processing module 33 is suitable to the timestamp of the data in caching is determined less than number corresponding with the data in data base
During according to the timestamp for recording, the data in caching are deleted;Otherwise, it is the data and its corresponding timestamp in caching are lasting
Change and store in data base.
Specifically, timestamp is used to represent write time of data, can be indicated by numeric form.In the present invention
In, the priority of time is represented by the size of timestamp:The timestamp of time more forward data is less, and the time is more rearward
Data timestamp it is bigger.Therefore, the size that processing module 33 compares timestamp is actually the priority of determination time.Example
Such as, it is assumed that the write time of data one is 01 minute morning 10 point of on December 13rd, 2016, and timestamp is 201612131001;Data
Two write time is 01 minute morning 10 point of on December 14th, 2016, and timestamp is 201612141001, due to writing for data one
The angle of incidence earlier than data two write time, then the timestamp of data one less than data two timestamp.When processing module 33 it is true
During the timestamp of the data in data base less than during the timestamp of data record corresponding with the data, illustrating to cache in fixed caching
The data write time earlier than corresponding data record in data base write time, i.e.,:The data in caching are for
Expired invalid data, thus the data in caching are deleted, to prevent error in data phenomenon.When the data in caching
Timestamp is not less than the write that the data in caching during the timestamp of data record corresponding with the data, are illustrated in data base
Time is later than the write time of corresponding data record in data base, i.e.,:The data in caching are not out of date valid data,
Thus processing module 33 by the data in caching and its corresponding timestamp persistent storage in data base.By timestamp
Judgement, the correctness of the data stored in data base can be lifted.
Time memory module 34 is suitable to the data in data base and its corresponding timestamp be preserved into caching.
When processing module 33 determines the timestamp of the data in caching less than data note corresponding with the data in data base
During the timestamp of record, further the data in data base and its corresponding timestamp can also be protected by time memory module 34
Deposit into caching.Many advantages are waited soon because caching possesses access speed, therefore, being somebody's turn to do in judging to cache of processing module 33
When data are invalid data out of date, time memory module 34 is further by the data in data base and its corresponding timestamp
Preserve into caching, to realize lifting the effect of access speed.In addition, time memory module 34 is by by the data in data base
And its corresponding timestamp preserves into caching and the data lost during caching failure can also be enable to recover.Specifically, number
Actually refer to data record corresponding with the data in data base according to the data in storehouse.In a practical situation, in caching
The storage form of data both can be the same or different with the storage form of data record corresponding with the data in data base.
For example, the data in caching can be stored by Word document, and data record corresponding with the data can be with data base
Stored by Excel forms.
Storage processing module 35 is suitable to when caching failure is detected, by the data produced during caching failure and its correspondence
Timestamp persistent storage in data base.
Specifically, storing processing module 35 is used for when caching failure is detected, the data that will be produced during caching failure
And its corresponding timestamp persistent storage is in data base, and then the purpose of the disaster tolerance performance for realizing lifting caching system.
Here, the opportunity that performs for storing processing module 35 can exist after the data in above-mentioned caching is deleted in caching is found
Triggering storage processing module 35 performs corresponding persistent storage operation during wrong data out of date;Or can also be every
Default time interval automatic detection once caches whether failure, and triggering storage processing module 35 is held when testing result is to be
The corresponding persistent storage operation of row, the present invention was not limited the concrete execution opportunity for storing processing module 35.Therefore, pass through
Storage processing module 35 can be by the data produced during caching failure and its direct persistent storage of corresponding timestamp to number
According to storehouse, so as to prevent the loss of data, the reliability service for whole caching system provides guarantee.
The concrete structure and operation principle of above-mentioned modules can refer to the description of corresponding steps in embodiment of the method, herein
Repeat no more.
As can be seen here, in a kind of data persistence storage device based on caching that the present invention is provided, first by setting
When putting module 31 and writing data in caching, when according to Data writing time being that each data in caching are respectively provided with corresponding
Between stab;Then by acquisition module 32 by each data persistence storage in caching to data base when, for caching in it is every
Individual data, obtain respectively the time of the timestamp of the data and data record corresponding with the data in data base in caching
Stamp;And by the timestamp of the data in caching is determined of processing module 33 less than data note corresponding with the data in data base
During the timestamp of record, the data in caching are deleted;Otherwise, the data in caching and its corresponding timestamp persistence are deposited
In storing up data base.Meanwhile, further the data in data base and its corresponding timestamp are protected by time memory module 34
Deposit into caching, by store processing module 35 when detect caching failure when, by caching failure during produce data and its
Corresponding timestamp persistent storage is in data base.Therefore, the method in the present invention in embodiment is each number in caching
According to corresponding timestamp is respectively provided with, correspondingly, when by certain data persistence storage in caching to data base, by inciting somebody to action
The mode that the timestamp of the data is compared with the timestamp of data record corresponding with the data in data base in caching, sentences
Whether the data in disconnected caching are invalid data out of date, so as to improve data persistence during reliability, effectively
Error in data phenomenon caused by preventing because caching failure institute, even and if the exception feelings such as power-off or failure occurs in persistence caching
Condition, the data for storing thereon still retain, and whether still effectively can judge each data according to the timestamp of each data,
So as to improve the performance of caching system.
Provided herein algorithm and display be not inherently related to any certain computer, virtual system or miscellaneous equipment.
Various general-purpose systems can also be used together based on teaching in this.As described above, construct required by this kind of system
Structure be obvious.Additionally, the present invention is also not for any certain programmed language.It is understood that, it is possible to use it is various
Programming language realizes the content of invention described herein, and the description done to language-specific above is to disclose this
Bright preferred forms.
In description mentioned herein, a large amount of details are illustrated.It is to be appreciated, however, that the enforcement of the present invention
Example can be put into practice in the case of without these details.In some instances, known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this description.
Similarly, it will be appreciated that in order to simplify the disclosure and help understand one or more in each inventive aspect, exist
Above in the description of the exemplary embodiment of the present invention, each feature of the present invention is grouped together into single enforcement sometimes
In example, figure or descriptions thereof.However, the method for the disclosure should be construed to reflect following intention:I.e. required guarantor
The more features of feature that the application claims ratio of shield is expressly recited in each claim.More precisely, such as following
Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim itself
All as the separate embodiments of the present invention.
Those skilled in the art are appreciated that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more equipment different from the embodiment.Can be the module or list in embodiment
Unit or component are combined into a module or unit or component, and can be divided in addition multiple submodule or subelement or
Sub-component.In addition at least some in such feature and/or process or unit is excluded each other, can adopt any
Combine to all features disclosed in this specification (including adjoint claim, summary and accompanying drawing) and so disclosed
Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification is (including adjoint power
Profit is required, summary and accompanying drawing) disclosed in each feature can it is identical by offers, be equal to or the alternative features of similar purpose carry out generation
Replace.
Although additionally, it will be appreciated by those of skill in the art that some embodiments described herein include other embodiments
In included some features rather than further feature, but the combination of the feature of different embodiments means in of the invention
Within the scope of and form different embodiments.For example, in the following claims, embodiment required for protection appoint
One of meaning can in any combination mode using.
The present invention all parts embodiment can be realized with hardware, or with one or more processor operation
Software module realize, or with combinations thereof realization.It will be understood by those of skill in the art that can use in practice
Microprocessor or digital signal processor (DSP) are come in the consumption device for realizing message queue according to embodiments of the present invention
The some or all functions of some or all parts.The present invention is also implemented as performing method as described herein
Some or all equipment or program of device (for example, computer program and computer program).Such reality
The program of the existing present invention can be stored on a computer-readable medium, or can have the form of one or more signal.
Such signal can be downloaded from internet website and obtained, or be provided on carrier signal, or in any other form
There is provided.
It should be noted that above-described embodiment the present invention will be described rather than limits the invention, and ability
Field technique personnel can design without departing from the scope of the appended claims alternative embodiment.In the claims,
Any reference markss between bracket should not be configured to limitations on claims.Word "comprising" is not excluded the presence of not
Element listed in the claims or step.Word "a" or "an" before element does not exclude the presence of multiple such
Element.The present invention can come real by means of the hardware for including some different elements and by means of properly programmed computer
It is existing.If in the unit claim for listing equipment for drying, several in these devices can be by same hardware branch
To embody.The use of word first, second, and third does not indicate that any order.These words can be explained and be run after fame
Claim.
The invention discloses:A1, a kind of data persistence storage method based on caching, including:
To in caching write data when, according to Data writing time be caching in each data be respectively provided with corresponding when
Between stab;
During by each data persistence storage in the caching to data base, for each data in the caching,
The time of data record corresponding with the data in the timestamp of the data in the caching and the data base is obtained respectively
Stamp;
When it is determined that the timestamp of the data is less than data record corresponding with the data in the data base in the caching
Timestamp when, delete the data in the caching;Otherwise, the data and its corresponding timestamp in the caching are held
Longization is stored in the data base.
A2, the method according to A1, wherein, when it is determined that the timestamp of the data is less than the data base in the caching
In data record corresponding with the data timestamp when, methods described is further included:By data in data base and its right
The timestamp answered is preserved into the caching.
A3, the method according to A1 or A2, wherein, further include:When the caching failure is detected, will be described
The data produced during caching failure and its corresponding timestamp persistent storage are in the data base.
A4, according to the arbitrary described methods of A1-A3, wherein, the timestamp that the data in the caching are obtained respectively
And specifically include the step of the timestamp of data record corresponding with the data in the data base:
Corresponding timestamp when obtaining each write of the data in the caching respectively, when will be corresponding during each write
Between stab in maximum be defined as the timestamp of the data in the caching;
The timestamp of pieces of data record corresponding with the data in the data base is obtained respectively, and pieces of data is recorded
Timestamp in maximum be defined as the timestamp of data record corresponding with the data in the data base.
A5, the method according to A4, wherein, correspondence during each write of described data obtained respectively in the caching
Timestamp the step of specifically include:The key assignments of the data in the caching determines corresponding during each write of the data
Timestamp;
It is described concrete the step of obtain the timestamp of corresponding with data pieces of data record in the data base respectively
Including:The key assignments of the data in the caching searches pieces of data record corresponding with the data in the data base,
And determine the timestamp of pieces of data record.
A6, the method according to A1-A5 is arbitrary, wherein, the caching is persistence caching, wherein,
The persistence caching includes:Solid state hard disc and/or mechanical hard disk.
The invention also discloses:B7, a kind of data persistence storage device based on caching, including:
Setup module, is each data point in caching according to Data writing time when being suitable to write data in caching
Corresponding timestamp is not set;
Acquisition module, when being suitable to for each data persistence storage in the caching to arrive data base, for the caching
In each data, number corresponding with the data in the timestamp of the data and the data base is obtained in the caching respectively
According to the timestamp of record;
Processing module, is suitable in the timestamp of the data in the determination caching is less than the data base and the data pair
During the timestamp of the data record answered, the data in the caching are deleted;Otherwise, by data in the caching and its right
The timestamp persistent storage answered is in the data base.
B8, the device according to B7, wherein, described device is further included:Time memory module, is suitable to data base
In the data and its corresponding timestamp preserve into the caching.
B9, the device according to B7 or B8, wherein, described device is further included:Storage processing module, is suitable to when inspection
When measuring the caching failure, by the data produced during the caching failure and its corresponding timestamp persistent storage to institute
In stating data base.
B10, according to the arbitrary described devices of B7-B9, wherein, the acquisition module is further used for:
Corresponding timestamp when obtaining each write of the data in the caching respectively, when will be corresponding during each write
Between stab in maximum be defined as the timestamp of the data in the caching;
The timestamp of pieces of data record corresponding with the data in the data base is obtained respectively, and pieces of data is recorded
Timestamp in maximum be defined as the timestamp of data record corresponding with the data in the data base.
B11, the device according to B10, wherein, the acquisition module specifically for:The number in the caching
According to key assignments determine the write of each time of the data when corresponding timestamp;
The acquisition module specifically for:The key assignments of the data in the caching is searched in the data base and is somebody's turn to do
The corresponding pieces of data record of data, and determine the timestamp of the pieces of data record.
B12, the device according to B7-B11 is arbitrary, wherein, the caching is persistence caching, wherein, it is described lasting
Changing caching includes:Solid state hard disc and/or mechanical hard disk.
Claims (10)
1. it is a kind of based on caching data persistence storage method, including:
To in caching write data when, according to Data writing time be caching in each data be respectively provided with the corresponding time
Stamp;
During by each data persistence storage in the caching to data base, for each data in the caching, difference
Obtain the timestamp of data record corresponding with the data in the timestamp of the data in the caching and the data base;
When it is determined that in the caching data timestamp less than in the data base data record corresponding with the data when
Between when stabbing, delete the data in the caching;Otherwise, by the data in the caching and its corresponding timestamp persistence
In storing the data base.
2. method according to claim 1, wherein, when it is determined that the timestamp of the data is less than the data in the caching
In storehouse during the timestamp of data record corresponding with the data, methods described is further included:By the data in data base and its
Corresponding timestamp is preserved into the caching.
3. method according to claim 1 and 2, wherein, further include:When the caching failure is detected, by institute
The data and its corresponding timestamp persistent storage of generation are in the data base during stating caching failure.
4. according to the arbitrary described method of claim 1-3, wherein, the timestamp that the data in the caching are obtained respectively
And specifically include the step of the timestamp of data record corresponding with the data in the data base:
Corresponding timestamp when obtaining each write of the data in the caching respectively, by corresponding timestamp during each write
In maximum be defined as the timestamp of the data in the caching;
Obtain the timestamp of corresponding with data pieces of data record in the data base respectively, by pieces of data record when
Between stab in maximum be defined as the timestamp of data record corresponding with the data in the data base.
5. method according to claim 4, wherein, during each write of described data obtained respectively in the caching pair
The step of timestamp answered, specifically includes:The correspondence when key assignments of the data in the caching determines each write of the data
Timestamp;
It is described to specifically include the step of obtain the timestamp of corresponding with data pieces of data record in the data base respectively:
The key assignments of the data in the caching searches pieces of data record corresponding with the data in the data base, and determines
The timestamp of the pieces of data record.
6. according to the arbitrary described method of claim 1-5, wherein, the caching is persistence caching,
Wherein, the persistence caching includes:Solid state hard disc and/or mechanical hard disk.
7. it is a kind of based on caching data persistence storage device, including:
Setup module, be suitable to in caching write data when, according to Data writing time be caching in each data set respectively
Put corresponding timestamp;
Acquisition module, when being suitable to for each data persistence storage in the caching to arrive data base, in the caching
Each data, obtains respectively in the caching data note corresponding with the data in the timestamp of the data and the data base
The timestamp of record;
Processing module, the timestamp for being suitable to the data in the caching is determined is corresponding with the data less than in the data base
During the timestamp of data record, the data in the caching are deleted;Otherwise, by data in the caching and its corresponding
Timestamp persistent storage is in the data base.
8. device according to claim 7, wherein, described device is further included:Time memory module, is suitable to data
The data and its corresponding timestamp are preserved into the caching in storehouse.
9. the device according to claim 7 or 8, wherein, described device is further included:Storage processing module, is suitable to work as
When detecting the caching failure, the data produced during the caching failure and its corresponding timestamp persistent storage are arrived
In the data base.
10. according to the arbitrary described device of claim 7-9, wherein, the acquisition module is further used for:
Corresponding timestamp when obtaining each write of the data in the caching respectively, by corresponding timestamp during each write
In maximum be defined as the timestamp of the data in the caching;
Obtain the timestamp of corresponding with data pieces of data record in the data base respectively, by pieces of data record when
Between stab in maximum be defined as the timestamp of data record corresponding with the data in the data base.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201611249423.2A CN106682193B (en) | 2016-12-29 | 2016-12-29 | Data persistence storage method and device based on cache |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201611249423.2A CN106682193B (en) | 2016-12-29 | 2016-12-29 | Data persistence storage method and device based on cache |
Publications (2)
Publication Number | Publication Date |
---|---|
CN106682193A true CN106682193A (en) | 2017-05-17 |
CN106682193B CN106682193B (en) | 2021-01-08 |
Family
ID=58873390
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201611249423.2A Active CN106682193B (en) | 2016-12-29 | 2016-12-29 | Data persistence storage method and device based on cache |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106682193B (en) |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107589907A (en) * | 2017-08-10 | 2018-01-16 | 上海壹账通金融科技有限公司 | Data processing method, electronic equipment and computer-readable recording medium |
CN109032505A (en) * | 2018-06-26 | 2018-12-18 | 深圳忆联信息系统有限公司 | Data read-write method, device, computer equipment and storage medium with timeliness |
CN109408496A (en) * | 2018-09-18 | 2019-03-01 | 沈文策 | A kind of method and device reducing data redundancy |
CN109815287A (en) * | 2019-01-30 | 2019-05-28 | 网易(杭州)网络有限公司 | A kind of data base access system |
CN109902130A (en) * | 2019-01-31 | 2019-06-18 | 北京明略软件系统有限公司 | A kind of date storage method, data query method and apparatus, storage medium |
CN110347545A (en) * | 2019-05-21 | 2019-10-18 | 深圳壹账通智能科技有限公司 | A kind of test method and device of business platform cache policy |
CN111708783A (en) * | 2020-06-18 | 2020-09-25 | 北京金山云网络技术有限公司 | Data storage and data recovery method and device and electronic equipment |
CN112434066A (en) * | 2020-11-24 | 2021-03-02 | 深圳市前海迅智系统科技有限公司 | Multi-channel high-frequency real-time data aggregation method and terminal |
CN113778723A (en) * | 2021-11-11 | 2021-12-10 | 中汽数据(天津)有限公司 | Data playback method, electronic device and readable storage medium |
CN114490775A (en) * | 2022-02-11 | 2022-05-13 | 广东美房智高机器人有限公司 | Robot real-time operation data processing method, device, server and medium |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101075864A (en) * | 2007-07-16 | 2007-11-21 | 腾讯科技(深圳)有限公司 | Method for synchronizing and processing data, customer terminal equipment and servo |
CN101257485A (en) * | 2007-03-02 | 2008-09-03 | 华为技术有限公司 | Web applied system and method |
CN102024032A (en) * | 2010-11-29 | 2011-04-20 | 广州明朝网络科技有限公司 | Distributed data caching and persisting method and system based on Erlang |
US20140115245A1 (en) * | 2012-10-19 | 2014-04-24 | Oracle International Corporation | Apparatus system and method for providing raw data in a level-two cache |
CN104599032A (en) * | 2014-11-28 | 2015-05-06 | 国家电网公司 | Distributed memory power grid construction method and system for resource management |
CN105741070A (en) * | 2016-04-18 | 2016-07-06 | 深圳马可孛罗科技有限公司 | Rapid differentiation update and storage method for mass real-time shipping space and freight rate result data |
-
2016
- 2016-12-29 CN CN201611249423.2A patent/CN106682193B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101257485A (en) * | 2007-03-02 | 2008-09-03 | 华为技术有限公司 | Web applied system and method |
CN101075864A (en) * | 2007-07-16 | 2007-11-21 | 腾讯科技(深圳)有限公司 | Method for synchronizing and processing data, customer terminal equipment and servo |
CN102024032A (en) * | 2010-11-29 | 2011-04-20 | 广州明朝网络科技有限公司 | Distributed data caching and persisting method and system based on Erlang |
US20140115245A1 (en) * | 2012-10-19 | 2014-04-24 | Oracle International Corporation | Apparatus system and method for providing raw data in a level-two cache |
CN104599032A (en) * | 2014-11-28 | 2015-05-06 | 国家电网公司 | Distributed memory power grid construction method and system for resource management |
CN105741070A (en) * | 2016-04-18 | 2016-07-06 | 深圳马可孛罗科技有限公司 | Rapid differentiation update and storage method for mass real-time shipping space and freight rate result data |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107589907A (en) * | 2017-08-10 | 2018-01-16 | 上海壹账通金融科技有限公司 | Data processing method, electronic equipment and computer-readable recording medium |
CN107589907B (en) * | 2017-08-10 | 2019-12-13 | 深圳壹账通智能科技有限公司 | Data processing method, electronic device and computer readable storage medium |
CN109032505A (en) * | 2018-06-26 | 2018-12-18 | 深圳忆联信息系统有限公司 | Data read-write method, device, computer equipment and storage medium with timeliness |
CN109408496A (en) * | 2018-09-18 | 2019-03-01 | 沈文策 | A kind of method and device reducing data redundancy |
CN109815287A (en) * | 2019-01-30 | 2019-05-28 | 网易(杭州)网络有限公司 | A kind of data base access system |
CN109902130A (en) * | 2019-01-31 | 2019-06-18 | 北京明略软件系统有限公司 | A kind of date storage method, data query method and apparatus, storage medium |
CN110347545A (en) * | 2019-05-21 | 2019-10-18 | 深圳壹账通智能科技有限公司 | A kind of test method and device of business platform cache policy |
CN111708783A (en) * | 2020-06-18 | 2020-09-25 | 北京金山云网络技术有限公司 | Data storage and data recovery method and device and electronic equipment |
CN112434066A (en) * | 2020-11-24 | 2021-03-02 | 深圳市前海迅智系统科技有限公司 | Multi-channel high-frequency real-time data aggregation method and terminal |
CN113778723A (en) * | 2021-11-11 | 2021-12-10 | 中汽数据(天津)有限公司 | Data playback method, electronic device and readable storage medium |
CN114490775A (en) * | 2022-02-11 | 2022-05-13 | 广东美房智高机器人有限公司 | Robot real-time operation data processing method, device, server and medium |
Also Published As
Publication number | Publication date |
---|---|
CN106682193B (en) | 2021-01-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106682193A (en) | Device and method for data persistent storage on basis of cache | |
US8429134B2 (en) | Distributed database recovery | |
CN102929750B (en) | Nonvolatile media dirty region tracking | |
US8396840B1 (en) | System and method for targeted consistency improvement in a distributed storage system | |
CN109086388B (en) | Block chain data storage method, device, equipment and medium | |
US8560500B2 (en) | Method and system for removing rows from directory tables | |
KR101870521B1 (en) | Methods and systems for improving storage journaling | |
US8468134B1 (en) | System and method for measuring consistency within a distributed storage system | |
US6675180B2 (en) | Data updating apparatus that performs quick restoration processing | |
US11176110B2 (en) | Data updating method and device for a distributed database system | |
US10261705B2 (en) | Efficient data consistency verification for flash storage | |
CN103475716B (en) | The method and system of data sharing is realized by shared storage | |
US10169391B2 (en) | Index management | |
CN113111129A (en) | Data synchronization method, device, equipment and storage medium | |
CN112579327B (en) | Fault detection method, device and equipment | |
CN106557278A (en) | A kind of method of data cached persistence | |
CN1696906A (en) | Method, system and program product for autonomous error recovery for memory devices | |
CN111367926A (en) | Data processing method and device for distributed system | |
US20140279943A1 (en) | File system verification method and information processing apparatus | |
CN108021562B (en) | Disk storage method and device applied to distributed file system and distributed file system | |
US10783042B2 (en) | System and method of assessing and managing storage device degradation | |
WO2021027271A1 (en) | Bad block information protection method and apparatus, computer device and storage medium | |
US7409519B2 (en) | Synchronizing logical systems | |
US20100169572A1 (en) | Data storage method, apparatus and system for interrupted write recovery | |
CN111949479B (en) | Interactive system and index creation condition determining method and equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |