CN107103055A - Towards Hourglass the and Piggyback algorithms of memory refreshing intensive procedure - Google Patents
Towards Hourglass the and Piggyback algorithms of memory refreshing intensive procedure Download PDFInfo
- Publication number
- CN107103055A CN107103055A CN201710237896.9A CN201710237896A CN107103055A CN 107103055 A CN107103055 A CN 107103055A CN 201710237896 A CN201710237896 A CN 201710237896A CN 107103055 A CN107103055 A CN 107103055A
- Authority
- CN
- China
- Prior art keywords
- data
- checkpoint
- updater
- pointer
- threads
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Abstract
The present invention discloses a kind of Hourglass and Piggyback algorithms towards memory refreshing intensive procedure, and its excellent in performance can avoid mass data from copying from the pointer switching technology between updater and dumper threads.It periodically occurs role swap of updater and dumper threads, can by periodically endless multiplexing, once part above for it is empty when exchange above and below role.Less internal memory and the delay effect do not shaken substantially can be possessed.It belongs to one kind of lightweight checkpointing algorithm, and Hourglass combines two best at present algorithms zigzag and pingpong, so as to be exchanged and bit bit flag using the advantage pointer of two.Piggyback algorithms improve performance by providing a kind of full dose snapshot, so as to support real-time olap and oltp application.It has the advantages that smaller EMS memory occupation, full dose snapshot expense, smaller delay, delay evenly.
Description
Technical field
The present invention relates to the Backup and Restore technical field of memory database, relate in particular to a kind of close towards memory refreshing
Hourglass the and Piggyback algorithms of collection type program, belong to one kind of lightweight checkpointing algorithm.
Background technology
One major class application is related to data-intensive renewal operation, most of them can all pursue it is instantaneous respond, it is high
The performance handled up with High Availabitity.Such as in 20 end of the centurys, the backstage of large-scale stock exchange (such as Shanghai Stock Exchange)
The performance of electronic trading system (ETS) has reached that 100000 affairs are per second, and can guarantee that when disaster recovery
Affairs zero are lost.Also more examples, such as large-scale game on line (MMOS) and being enough supports the performance of Black Friday
The electric business system of scape.These data update the technology realization that intensive program is generally calculated using internal memory.One of which strategy
It is to use memory database, it can be provided faster and the predictable performance advantage of relative disk database is (because data are direct
It can be accessed in internal memory).
The high availability of memory database is dependent on a more accurate recovery subsystem.Due to for disk database
(DRDB) it is special for updating intensive program that traditional Restoration Mechanism does not have, if directly using former recovery system
And make no modifications, the performance of whole system may be limited by recovery system.Based on this, it is believed that memory database
Recovery system, especially transaction journal and check point subsystem, it is necessary to be redesigned.
For transaction journal, due to the high-throughput performance of memory database, more daily records can be produced in the unit interval,
Therefore IO expense can be aggravated, in order to prevent transaction journal from turning into the bottleneck of systematic function, many relatively optimization methods are carried
Go out, such as accelerate IO performances (by New Hardware), reduction I/O frequency (transaction set submits strategy), or reduce IO size
(log compression, logredo).The research of this part is relatively more abundant at present.
For checkpoint technology, we can be by periodically generating snapshot and truncated log to reach reduces the mesh of daily record
's.It is apparent that the frequency of checkpoint generation is more frequent more better soon, benefit is that the daily record amount generated twice between checkpoint is relative
It is smaller.However, it is very important so frequently to perform the overhead brought checkpoint.Firstly, since checkpoint is held
Row causes delay increase to have a great impact to throughput of system and response time.Secondly, different from disk check point, in number
According to before brushing disk, internal memory checkpoint needs to safeguard a completely the same data copy in internal memory.Inappropriate processing
The obvious shake of delay may be caused, even system is stalled.
Consistency check point algorithm under serial tranactions processing model.It is required that checkpointing algorithm can back up uniformity number
According to and not having excessive loss of throughput, its difficult point is how as fast as possible to generate in uniformity in internal memory
Snapshot is deposited, then asynchronous Backup Data is to disk.Most direct algorithm isSnapshot (NS), it is by pinning
There are data and then generate a data copy.However, NS make use of the lock operation of a coarseness, this will cause one very
Obvious latency shakes.Another algorithm Copy On Update (COU), using the form of shadow copy, in first number
When according to write-in.The generation snapshot stage, it will not cause system is excessive to get stuck, but all be needed for data access each time
Want memory copying operation and fine-grained lock operation.NS and COU are required for going to safeguard the state of uniformity using lock operation.In order to
Ensure system low delay and the phenomenon that do not get stuck, lock free algorithm design is highly significant.
As far as we know, only a small amount of research is related to lock free consistency check point algorithm, and typical work is
Tuan cao propose two wait free checkpointing algorithm, Zigzag and Pingpong.Main thinking be by using
Extra internal memory and bit array goes to avoid the use of lock, and can eliminate delay jitter phenomenon.Unfortunately work as data
When amount is very big, Zigzag still has very serious jitter phenomenon.Pingpong algorithms need 3 to 4 times of EMS memory occupation,
And the mode write dependent on a kind of redundancy.It is as far as possible few to make although the physical memory of high-capacity and high-speed largely occurs
With internal memory, no matter being all rational from price or energy angle, especially for renewal intensive procedure.Therefore, synthesis is examined
Consider memory cost and performance (low delay, uniform delay distribution, fast quick-recovery), the internal memory inspection without lock of design lightweight is counted
Method is necessary.
The content of the invention
For the defect of above-mentioned prior art, the present invention provides a kind of towards memory refreshing intensive procedure
Hourglass and Piggyback algorithms, belong to one kind of lightweight checkpointing algorithm.It can solve the problem that the deficiencies in the prior art,
There is more preferable high availability.
To achieve the above object, the technical solution used in the present invention is：
A kind of Hourglass algorithms towards memory refreshing intensive procedure, comprise the following steps：
S1, hypothesis renewal intensive procedure can periodically reach the state of a uniformity, with tick=1/T concept
The periodicity of description affairs reaches the characteristic of coherency state, and wherein T is tick duration；What one tick was represented is several
Interval between the time point for reaching coherency state twice according to collection；
S2, in moment t, it is assumed that data D state DtRepresent, D is the data set in internal memory；One checkpoint C is responsible for
Safeguard DtA compact data snapshot into disk unit, but must be ensured of in t+tcBefore moment, wherein tcIt is checkpoint
The execution cycle；
S3, by data D it is abstract be a continuous memory headroom, according to the form tissue of page；Any extra use
In the region of memory for completing checkpoint, useForm represent；
S4, simulate IMDB recovery system be two concurrent threads of Updater threads and Dumper threads, D andIt is different
Algorithm two thread accesses；
S5, will periodically check be divided into two stages of prepare stages and dump stages；DtCheckpoint is actually generated
Moment is in the finish time in each dump stages, tpAnd tdThe duration in prepare and dump stages is represented respectively；Perform checkpoint
Total time expense to=tp+tdNot over the time t in checkpoint cyclec=tp+k(k∈N+) that is to≤tc；
S6, the pointer switching technology using pingpong and zigzag bit Examined effect are uniform to obtain one
Accessing delay but also ensuring only needs two parts of internal memory, data set D and shadow copy numberAccording to Every part of data set is along with one
Bit arrayWith
Value represent whether D [i] is updated and whether needs in next round inspection within the current check point cycle
The point cycle is backed up；Value represent whether D [i] needs to persist within the current check point cycle；
S7, in a proof cycle, for renewal data set (D or) use " pU " pointer representation, another data
Collect use " pD " pointer representation for backup；
S8, pointer, which are exchanged, occurs the section at the beginning of each checkpoint；To ensure that updater threads do not read dirty data,
Extra flag bit dataFor indicating that number according to for reading newest dataNumber is read from D [i]
According to, otherwise fromRead data.
As the improvement to above-mentioned technical proposal, Updater threads, which are represented, updates intensive procedure, and Dumper threads are responsible for
Periodically perform checkpoint.
The present invention also provides a kind of Piggyback algorithms towards memory refreshing intensive procedure, comprises the following steps：
S1, pointer pU and pD be respectively directed to D andThe data that dumper thread batch duplicating persistences pD is pointed to,
Piggyback safeguards one or two of bit array
There are three kinds of states, { 0,1,2 } represents we need which number to read newest data in from；Updater readings data page i is all right from any a data set, becauseWork as Db[i]=1,
Updater reads data page i from D [i].Work as db[i]=2, updater fromRead data page；
PU and pD be respectively directed to D andBe initialised a full 0, within first checkpoint cycle, it is assumed that Updater
Thread have updated data page D [0], D [2], and D [3], and corresponding flag bit is configured to 1；
S2, start second checkpoint, pointer pU and pD are exchanged, and current pD is backed up in the way of full dose and is pointed to
D data；Meanwhile,Receive updater renewal operation；Second checkpoint weekPhaseData in finish time, D are all
It is current last state,
The extra Dumper of S3, definition：：WriteToOnline () function.It is responsible for covering data newest in the newest shapes of pD
Cover the data of relevant position in pU.
Compared with prior art, the present invention has the advantages and positive effects of：
Hourglass the and Piggyback algorithms towards memory refreshing intensive procedure of the present invention, its excellent in performance is come
The pointer switching technology between updater and dumper threads is come from, it can avoid mass data from copying.It is periodic
Occur the role swap of updater and dumper threads, just look like that hourglass is the same, can be by periodically endless multiplexing, one
Role above and below being exchanged when denier part above is sky.Less internal memory and the delay do not shaken substantially effect can be possessed
Really.
Above two algorithm, belongs to one kind of lightweight checkpointing algorithm, and Hourglass combines best at present two
Individual algorithm zigzag and pingpong, so as to be exchanged and bit bit flag using the advantage pointer of two.Piggyback algorithms are carried
High-performance is by providing a kind of full dose snapshot, so as to support real-time olap and oltp application.It has smaller internal memory
Occupancy, full dose snapshot expense, smaller delay, the advantage of delay evenly.
Embodiment
Below in conjunction with the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described,
Obviously, described embodiment is only a part of embodiment of the invention, rather than whole embodiments.Based in the present invention
Embodiment, the every other embodiment that those of ordinary skill in the art are obtained under the premise of creative work is not made, institute
Any modification, equivalent substitution and improvements of work etc., should be included in the scope of the protection.
The Hourglass algorithms towards memory refreshing intensive procedure of the present invention, comprise the following steps：
S1, data set D and shadow copy dataEvery part of data set is along with a bit arrayWith
Value represent whether D [i] is updated within the current check point cycle and whether needs to be backed up in the next round checkpoint cycle；Value represent whether D [i] needs to persist within the current check point cycle；
S2, in first proof cycle, for data set D " pU " pointer representations of renewal, another data set is used
In use " pD " pointer representation of backup.Meanwhile, when Updater is updated the data, it is required for settingData be 1.Dumper lines
Journey is directly backed upInterior data.
S3, pointer, which are exchanged, occurs the section at the beginning of next checkpoint；To ensure that updater threads do not read dirty data,
Extra flag bit dataFor indicating that number according to for reading newest dataNumber is read from D [i]
According to, otherwise fromRead data.
As the improvement to above-mentioned technical proposal, the present invention also provides a kind of towards memory refreshing intensive procedure
Piggyback algorithms, comprise the following steps：
S1, pointer pU and pD be respectively directed to D andThe data that dumper thread batch duplicating persistences pD is pointed to,
Piggyback safeguards the array of one or two of bit
There are three kinds of states, { 0,1,2 } represents we need which number to read newest data in from；Updater readings data page i is all right from any a data set, becauseWork as Db[i]=1,
Updater reads data page i from D [i].Work as Db[i]=2, updater fromRead data page；
PU and pD be respectively directed to D andBe initialised a full 0, within first checkpoint cycle, it is assumed that Updater
Thread have updated data page D [0], D [2], and D [3], and corresponding flag bit is configured to 1；
S2, start second checkpoint, pointer pU and pD are exchanged, and current pD is backed up in the way of full dose and is pointed to
D data；Meanwhile,Receive updater renewal operation；Second checkpoint weekPhaseData in finish time, D are all
It is current last state；
The extra Dumper of S3, definition：：WriteToOnline () function；For correcting the data in pU, in checkpoint
Start time ensures newest data mode.
Compared with prior art, the present invention has the advantages and positive effects of：
Hourglass the and Piggyback algorithms towards memory refreshing intensive procedure of the present invention, its excellent in performance is come
The pointer switching technology between updater and dumper threads is come from, it can avoid mass data from copying.It is periodic
Occur the role swap of updater and dumper threads, just look like that hourglass is the same, can be by periodically endless multiplexing, one
Role above and below being exchanged when denier part above is sky.Less internal memory and the delay do not shaken substantially effect can be possessed
Really.
Above two algorithm, belongs to one kind of lightweight checkpointing algorithm, and Hourglass combines best at present two
Individual algorithm zigzag and Ping-Pong, so as to be exchanged and bit bit flag using the advantage pointer of two.Piggyback algorithms
Performance is improved by providing a kind of full dose snapshot, so as to support real-time olap and oltp application.It has smaller interior
Deposit occupancy, full dose snapshot expense, smaller delay, the advantage of delay evenly.
General principle, principal character and the advantages of the present invention of the present invention has been shown and described above.The technology of the industry
Personnel are it should be appreciated that the present invention is not limited to the above embodiments, and the simply explanation described in above-described embodiment and specification is originally
The principle of invention, various changes and modifications of the present invention are possible without departing from the spirit and scope of the present invention, these changes
Change and improvement all fall within the protetion scope of the claimed invention.The claimed scope of the invention by appended claims and its
Equivalent is defined.
Claims (2)
1. a kind of Hourglass algorithms towards memory refreshing intensive procedure, it is characterised in that：Comprise the following steps：
S1, hypothesis renewal intensive procedure can periodically reach the state of a uniformity, with tick=1/T conceptual description
The periodicity of affairs reaches the characteristic of coherency state, and wherein T is tick duration；What one tick was represented is data set
Interval between the time point for reaching coherency state twice；
S2, in moment t, it is assumed that data D state DtRepresent, D is the data set in internal memory；One checkpoint C is responsible for safeguarding Dt
A compact data snapshot into disk unit, but must be ensured of in t+tcBefore moment, wherein tcIt is the execution of checkpoint
Cycle；
S3, by data D it is abstract be a continuous memory headroom, according to the form tissue of page；Any extra has been used for
Into the region of memory of checkpoint, useForm is represented；
S4, simulate IMDB recovery system be two concurrent threads of Updater threads and Dumper threads, D andDifferent calculations
Two thread accesses of method；
S5, will periodically check be divided into two stages of prepare stages and dump stages；DtCheckpoint actually generates the moment
In the finish time in each dump stages, tpAnd tdThe duration in prepare and dump stages is represented respectively；Perform the total of checkpoint
Time overhead to=tp+tdNot over the time t in checkpoint cyclec=tp+k(k∈N+) that is to≤tc；
S6, the pointer switching technology using pingpong and zigzag bit Examined effect uniformly access to obtain one
Being delayed but also ensuring only needs two parts of internal memory, data set D and shadow copy dataEvery part of data set is along with a bit
Bit arrayWith
Value represent D [i] whether be updated within the current check point cycle and whether need next round checkpoint week
Phase is backed up；Value represent whether D [i] needs to persist within the current check point cycle；
S7, in a proof cycle, for renewal data set (D or) " pU " pointer representation is used, another data set is used
In use " pD " pointer representation of backup；
S8, pointer, which are exchanged, occurs the section at the beginning of each checkpoint；To ensure that updater threads do not read dirty data, additionally
Flag bit dataFor indicating that number according to for reading newest dataData are read from D [i], it is no
Then fromRead data.
2. a kind of Piggyback algorithms towards memory refreshing intensive procedure, it is characterised in that：Comprise the following steps：
S1, pointer pU and pD be respectively directed to D andThe data that dumper thread batch duplicating persistences pD is pointed to,
Piggyback safeguards one or two of bit array
There are three kinds of states, { 0,1,2 } represents we need which number to read newest data in from；
Updater readings data page i is all right from any a data set, becauseWork as Db[i]=1, updater is from D
[i] reads data page i；Work as Db[i]=2, updater fromRead data page；
PU and pD be respectively directed to D andBe initialised a full 0, within first checkpoint cycle, it is assumed that Updater threads
Data page D [0], D [2], and D [3] are have updated, corresponding flag bit is configured to 1；
S2, start second checkpoint, pointer pU and pD are exchanged, and the D that current pD is pointed to is backed up in the way of full dose
Data；Meanwhile,Operated by updater renewal；Second checkpoint weekPhaseData in finish time, D are all to work as
Preceding last state；
The extra Dumper of S3, definition：：WriteToOnline () function；It is responsible for data cover pU newest in the newest shapes of pD
The data of middle relevant position.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710237896.9A CN107103055B (en) | 2017-03-29 | 2017-03-29 | Hourglass method for intensive program for memory updating |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710237896.9A CN107103055B (en) | 2017-03-29 | 2017-03-29 | Hourglass method for intensive program for memory updating |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107103055A true CN107103055A (en) | 2017-08-29 |
| CN107103055B CN107103055B (en) | 2020-05-12 |
Family
ID=59676053
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710237896.9A Active CN107103055B (en) | 2017-03-29 | 2017-03-29 | Hourglass method for intensive program for memory updating |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107103055B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1787532A (en) * | 2004-12-07 | 2006-06-14 | Nvlsoft有限公司 | System and method for providing 3D image production service |
| CN101395602A (en) * | 2005-12-29 | 2009-03-25 | 亚马逊科技公司 | Method and apparatus for a distributed file storage and indexing service |
| CN103051474A (en) * | 2012-12-19 | 2013-04-17 | 浪潮电子信息产业股份有限公司 | Method of large scale cluster network cabling |
-
2017
- 2017-03-29 CN CN201710237896.9A patent/CN107103055B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1787532A (en) * | 2004-12-07 | 2006-06-14 | Nvlsoft有限公司 | System and method for providing 3D image production service |
| CN101395602A (en) * | 2005-12-29 | 2009-03-25 | 亚马逊科技公司 | Method and apparatus for a distributed file storage and indexing service |
| CN103051474A (en) * | 2012-12-19 | 2013-04-17 | 浪潮电子信息产业股份有限公司 | Method of large scale cluster network cabling |
Non-Patent Citations (2)
| Title |
|---|
| 程春玲等: "基于SSC-tree流聚类的入侵检测算法", 《系统工程与电子技术》 * |
| 袁梅宇: "基于动态链接库进行BLOB数据交换的方法", 《微计算机应用》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107103055B (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220309049A1 (en) | Key-Value Store System | |
| Arulraj et al. | Write-behind logging | |
| Stenström et al. | Comparative performance evaluation of cache-coherent NUMA and COMA architectures | |
| Agawal et al. | Memory-reference characteristics of multiprocessor applications under MACH | |
| DE102011076894B9 (en) | Persistent storage for a main memory of a processor | |
| Pelley et al. | Storage management in the NVRAM era | |
| Papagiannis et al. | An efficient memory-mapped key-value store for flash storage | |
| Dubois et al. | Delayed consistency and its effects on the miss rate of parallel programs | |
| US5574874A (en) | Method for implementing a checkpoint between pairs of memory locations using two indicators to indicate the status of each associated pair of memory locations | |
| US20120017037A1 (en) | Cluster of processing nodes with distributed global flash memory using commodity server technology | |
| US8396937B1 (en) | Efficient hardware scheme to support cross-cluster transactional memory | |
| Kissinger et al. | KISS-Tree: smart latch-free in-memory indexing on modern architectures | |
| Graefe | A survey of B-tree logging and recovery techniques | |
| JPH04205041A (en) | Cache memory block replacing method | |
| DE112020006443T5 (en) | STORAGE CLASS MEMORY | |
| Xia et al. | Taurus: Lightweight parallel logging for in-memory database management systems | |
| CN114780025A (en) | Software RAID (redundant array of independent disks) request processing method, controller and RAID storage system | |
| DE102021108478B4 (en) | Device and control for cache directory CAM error detection and correction | |
| Pan et al. | Hart: A concurrent hash-assisted radix tree for dram-pm hybrid memory systems | |
| CN113220490A (en) | Transaction persistence method and system for asynchronous write-back persistent memory | |
| CN107103055A (en) | Towards Hourglass the and Piggyback algorithms of memory refreshing intensive procedure | |
| Zhong et al. | Redesigning High-Performance LSM-based Key-Value Stores with Persistent CPU Caches | |
| DE102021129138A1 (en) | COVER-BASED COMPRESSION WITH LOW COMPLEXITY | |
| Fujimoto et al. | The roll back chip: Hardware support for distributed simulation using Time Warp | |
| CN113722052A (en) | Nonvolatile memory updating method based on data double versions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |