CN109800246A - A kind of Data lifecycle management method towards magnanimity scale KV caching - Google Patents
A kind of Data lifecycle management method towards magnanimity scale KV caching Download PDFInfo
- Publication number
- CN109800246A CN109800246A CN201910128202.7A CN201910128202A CN109800246A CN 109800246 A CN109800246 A CN 109800246A CN 201910128202 A CN201910128202 A CN 201910128202A CN 109800246 A CN109800246 A CN 109800246A
- Authority
- CN
- China
- Prior art keywords
- time window
- data
- time
- expired
- management method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of Data lifecycle management methods towards magnanimity scale KV caching, it is characterised in that: including following process: 1, the data cached user side of KV specifies expired time；2, according to specified expired time, corresponding time window is calculated, executes operating process, operating process includes newly-increased data, deletes data and/or more new data；3, storage and life cycle management are carried out to KV, the present invention has formulated the Data lifecycle management method towards magnanimity scale KV caching, and precious memory source can be saved in the case where not influencing KV caching performance.
Description
Technical field
The present invention relates to computer big data fields, and in particular to the data life period pipe towards magnanimity scale KV caching
Reason method.
Background technique
KV cache database has in the industry cycle been widely used because of its high performance key-value access module.Such as
Redis, memcached are used in double 11 high concurrent ordering systems, user's portrait system that Domestic News are recommended etc..By
It is to use precious memory source as storage medium, therefore the life cycle needs of KV data are effective in KV cache database
Management, in order to reduce the waste of memory source.
The existing KV cache database of industry at present, in order to balance memory use and service performance, in data life period
Lazy delete mode is generallyd use in management: i.e. when KV data have reached life cycle, data not will be deleted, but be marked
It can be deleted, when data are read out next time, return is empty and is deleted.In the case where extensive KV is data cached,
Lazy delete mode makes a large amount of stale data still occupy precious memory source, serious waste memory source.
Term is explained:
The abbreviation of 1.KV:key-value by key inquiry, updates, deletion value.Common KV caching such as redis,
Memcached etc..
2. life cycle: key-value data cannot if it exceeds specified duration in the storage time of database
It is accessed to.
Summary of the invention
In order to solve problem above, under the data cached scene of the KV of magnanimity scale, it is raw that the application introduces a kind of data
Cycle management method, i.e. the Data lifecycle management method towards magnanimity scale KV caching are ordered, memory source is saved.
The present invention solves technical solution used in technical problem:
A kind of Data lifecycle management method towards magnanimity scale KV caching, including following process:
The data cached user side of 1.KV specifies expired time；
2. calculating corresponding time window according to specified expired time, operating process is executed, operating process includes newly-increased
Data delete data and/or more new data；
3. couple KV carries out storage and life cycle management.
Wherein, during time window is the expired time of the data cached user side's setting of KV, but KV caching execution is expired,
More than time window, batch deletes expired data.
The calculating process of time window are as follows:
The expired time length (second) that expired time=present system time (to the second)+user specifies is calculated,
Expired time is calculated to minute, the creation area where calculating expired time is then obtained according to time window size
Between.
The time window can be adjusted dynamically according to the actual situation, i.e., user is to change expired time, and inside can trigger
Following execution process.
The logical construction of time window uses zipper method.
The newly-increased data step is as follows:
Newly-increased one data<k, v>,
A. write-in<k, v>data；
B. according to specified expired time, corresponding time window is calculated, and update life cycle management；
(1) if time window has existed, the k is inserted into the corresponding key list of the time window；
(2) if time window is not present, it is inserted into the time window in the suitable position of time window list, guarantees have
Sequence, while the time window needs to be directed toward k；
The deletion data step is as follows:
Deletion<k, v>data,
A. execution<k, v>deletion；
B. according to the expired time of k, corresponding time window is calculated；
C. the key list being directed toward from the time window is searched the key node comprising k and is deleted；
It is suddenly as follows to update expired time step:
Update<k, v>expired time,
A. inquiry k has expired time, calculates corresponding time window；
B. from the corresponding key list lookup of the time window include the key node of k, and delete；
C. the corresponding time window of new expired time is calculated；
D. according to specified expired time, corresponding time window is calculated, and update life cycle management；
(1) if time window has existed, the k is inserted into the corresponding key list of the time window；
(2) if time window is not present, it is inserted into the time window in the suitable position of time window list, guarantees have
Sequence, while the time window needs to be directed toward k.
Process of the present invention is as follows:
One, defines time window；
Two, define time window data structure；
Three, execute the operating processes such as data increase/deletion/update expired time；
Four, carry out storage and life cycle management to KV according to above-mentioned 3 points.
The present invention has formulated the Data lifecycle management method towards magnanimity scale KV caching, is not influencing KV caching property
Precious memory source can be saved in the case where energy.Performance test data is as follows:
Detailed description of the invention
Fig. 1 is the zipper method logical construction flow chart of time window of the present invention.
Fig. 2 is to be inserted into the schematic diagram of the k in the corresponding key list of the time window when time window has existed.
Fig. 3 is to be not present when time window, is inserted into the time window in the suitable position of time window list, at the same this when
Between window need to be directed toward the schematic diagram of k.
Fig. 4 is the schematic diagram for deleting data.
Fig. 5 has existed for time window, is inserted into the schematic diagram of the k in the corresponding key list of the time window.
Fig. 6 is not present for time window, is inserted into the time window, while the time in the suitable position of time window list
The schematic diagram of window direction k.
Specific embodiment
3.2.1 time window is defined
The key concept that this programme introduces: time window (regular hour span).That is the data cached user side of KV
Set expired time, but KV caching execute it is expired during, with 10 minutes for a window, go batch to delete expired
Data.(this window value can be adjusted dynamically according to the actual situation)
Such as one KV data of write-in:<" hello ", 2>, the presently written time are as follows: 2018-12-10 12:02:00, mistake
Time phase is 600s, then theoretically this data can be deleted in 2018-12-10 12:12:00 because of expired.
Similarly, an other data:<" world ", 3>, the presently written time are as follows: 2018-12-10 12:07:00, mistake
Time phase is 600s, then theoretically this data can be deleted in 2018-12-10 12:17:00 because of expired.
After introducing time window mechanism, this two data all will not expired time point deletion theoretically.Their mistake
Time point phase belongs to: [2018-12-10_12:10,2018-12-10_12:20] this time window, therefore in 2018-
When 12-10_12:20, this two expired data are successively deleted.Similarly, if stale data in this time window
More, batch is also only needed successively to execute deletion, memory source will be recovered quickly.
3.2.2 the data structure of time window is defined
The logical construction of time window uses zipper method, such as Fig. 1.
Wherein, TimeWindow node indicates a time window, such as: [2018-12-10_12:00,2018-12-
10_12:10], it is orderly connected in a manner of from small to large between TimeWindow node, quickly can search and be inserted into this way and is new
TimeWindow node.
TimeWindow node is also directed to the list of a key, indicates the expired time of key all in this list,
All in this time window, i.e., time point at the end TimeWindow, all key in this list can be deleted in batches
And its data.
3.2.3 data update operating process
1) data are increased newly
Newly-increased one data<k, v>the step of it is as follows:
C. write-in<k, v>data；
D. according to specified expired time, corresponding time window is calculated, and update life cycle management.
(3) if time window has existed, the k is inserted into the corresponding key list of the time window；As shown in Figure 2.
(4) if time window is not present, it is inserted into the time window in the suitable position of time window list, guarantees have
Sequence, while the time window needs to be directed toward k.As shown in Figure 3.
2) data are deleted
Deletion<k, v>data operating procedure are as follows:
D. execution<k, v>deletion；
E. according to the expired time of k, corresponding time window is calculated；
F. the key list being directed toward from the time window searches the key node comprising k and deletes (as shown in Figure 4).
1) expired time is updated
The step of update<k, v>expired time, is as follows:
E. inquiry k has expired time, calculates corresponding time window；
F. from the corresponding key list lookup of the time window include the key node of k, and delete；
G. the corresponding time window of new expired time is calculated；
H. according to specified expired time, corresponding time window is calculated, and update life cycle management.
(1) if time window has existed, the k is inserted into the corresponding key list of the time window；As shown in Figure 5.
(3) if time window is not present, it is inserted into the time window in the suitable position of time window list, guarantees have
Sequence, while the time window needs to be directed toward k.As shown in Figure 6.
Performance test data is as follows:
The present invention has formulated the Data lifecycle management method towards magnanimity scale KV caching, is not influencing KV caching property
Precious memory source can be saved in the case where energy.
Claims (8)
1. a kind of Data lifecycle management method towards magnanimity scale KV caching, it is characterised in that: including following process:
The data cached user side of 1.KV specifies expired time；
2. calculating corresponding time window according to specified expired time, operating process is executed, operating process includes newly-increased number
According to, delete data and/or more new data；
3. couple KV carries out storage and life cycle management.
2. Data lifecycle management method according to claim 1, it is characterised in that: time window is that KV is data cached
The expired time of user side's setting, but KV caching execute it is expired during, be more than time window, batch deletes expired number
According to.
3. Data lifecycle management method according to claim 1, it is characterised in that: the calculating process of time window
Are as follows:
The expired time length (second) that expired time=present system time (to the second)+user specifies is calculated,
Expired time is calculated to minute, the creation section where calculating expired time is then obtained according to time window size.
4. Data lifecycle management method according to claim 1, it is characterised in that: the time window can basis
Actual conditions dynamic adjusts.
5. Data lifecycle management method according to claim 1, it is characterised in that: the logical construction of time window is adopted
With zipper method.
6. Data lifecycle management method according to claim 1, it is characterised in that: the newly-increased data step is such as
Under:
Newly-increased one data<k, v>,
A. write-in<k, v>data；
B. according to specified expired time, corresponding time window is calculated, and update life cycle management；
(1) if time window has existed, the k is inserted into the corresponding key list of the time window；
(2) if time window is not present, it is inserted into the time window in the suitable position of time window list, is guaranteed orderly,
The time window needs to be directed toward k simultaneously.
7. Data lifecycle management method according to claim 1, it is characterised in that: the deletion data step is such as
Under:
Deletion<k, v>data,
A. execution<k, v>deletion；
B. according to the expired time of k, corresponding time window is calculated；
C. the key list being directed toward from the time window is searched the key node comprising k and is deleted.
8. Data lifecycle management method according to claim 1, it is characterised in that: the expired time step of update is rapid
It is as follows:
Update<k, v>expired time,
A. inquiry k has expired time, calculates corresponding time window；
B. from the corresponding key list lookup of the time window include the key node of k, and delete；
C. the corresponding time window of new expired time is calculated；
D. according to specified expired time, corresponding time window is calculated, and update life cycle management；
(1) if time window has existed, the k is inserted into the corresponding key list of the time window；
(2) if time window is not present, it is inserted into the time window in the suitable position of time window list, is guaranteed orderly,
The time window needs to be directed toward k simultaneously.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910128202.7A CN109800246A (en) | 2019-02-21 | 2019-02-21 | A kind of Data lifecycle management method towards magnanimity scale KV caching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910128202.7A CN109800246A (en) | 2019-02-21 | 2019-02-21 | A kind of Data lifecycle management method towards magnanimity scale KV caching |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109800246A true CN109800246A (en) | 2019-05-24 |
Family
ID=66561146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910128202.7A Pending CN109800246A (en) | 2019-02-21 | 2019-02-21 | A kind of Data lifecycle management method towards magnanimity scale KV caching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109800246A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105426415A (en) * | 2015-10-30 | 2016-03-23 | Tcl集团股份有限公司 | Management method, device and system of website access request |
| CN109087055A (en) * | 2018-06-06 | 2018-12-25 | 北京达佳互联信息技术有限公司 | The control method and device of service request |
| CN109344296A (en) * | 2018-08-30 | 2019-02-15 | 武汉斗鱼网络科技有限公司 | Realize domain life cycle control method, system, server and the storage medium of the HASH key of Redis |
-
2019
- 2019-02-21 CN CN201910128202.7A patent/CN109800246A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105426415A (en) * | 2015-10-30 | 2016-03-23 | Tcl集团股份有限公司 | Management method, device and system of website access request |
| CN109087055A (en) * | 2018-06-06 | 2018-12-25 | 北京达佳互联信息技术有限公司 | The control method and device of service request |
| CN109344296A (en) * | 2018-08-30 | 2019-02-15 | 武汉斗鱼网络科技有限公司 | Realize domain life cycle control method, system, server and the storage medium of the HASH key of Redis |
Non-Patent Citations (3)
| Title |
|---|
| 壹頁書: ""Redis过期删除策略"", 《HTTP://BLOG.ITPUB.NET/29254281/VIEWSPACE-2106910/》 * |
| 程序猿大哥: ""redis过期键删除策略"", 《HTTPS://WWW.CNBLOGS.COM/ZHANGCHAO-LETV/ARTICLES/6119313.HTML》 * |
| 舒小贱: ""redis的过期时间和过期删除机制"", 《HTTPS://WWW.JIANSHU.COM/P/9352D20FB2E0》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10331572B2 (en) | Selective data mirroring for in-memory databases | |
| Macko et al. | Llama: Efficient graph analytics using large multiversioned arrays | |
| CN108475276B (en) | In-memory key-value storage for multi-model databases | |
| EP2987096B1 (en) | Caching external data sources for sql processing | |
| US9305046B2 (en) | Compressing a multi-version database | |
| US8868831B2 (en) | Caching data between a database server and a storage system | |
| EP3465473B1 (en) | Versioning and non-disruptive servicing of in-memory units in a database | |
| EP1504375B1 (en) | Providing a useable version of the data item | |
| Bader et al. | Stinger: Spatio-temporal interaction networks and graphs (sting) extensible representation | |
| CN102117338B (en) | Data base caching method | |
| US10437688B2 (en) | Enhancing consistent read performance for in-memory databases | |
| US20170116280A1 (en) | Compression units in an index block | |
| WO2019184618A1 (en) | Method and device for storing data, server, and storage medium | |
| WO2018205151A1 (en) | Data updating method and storage device | |
| CN109690522B (en) | Data updating method and device based on B+ tree index and storage device | |
| US9811560B2 (en) | Version control based on a dual-range validity model | |
| CN101236564A (en) | Mass data high performance reading display process | |
| CN104424222B (en) | Database index method and device | |
| CN112148736A (en) | Method, device and storage medium for caching data | |
| CN109800246A (en) | A kind of Data lifecycle management method towards magnanimity scale KV caching | |
| US10417215B2 (en) | Data storage over immutable and mutable data stages | |
| US10747782B2 (en) | Efficient dual-objective cache | |
| CN115469810A (en) | Data acquisition method, device, equipment and storage medium | |
| Duffy et al. | Dotori: A key-value ssd based kv store | |
| US10339069B2 (en) | Caching large objects in a computer system with mixed data warehousing and online transaction processing workload |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20190718 Address after: 344000 Pengjiayuan Group No. 12, Jiangjiacun, Xujia, Nancheng County, Fuzhou City, Jiangxi Province Applicant after: Pan Hongan Address before: 100025 Building No. 12, Xidawang Road, Chaoyang District, Beijing (Incubator 27660, National Advertising Industry Park) Applicant before: Beijing ah Ke Technology Co., Ltd. |
|
| TA01 | Transfer of patent application right | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190524 |
|
| WD01 | Invention patent application deemed withdrawn after publication |