CN105138622A - Append operation method for LSM tree memory system and reading and merging method for loads of append operation - Google Patents
Append operation method for LSM tree memory system and reading and merging method for loads of append operation Download PDFInfo
- Publication number
- CN105138622A CN105138622A CN201510501523.9A CN201510501523A CN105138622A CN 105138622 A CN105138622 A CN 105138622A CN 201510501523 A CN201510501523 A CN 201510501523A CN 105138622 A CN105138622 A CN 105138622A
- Authority
- CN
- China
- Prior art keywords
- key
- data
- value
- field
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2237—Vectors, bitmaps or matrices
Abstract
The invention provides an append operation method for an LSM tree memory system. The append operation method comprises following steps of: 1), constructing value to be appended and a key to be appended to be in a key-value structural body and utilizing a newly-constructed key-value structural body as latest data segments of the key to be appended for storage in a database; saving aggregate data amount of accumulated data segments before the key to be appended as for the newly-constructed key-value structural body and saving a storage position of a data segment before the key to be appended; and 2), updating table items of the key to be appended in the append table which records aggregate data amount of all data segments of all keys and storage positions of latest data segments. The invention further provides a reading and merging method for corresponding loads of append operation. The append operation method for the LSM tree memory system and the reading and merging method for loads of append operation have following beneficial effects: increase in system cost is avoided due to a read-write amplification problem of append operation; during the course of moving from a lower level to a higher level, loads of append operation are aggregated; and reading performance of loads of append operation can be improved.
Description
Technical field
The present invention relates to area information storage, specifically, the present invention relates to a kind of insertion (i.e. Append) of setting (i.e. LSMTree) storage system for LSM and operate and the reading of respective load and merging method.
Background technology
LSMTree storage system for random I/O, the performance impact of persistent storage is made to the storage system of optimization.The full name of LSMTree is the structure and working principle that Log-StructuredMerge-Tree, Fig. 1 show existing LSMTree storage system, and its major design is:
1, data hierarchy is placed, and wherein the data of lowermost layer are positioned at internal memory, and the data of other levels are positioned at persistence equipment (HDD/SSD etc.).
2, data strange land upgrades, and data write in log mode.
3, data constantly merge (merge) by the background process of system from low layer to high level, that is, deleted by the older version in K-V structure identical for key, retain the redaction of sequence.Data are along with the carrying out merged, and data are constantly moved to high level by low layer.
This design of LSMTree can reduce the tracking number of times of disk by batch write, and then improves the performance of persistent storage, and it is particularly useful for the situation of the random I/O of a large amount of disk.But, in LSMTree storage system, when update is carried out to data, need to read the value in K-V structure, splice and write back, and also data constantly can be moved to high level by low layer due to background process, therefore in fact the read/write number of times of update can be exaggerated, and this may cause system overhead to enlarge markedly, and affects memory property.
Summary of the invention
Therefore, task of the present invention is to provide the reading of update in a kind of LSMTree storage system that can overcome the above-mentioned defect of prior art and respective load and merges solution.
According to an aspect of the present invention, provide a kind of update method for LSMTree storage system, comprise the following steps:
1) value (sometimes value being called value herein) will be inserted into and be inserted into key (sometimes key being called key herein) and be configured to K-V structure and by the K-V structure newly built (sometimes K-V structure being called key-value structure herein) as being inserted into the latest data fragmented storage of key in database; For the new K-V structure built, preserve total size of the data sectional accumulated before being inserted into key, preserve the memory location being inserted into the last data segmentation of key;
2) list item being inserted into key of Append table (sometimes Append table being called insertion table herein) is upgraded; Total size of all data sectionals of each key of described insertion table record and the memory location of latest data segmentation.
Wherein, described step 1) in: in the K-V structure of described new structure, prevSize field and prevPos field are set, total size of the data sectional that prevSize field has been accumulated before preserving same key, prevPos field preserves the memory location of the last data segmentation of this K-V structure of same key.
Wherein, described step 1) comprise substep:
11) search at insertion table and be inserted into key, obtain corresponding list item;
12) according to the list item being inserted into key of gained, obtain and be inserted into total size of all data sectionals of key and the memory location of latest data segmentation before this update;
13) key will be inserted into and be inserted into value and be configured to K-V structure, step 12 is inserted in the prevSize field of this K-V structure) the total size being inserted into all data sectionals of key before this update of obtaining, in the prevPos field of this K-V structure, insert step 12) memory location being inserted into the latest data segmentation of key before this update of obtaining.
Wherein, each list item of described insertion table comprises size field and pos field, total size of all data sectionals of the key of this list item of size field record, the memory location of the latest data segmentation of the key of this list item of pos field record.
Wherein, described step 2) comprise substep:
21) the size field being inserted into the list item of key is updated to the total size adding all data sectionals after being inserted into value;
22) the pos field being inserted into the list item of key is updated to step 13) memory location of the K-V structure of gained.
According to a further aspect in the invention, additionally provide a kind of read method of the update load for LSMTree storage system, described update load comprises at least two the K-V structures as data sectional belonging to same key, and each K-V structure comprises prevSize field and prevPos field, wherein, the total size of data sectional of prevSize field for having accumulated before preserving this K-V structure under same key, prevPos field is for preserving the memory location of the last data segmentation of this K-V structure under same key;
Described read method comprises the following steps:
101) list item of the key that will read is searched at insertion table; Total size of all data sectionals of each key of described insertion table record and the memory location of latest data segmentation;
102) according to total size of all data sectionals of hitting list item record in insertion table, and total size of the data sectional accumulated before preserved in each K-V structure of the key that will read, find the K-V structure of the value commensurate in scope that will read and therefrom read the value that will read.
Wherein, described step 102) comprise substep:
1021) according to the latest data segmentation position of hitting list item record in insertion table, the K-V structure of the latest data segmentation of the key that will read is found to it can be used as current K-V structure;
1022) according to value size and the prevSize field of current K-V structure self, judge current K-V structure whether with the value commensurate in scope that will read, if so, then export and read result; If not, then continue to perform step 103);
1023) the K-V structure of last data segmentation is found according to the prevPos field of current K-V structure, then using the K-V structure of this last data segmentation as current K-V structure, re-execute step 1022), until all data sectionals of the key that will read are all found.
Wherein, described step 1022) in, export after reading result, whether the value scope continuing to judge to read all reads complete, if the judgment is Yes, then directly releases this and reads, if the judgment is No, jump to step 1023).
According to another aspect of the invention, additionally provide a kind of merging method of the update load for LSMTree storage system, described update load comprises at least two the K-V structures as data sectional belonging to same key, and each K-V structure comprises prevSize field and prevPos field, wherein, the total size of data sectional of prevSize field for having accumulated before preserving this K-V structure under same key, prevPos field is for preserving the memory location of the last data segmentation of this K-V structure under same key;
Described merging method comprises: in the union operation process of LSMTree storage system, is merged by each K-V structure under the same key belonging to described update load.
Wherein, the SST (i.e. orderly string table file) produced by Append class formation body is as the starting point of SST evolution table; The process that each K-V structure under the same key belonging to described update load merges is comprised the following steps:
201), when carrying out union operation in persistent storage layer, in SST evolution table, each SST participating in this union operation is searched;
202), when reading the structure participating in this union operation, the K-V structure of the different pieces of information segmentation as same key is merged.
Compared with prior art, the present invention has following technique effect:
1, the present invention is while LSMTree storage system realizes update function, avoids the system overhead increase that update causes because of read/write scale-up problem.
2, each K-V under the present invention same key that in the process of high level movement, update can be produced by low layer in data
partstructure is polymerized.
What 3, the present invention can improve the K-V structure that update produces reads performance.
Accompanying drawing explanation
Below, describe embodiments of the invention in detail by reference to the accompanying drawings, wherein:
Fig. 1 shows the structure and working principle of existing LSMTree storage system;
Fig. 2 shows the update principle schematic of one embodiment of the invention;
Fig. 3 shows the process flow diagram of the update method of one embodiment of the invention;
The process flow diagram of the read method of the load that the update that Fig. 4 shows one embodiment of the invention produces;
The data that Fig. 5 shows the Append class formation body of one embodiment of the invention merge process flow diagram.
Embodiment
According to one embodiment of present invention, a kind of update method in LSMTree storage system is provided.This embodiment performs insertion (i.e. Append) operation based on fragmented storage technology, simultaneously by the transformation of key and the introducing of insertion table, the problem that the value solving the different sections of same key connects each other, thus while realizing update normal function, avoid the read/write scale-up problem of update in LSMTree storage system, and then save system overhead.
Fig. 2 shows the update principle schematic of the present embodiment.As shown in Figure 2, for improving the performance of update, set up in the present embodiment and insert table, new project is added in the key field of the K-V structure of Append class, also establish SST evolution table simultaneously, first the K-V structure of insertion table, Append class and SST evolution table are described respectively below, and then describe the update method after improving.
1, insertion table is set up
That store in the insertion table (Append table) set up in the present embodiment is all key by update write into Databasce (DB), and total size (the total size of namely accumulated in a database value) of all value sections corresponding to this key, and the memory location of the current up-to-date one section of value of this key.As shown in Figure 2, in the insertion table of the present embodiment, each element comprises three parts: key, size (size represents the total size of value that this key accumulates in DB), and pos (memory location of up-to-date one section of value).If up-to-date one section of value exists in persistent storage equipment, then the memory location of value adopts SSTID to represent, if exist in internal memory, is then labeled as mem, and for convenience of following description, mem also can be considered as a special SSTID.
In a preferred embodiment, inserting table adopts RBTree structure to set up.
2, the increase of key field in Append class K-V structure
As everyone knows, LSMTree storage system realizes reading and writing data and other operation various based on K-V structure.Wherein, perform in the mode based on fragmented storage technology of the present embodiment the K-V structure that update produces and be called Append class K-V structure, also Append class K-V structure is called update load herein.In the present embodiment, be provided with update interface, wherein increase a update parameter, it is that the mode of " reading-splice value-to write back " traditionally realizes or realizes by the mode based on fragmented storage technology of the present embodiment that this update parameter is used to specify each update.This update parameter can carry out dynamic conditioning, to determine the implementation of each update by database (DB) according to the value size and performance requirement of carrying out update.
In the present embodiment, for the key in Append class K-V structure, except userKey, SeqNum, valueType field that K-V structure itself has, also increase following field: prevSize and PrevPos, as shown in Figure 2.Wherein, the prevSize representative value size that this key has accumulated in a database before this section of value write; PrevPos represents this section of value K-V the last period
partthe SSTID at the value place of structure, if this section is first paragraph, this field mark is head, wherein, K-V
partstructure represents the K-V structure corresponding to value segmentation under same key value.
3, SST evolution table is set up
As everyone knows, SST refers to the orderly string table file in LSMTree database, i.e. SortedStringTable, is abbreviated as SSTable or SST.
As shown in Figure 2, in the present embodiment, SST evolution table have recorded the alteration of the SST in storage system in the overall situation with the structure of digraph: each SST is after overcompression (compact) operation, and the key in this SST enters which or which new SST respectively.Each SST has oneself a counting, is called reference count, its representative be the number of not yet merged K-V on each SST.
When the key in a SST is compressed in (compact) to multiple SST, also need the scope of the key recording different branch (namely record is compressed to the key scope of each SST).For the new SST that squeeze operation is formed, when a key is compressed into this new SST, the reference count that this new SST is corresponding in SST evolution table adds 1; When squeeze operation to the value of the different sections of same key do merge time (during merging, prevPos field can be deleted), this SSTID (prevPos) reference count in SST evolution table subtracts 1, when certain SSTID is positioned at oriented source of graph and its reference count is 0, then delete the record of this SSTID.
SST evolution table can be used for the read operation accelerating multistage value, and its use-pattern describes in detail in follow-up read operation flow process.
Below, in conjunction with an embodiment, the update method after the present invention's improvement is described.
Fig. 3 shows the process flow diagram of the update method of one embodiment of the invention, and it have employed one and is different from the peculiar mode of " reading-splice value-to write back " to realize update.With reference to figure 3, this update method comprises the steps:
Step 301: receive the key that will perform update and the value that will insert, then searches this key (for convenience of describing, being hereinafter referred to as inserted into key) in insertion table.
Step 302: judge whether to find in insertion table to be inserted into key, if so, enter step 303, otherwise, enter step 308.
Step 303: read the size field and the pos field that are inserted into key from insertion table.
Step 304: judge that pos field is mem? if so, enter step 306, otherwise enter step 305.The object of this step is exactly judge that the preceding paragraph value of same key is whether also in internal memory.
Step 305: by current received key be inserted into value (i.e. key and value of step 301) and be configured to K-V
partstructure, puts the pos (refer to the content of pos field step 303 read-out by) of pervPos field for finding in insertion table in the key of this structure.Due to store in insertion table be this update before, the memory location of the up-to-date one section of value corresponding to same key, therefore in step 305, this update the last period value memory location be written with current K-V
partthe pervPos field of structure.After this step, perform step 307.
Step 306: by current received key be inserted into value (i.e. key and value of step 301) and be configured to K-V structure, the pervPos field of putting in the key of this structure is mem (in this case, a upper field is also in internal memory).After this step, perform step 307.
Step 307: by constructed K-V
partprevSize in structure key is set to the size (referring to the content of the size field read-out by step 303) found in insertion table.Due to store in insertion table be this update before, the total size of all value corresponding to same key, therefore, the total size of value that same key accumulates in a database before is just incorporated current K-V by the operation of this step
partthe size field of structure.After this step, perform step 314.
Step 308: in a database by taking (Get) operation to obtain the key be inserted into, i.e. the key of step 301.
Step 309: whether successfully judge to take operation, if unsuccessful, then think do not have corresponding key in system, enter step 310, if success, then enter step 311.
Step 310: key and value directly step 301 received, as the first paragraph of the structure of Append class, is designated as K-V
part.After enter step 312.
Step 311: take out corresponding K-V structure from database, then splicing is inserted into value, the structure obtained is designated as K-V
part, by K-V
partas the first paragraph of Append class formation body.After enter step 312.
Step 312: by K-V
partthe prevSize field of structure sets to 0, and pervPos field is set to head, to represent this K-V
partstructure is the first paragraph of the Append class formation body of same key value,
Step 313: insert constructed K-V in insertion table
partthe key of structure, and size field corresponding in insertion table is set to current K-V
partthe size of the value of structure, pos field is set to mem, represents current K-V
partstructure position is internal memory.Enter step 314.
Step 314: by putting (Put) operation by current K-V
partin structure write memory.
Step 315: needs by inclining operation (i.e. Dump operation) data are moved to database from internal memory time, be the element of mem for insertion table pos field, its pos field is set to the SSTID that will incline to the disk of (namely moving to).
Step 316: be the K-V of mem for the participation pervPos field operation of inclining
partstructure, is set to the SSTID that will incline to the disk of (namely will move to) by its pervPos field.Perform to incline again after step 316 completes and operation, data are moved to database from internal memory.
The introducing of above-mentioned flow process by showing transformation and the insertion of the key of update, the problem that the different section value solving same key connect each other, the basic function of update is achieved with the form of fragmented storage, the read/write that simultaneously also avoid the update of traditional " reading-splice value-to write back " formula is amplified, and saves system overhead.
The process flow diagram of the read method of the load that the update that Fig. 4 shows one embodiment of the invention produces.The load that wherein update produces refers to the K-V structure of Append class.The method comprises the following steps:
Step 401: receive the key that will read and the value scope that will read, search this key in insertion table.For ease of describing, be hereinafter referred to as key to be read, the value scope that read is called scope to be read.When reading large scale file, user does not often need to read whole file, but only needs the value reading wherein one section of specified scope, in this step, and the value scope that will read that what scope to be read referred to is exactly specified by this read operation.
Step 402: if find key to be read in insertion table, enter step 403, otherwise, enter step 404.
Step 403: read the K-V mated with key to be read in memtable
partstructure.As everyone knows, memtable is the memory data structure that LSMTree is conventional.Step 405 is entered after this step.
Step 404: read the K-V structure mated with key to be read in a traditional way from database.When this step, because there is not key to be read in insertion table, so can think that what will read is the load of non-Append class, therefore, can directly read by traditional mode.Now, namely this step terminates this read operation after completing.
Step 405: judge the K-V read from internal memory
partwhether structure and scope to be read have overlap, if the judgment is Yes, enter step 406, otherwise, enter step 409.
Step 406: judge the K-V read from internal memory
partwhether structure can cover scope to be read, if the judgment is No, enters step 407, otherwise, enter step 408.
Step 407: by current K-V
partbuffer memory (buffer) district is read out to the value of overlapping ranges to be read in structure.Then step 409 is entered.
Step 408: by current K-V
partdirectly read with the value of overlapping ranges to be read in structure, complete this read operation.
Step 409: read insertion table, finds the element mated with key to be read, obtains corresponding size and pos field.
Step 410: the SSTID recorded with pos field is starting point, read SST evolution table, wherein, according to branch's trend that key to be read determines evolution table, the SSTID of the first paragraph value position (i.e. place SST) of key to be read in lasting accumulation layer is finally obtained.Herein, first paragraph value refers to one section of up-to-date in persistent storage layer value of Append class formation body that key to be read with this mate.
Step 411: find the K-V mated with key to be read in corresponding SST according to current obtained SSTID
partstructure, according to this K-V
partthe prevSize field of structure judges this K-V
partwhether structure has overlapping with the scope to be read of current residual, if the judgment is No, then enters step 412, if the judgment is Yes, then enters step 413.
Step 412: current found K-V
partthe SSTID that the prevPos field of structure records, with this SSTID for starting point reads SST evolution table, wherein, branch's trend of evolution table is determined according to key to be read, finally obtain the SSTID of the preceding paragraph value position of mating with key to be read in persistent storage layer, then return step 411.
Step 413: judge current K-V
partwhether structure can cover the scope to be read of current residual, if not, enters step 414, otherwise, enter step 415.
Step 414: by current K-V
partread out to buffer area with the value of the overlapping ranges to be read of current residual in structure, then enter step 412.
Step 415: according to remaining scope to be read by current K-V
partin structure, corresponding value reads, and splices with the value in buffer memory, and obtain complete scope value to be read and returned, this is complete to the read operation of Append class formation body.
When user only needs a part of value, based on above-mentioned read operation flow process, the value scope that will read can be found exactly, and whole value need not be read, thus improve update produce Append class formation body read performance.
The data that Fig. 5 shows the Append class formation body of one embodiment of the invention merge process flow diagram.SST evolution table usually operation along with inclining of memTable and generates, and, evolution in the squeeze operation of persistent storage layer.Wherein, squeeze operation completes the typical operation that data merge from low layer to high level in LSMTree storage system.
As shown in Figure 5, the data of the Append class formation body of the present embodiment merge process flow diagram:
Step 501: when memTable carry out inclining operation time, inclining operation and can produce a series of SST.Now, for participation, this inclines and the Append class formation body of operation, generates SST evolution table and using the starting point of produced SST as this SST evolution table.In SST evolution table, represent corresponding SST with SSTID.
Step 502: when carrying out squeeze operation in persistent storage layer, searches each SST participating in this squeeze operation in SST evolution table.
Step 503: reading the K-V participating in this squeeze operation
parttime, merge the different section value of same key, the K-V structure after merging is stored in the new SST be compressed into.
Step 504: reading the K-V participating in this squeeze operation
parttime, the ref (i.e. reference count) of the SST corresponding to corresponding prevPos in SST evolution table is subtracted 1, if the ref of certain SST reduces to 0, then in SST evolution table, deletes this node.
Step 505: after generating new SST, adds the node corresponding to new SST in SST evolution table, and allows the node corresponding to old SST (SST deleted in this time squeeze operation) point to new node according to key scope.
The data of above-mentioned Append class formation body merge each K-V under the same key that update can produce by low layer in data by flow process in the process of high level movement
partstructure is polymerized.Further, in above-mentioned merging flow process, also achieve generation and the evolution of SST evolution table, contribute to the reading performance improving Append class formation body.
Finally it should be noted that, above embodiment is only in order to describe technical scheme of the present invention instead of to limit this technical method, the present invention can extend in application other amendment, change, application and embodiment, and therefore think that all such amendments, change, application, embodiment are all in spirit of the present invention and teachings.
Claims (10)
1. set a update method for storage system for LSM, comprise the following steps:
1) key-value structure is configured to and using the key of this new structure-value structure as being inserted into the latest data fragmented storage of key in database by being inserted into value and being inserted into key; For the key-value structure of this new structure, preserve the total amount of data of the data sectional accumulated before being inserted into key, preserve the memory location being inserted into the last data segmentation of key;
2) list item being inserted into key of insertion table is upgraded; The total amount of data of all data sectionals of each key of described insertion table record and the memory location of latest data segmentation.
2. update method according to claim 1, it is characterized in that, described step 1) in: in the key-value structure of described new structure, prevSize field and prevPos field are set, the total amount of data of the data sectional that prevSize field has been accumulated before preserving same key, prevPos field preserves the memory location of the last data segmentation of this key-value structure of same key.
3. update method according to claim 2, is characterized in that, described step 1) comprise substep:
11) search at insertion table and be inserted into key, obtain corresponding list item;
12) according to the list item being inserted into key of gained, obtain and be inserted into the total amount of data of all data sectionals of key and the memory location of latest data segmentation before this update;
13) key will be inserted into and be inserted into value and be configured to key-value structure, step 12 is inserted in the prevSize field of this key-value structure) total amount of data being inserted into all data sectionals of key before this update of obtaining, in the prevPos field of this key-value structure, insert step 12) memory location being inserted into the latest data segmentation of key before this update of obtaining.
4. update method according to claim 3, it is characterized in that, each list item of described insertion table comprises size field and pos field, the total amount of data of all data sectionals of the key of this list item of size field record, the memory location of the latest data segmentation of the key of this list item of pos field record.
5. update method according to claim 4, is characterized in that, described step 2) comprise substep:
21) the size field being inserted into the list item of key is updated to the total amount of data adding all data sectionals after being inserted into value;
22) the pos field being inserted into the list item of key is updated to step 13) memory location of the key-value structure of gained.
6. set the read method of the update load of storage system for LSM for one kind, wherein, described update load comprises at least two keys as the data sectional-value structure belonging to same key, and each key-value structure comprises prevSize field and prevPos field, wherein, the total amount of data of data sectional of prevSize field for having accumulated before preserving this key-value structure under same key, prevPos field is for preserving the memory location of the last data segmentation of this key-value structure under same key; Described read method comprises the following steps:
101) list item of the key that will read is searched at insertion table; The total amount of data of all data sectionals of each key of described insertion table record and the memory location of latest data segmentation;
102) according to the total amount of data of all data sectionals of hitting list item record in insertion table, and the total amount of data of the data sectional accumulated before preserved in each key-value structure of the key that will read, find the key of the value commensurate in scope that will read-value structure and therefrom read the value that will read.
7. read method according to claim 6, is characterized in that, described step 102) comprise substep:
1021) according to the latest data segmentation position of hitting list item record in insertion table, the key of the latest data segmentation of the key that will read-value structure is found to it can be used as current key-value structure;
1022) according to data volume and the prevSize field of current key-value structure own value, judge current key-value structure whether with the value commensurate in scope that will read, if so, then export and read result; If not, then continue to perform step 103);
1023) key-value structure of last data segmentation is found according to the prevPos field of current key-value structure, then using the key of this last data segmentation-value structure as current key-value structure, re-execute step 1022), until all data sectionals of the key that will read are all found.
8. read method according to claim 7, it is characterized in that, described step 1022) in, export after reading result, whether the value scope continuing to judge to read all reads complete, if the judgment is Yes, then directly releases this and reads, if the judgment is No, step 1023 is jumped to).
9. set the merging method of the update load of storage system for LSM for one kind, wherein, described update load comprises at least two keys as the data sectional-value structure belonging to same key, and each key-value structure comprises prevSize field and prevPos field, wherein, the total amount of data of data sectional of prevSize field for having accumulated before preserving this key-value structure under same key, prevPos field is for preserving the memory location of the last data segmentation of this key-value structure under same key; Described merging method comprises: set in the union operation process of storage system at LSM, by belong to described update load same key under each key-value structure merging.
10. merging method according to claim 9, it is characterized in that, to orderly string table file that class formation body the produces starting point as orderly string table file evolution table be inserted, the process of each key-value structure merging under the same key belonging to described update load is comprised the following steps:
201), when carrying out union operation in persistent storage layer, in orderly string table file evolution table, each the orderly string table file participating in this union operation is searched;
202), when reading the structure participating in this union operation, the key-value structure of the different pieces of information segmentation as same key is merged.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510501523.9A CN105138622B (en) | 2015-08-14 | 2015-08-14 | For the insertion operation of LSM tree storage systems and reading and the merging method of load |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510501523.9A CN105138622B (en) | 2015-08-14 | 2015-08-14 | For the insertion operation of LSM tree storage systems and reading and the merging method of load |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105138622A true CN105138622A (en) | 2015-12-09 |
| CN105138622B CN105138622B (en) | 2018-05-22 |
Family
ID=54723970
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510501523.9A Active CN105138622B (en) | 2015-08-14 | 2015-08-14 | For the insertion operation of LSM tree storage systems and reading and the merging method of load |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105138622B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106844650A (en) * | 2017-01-20 | 2017-06-13 | 中国科学院计算技术研究所 | A kind of daily record merges the merging method and system of tree |
| CN108804625A (en) * | 2018-05-31 | 2018-11-13 | 阿里巴巴集团控股有限公司 | A kind of optimization method, device and the computer equipment of LSM trees |
| CN110032565A (en) * | 2019-03-26 | 2019-07-19 | 阿里巴巴集团控股有限公司 | A kind of method, system and electronic equipment generating statistical information |
| CN112395212A (en) * | 2020-11-05 | 2021-02-23 | 华中科技大学 | Method and system for reducing garbage recovery and write amplification of key value separation storage system |
| CN112988208A (en) * | 2019-12-18 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Data updating method, device, equipment and storage medium |
| WO2021197493A1 (en) * | 2020-04-04 | 2021-10-07 | 厦门网宿有限公司 | File management method and apparatus based on lsm-tree storage engine |
| CN113495871A (en) * | 2020-04-04 | 2021-10-12 | 厦门网宿有限公司 | File management method and device based on LSM-Tree storage engine |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722449A (en) * | 2012-05-24 | 2012-10-10 | 中国科学院计算技术研究所 | Key-Value local storage method and system based on solid state disk (SSD) |
| US20140188870A1 (en) * | 2012-12-28 | 2014-07-03 | Dhrubajyoti Borthakur | Lsm cache |
| CN104142958A (en) * | 2013-05-10 | 2014-11-12 | 华为技术有限公司 | Storage method for data in Key-Value system and related device |
| CN104424326A (en) * | 2013-09-09 | 2015-03-18 | 华为技术有限公司 | Data processing method and device |
| CN104424219A (en) * | 2013-08-23 | 2015-03-18 | 华为技术有限公司 | Method and equipment of managing data documents |
| CN104572920A (en) * | 2014-12-27 | 2015-04-29 | 北京奇虎科技有限公司 | Data arrangement method and data arrangement device |
| CN104809237A (en) * | 2015-05-12 | 2015-07-29 | 百度在线网络技术（北京）有限公司 | LSM-tree (The Log-Structured Merge-Tree) index optimization method and LSM-tree index optimization system |
-
2015
- 2015-08-14 CN CN201510501523.9A patent/CN105138622B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722449A (en) * | 2012-05-24 | 2012-10-10 | 中国科学院计算技术研究所 | Key-Value local storage method and system based on solid state disk (SSD) |
| US20140188870A1 (en) * | 2012-12-28 | 2014-07-03 | Dhrubajyoti Borthakur | Lsm cache |
| CN104142958A (en) * | 2013-05-10 | 2014-11-12 | 华为技术有限公司 | Storage method for data in Key-Value system and related device |
| CN104424219A (en) * | 2013-08-23 | 2015-03-18 | 华为技术有限公司 | Method and equipment of managing data documents |
| CN104424326A (en) * | 2013-09-09 | 2015-03-18 | 华为技术有限公司 | Data processing method and device |
| CN104572920A (en) * | 2014-12-27 | 2015-04-29 | 北京奇虎科技有限公司 | Data arrangement method and data arrangement device |
| CN104809237A (en) * | 2015-05-12 | 2015-07-29 | 百度在线网络技术（北京）有限公司 | LSM-tree (The Log-Structured Merge-Tree) index optimization method and LSM-tree index optimization system |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106844650A (en) * | 2017-01-20 | 2017-06-13 | 中国科学院计算技术研究所 | A kind of daily record merges the merging method and system of tree |
| CN108804625A (en) * | 2018-05-31 | 2018-11-13 | 阿里巴巴集团控股有限公司 | A kind of optimization method, device and the computer equipment of LSM trees |
| CN108804625B (en) * | 2018-05-31 | 2020-05-12 | 阿里巴巴集团控股有限公司 | LSM tree optimization method and device and computer equipment |
| TWI710918B (en) * | 2018-05-31 | 2020-11-21 | 開曼群島商創新先進技術有限公司 | An optimization method, device and computer equipment of LSM tree |
| CN110032565A (en) * | 2019-03-26 | 2019-07-19 | 阿里巴巴集团控股有限公司 | A kind of method, system and electronic equipment generating statistical information |
| US11249990B2 (en) | 2019-03-26 | 2022-02-15 | Beijing Oceanbase Technology Co., Ltd. | Database management method and system, and electronic device |
| CN112988208A (en) * | 2019-12-18 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Data updating method, device, equipment and storage medium |
| CN112988208B (en) * | 2019-12-18 | 2023-06-30 | 腾讯科技（深圳）有限公司 | Data updating method, device, equipment and storage medium |
| WO2021197493A1 (en) * | 2020-04-04 | 2021-10-07 | 厦门网宿有限公司 | File management method and apparatus based on lsm-tree storage engine |
| CN113495871A (en) * | 2020-04-04 | 2021-10-12 | 厦门网宿有限公司 | File management method and device based on LSM-Tree storage engine |
| CN113495871B (en) * | 2020-04-04 | 2023-06-23 | 厦门网宿有限公司 | File management method and device based on LSM-Tree storage engine |
| CN112395212A (en) * | 2020-11-05 | 2021-02-23 | 华中科技大学 | Method and system for reducing garbage recovery and write amplification of key value separation storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105138622B (en) | 2018-05-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105138622A (en) | Append operation method for LSM tree memory system and reading and merging method for loads of append operation | |
| CN110879813B (en) | Binary log analysis-based MySQL database increment synchronization implementation method | |
| US9183236B2 (en) | Low level object version tracking using non-volatile memory write generations | |
| US9400816B1 (en) | System for indexing collections of structured objects that provides strong multiversioning semantics | |
| US9037556B2 (en) | Distributed, transactional key-value store | |
| CN105574093A (en) | Method for establishing index in HDFS based spark-sql big data processing system | |
| CN106462592A (en) | Systems and methods to optimize multi-version support in indexes | |
| CN109522271B (en) | Batch insertion and deletion method and device for B + tree nodes | |
| CN104933133A (en) | Metadata snap storage and access method in distributed file system | |
| EP2856345A1 (en) | Meta model driven data export from a database and meta model driven data import to a database | |
| CN105912687A (en) | Mass distributed database memory cell | |
| US20070156778A1 (en) | File indexer | |
| TWI549009B (en) | Database managing method, database managing system, and database tree structure | |
| CN109189759B (en) | Data reading method, data query method, device and equipment in KV storage system | |
| CN102609492A (en) | Metadata management method supporting variable table modes | |
| CN110866024A (en) | Vector database increment updating method and system | |
| US20130290375A1 (en) | Dynamic Partitioning of a Data Structure | |
| CN104516945A (en) | Hadoop distributed file system metadata storage method based on relational data base | |
| CN111694853B (en) | Data increment collection method and device based on lineage, storage medium and electronic equipment | |
| KR102139087B1 (en) | Method, server, and computer readable medium for index recovery using index redo log | |
| CN109614444B (en) | Data initialization method during data synchronization | |
| CN114896271B (en) | Method, device and application for efficiently maintaining node full path | |
| CN105556486A (en) | Method and system for automatic management of dynamically allocated memory in a computing unit | |
| CN110908830A (en) | Method for realizing file system to object storage difference comparison and backup through database | |
| CN113821508B (en) | Method and system for realizing array index |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210301 Address after: Room 1146, 11 / F, research complex building, Institute of computing technology, Chinese Academy of Sciences, No. 6, South Road, Haidian District, Beijing, 100190 Patentee after: Zhongkehai micro (Beijing) Technology Co.,Ltd. Address before: 100190 No. 6 South Road, Zhongguancun Academy of Sciences, Beijing, Haidian District Patentee before: Institute of Computing Technology, Chinese Academy of Sciences |
|
| TR01 | Transfer of patent right |