CN110188108A - Date storage method, device, system, computer equipment and storage medium - Google Patents
Date storage method, device, system, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110188108A CN110188108A CN201910494495.0A CN201910494495A CN110188108A CN 110188108 A CN110188108 A CN 110188108A CN 201910494495 A CN201910494495 A CN 201910494495A CN 110188108 A CN110188108 A CN 110188108A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- array
- stored
- volatile memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Abstract
The invention discloses a kind of date storage method, device, system, computer equipment and storage medium, wherein method includes: the write request for handling data to be stored in memory, and in memory by data to be stored caching；When the data cached in memory reach the first preset threshold, data to be stored in memory is converted into being stored in the accumulation layer for the matrix storage organization being set in advance in non-volatile memory device with the Multi-layer matrix array of index；When data reach the second preset threshold in non-volatile memory device, processing is merged to the data stored in non-volatile memory device, and is written in lower layer of tree construction of merging.Present invention can ensure that the readwrite performance of system, increases each layer of space size, amplification performance expense is write to handle the solution log-structured tree construction of multilayer to system bring with this, to reduce system fluctuation and the delay that pauses.
Description
Technical field
Technical field of data storage of the present invention, and in particular to a kind of date storage method, device, system, computer equipment
And storage medium.
Background technique
With the high speed development of information technology and popularizing for internet, there is rapid growth in data volume, using clothes
The data storage size of business and the amount of access of data also increase with it, and traditional relevant database is unable to satisfy demand, non-pass
It is that database comes into being.Key assignments storage is a kind of typical storage mode of non-relational database, in key assignments storage system
Data carry out tissue storage according to the form of key-value pair, due to its behavior extension mode extending transversely, can preferably cope with
The needs of big data era now have become one of the main way for storing industry now.
Today, most key assignments storage system all uses log to merge tree (Log-Structured-Merge Tree, LSM-
Tree) storage model, log merge tree framework and propose the algorithm of delay update and batch write-in, and random write is converted to batch
Sequential write, to greatly improve the write performance of system.But inventor has found in R&D process, currently based on LSM
The key assignments storage system of tree still has some problems in traditional DRAM-SSD storage architecture, for example, due to storing equipment
Caused by level possessed by the not reciprocity bring performance bottleneck of the speed of (DRAM and SSD) and LSM-Tree structure itself
Amplification defect excessively is write, leads to system performance fluctuations, pause latency issue of system performance etc. occurs.
Summary of the invention
The invention solves write amplification defect as existing for storage system in the prior art to lead to system performance fluctuations, stop
The problem of delay, to provide a kind of date storage method, device, system, computer equipment and storage medium.
An aspect of of the present present invention provides a kind of date storage method, comprising: handles writing for data to be stored in memory
Enter request, and in the memory by data to be stored caching；It is preset when the data cached in the memory reach first
When threshold value, data to be stored described in the memory is converted into being stored in and presetting with the Multi-layer matrix array of index
In the accumulation layer of the matrix storage organization in the non-volatile memory device, the accumulation layer of the matrix storage organization is to close
And the first floor of tree construction, it is described to merge the data structure that tree construction is key assignments storage；When number in the non-volatile memory device
When according to reaching the second preset threshold, processing is merged to the data stored in the non-volatile memory device, and be written to
In lower layer of the merging tree construction, wherein lower layer of the merging tree construction is arranged in solid state hard disk, described non-volatile
Storage equipment and accumulation layers all in the solid state hard disk composition is described to merge tree construction.
Optionally, data to be stored described in the memory is converted into being stored in the Multi-layer matrix array of index
It is set in advance in the accumulation layer of the matrix storage organization in the non-volatile memory device, comprising: by capable mode by institute
It states data to be stored and is written to the matrix storage organization；For the data of every a line, the corresponding member of each data cell is established
Data cell；As unit of each meta-data unit, constructs the Multi-layer matrix array with index and store.
Optionally, as unit of each meta-data unit, the Multi-layer matrix array with index is constructed, comprising: according to institute
The sequence for stating data cell successively executes following steps, to construct the Multi-layer matrix array with index, wherein the data sheet
Member sorts according to the numerical values recited of key: meta-data unit corresponding to current data unit, building and a upper Multi-layer matrix number
The index relative of the meta-data unit of Zu Zhong top layer, a upper Multi-layer matrix array are to the current character string data
A upper data cell processing after obtained Multi-layer matrix array；The corresponding meta-data unit of the current data unit is added
Enter into a upper Multi-layer matrix array, obtains new Multi-layer matrix array.
Optionally, in the Multi-layer matrix array, the index of each meta-data unit is directed toward key in its downward adjacent layer
Numerical value be more than or equal to the meta-data unit all meta-data units minimum value.
Optionally, after constructing the Multi-layer matrix array with index, further includes: in the Multi-layer matrix array
Array carries out segment processing, and the array of the first preset quantity is formed an array segment.
Optionally, when data reach the second preset threshold in the non-volatile memory device, to described non-volatile
The data stored in storage equipment merge processing, comprising: when the Multi-layer matrix array in non-volatile memory device
Array layer when reaching default number of plies threshold value, by since the bottom of the Multi-layer matrix array by the array of the second preset quantity
Mono- group of Duan Zucheng, and the array segment in each group is merged；The invalid data in each group after removal merging, obtain
Array segment after to merging；Cutting is carried out according to default times of capacity of former array capacity to the array segment after merging, obtains cutting
Array afterwards.
Optionally, processing is merged to the data stored in the non-volatile memory device, comprising: will be described non-easy
All arrays are deposited into the pre-set space marked off in advance in the property lost storage equipment；The pre-set space is indulged according to preset rules
To several column blocks are cut into, the array in each column block is executed respectively and is merged into the behaviour for merging lower layer of tree construction
Make.
Optionally, the pre-set space is longitudinally cut into several column blocks according to preset rules, comprising: be based on the head
Next layer of file extent of layer divides the pre-set space, obtains several described column blocks, the column block is with the first floor
Next layer of file boundaries be boundary, the corresponding column block of next layer each file of the first floor；Wherein, described pre-
If only file identical with next stratose block range of the first floor merges the array in each column block in space.
The another aspect of the embodiment of the present invention provides a kind of data storage device, comprising: cache module, including being used for
The write request of middle processing data to be stored is deposited, and in the memory by data to be stored caching；First writing module,
For when the data cached in the memory reach the first preset threshold, data to be stored described in the memory to be converted into
Multi-layer matrix array with index is stored in the matrix storage organization being set in advance in the non-volatile memory device
In accumulation layer, the accumulation layer of the matrix storage organization is the first floor for merging tree construction, and the merging tree construction is key assignments storage
Data structure；Second writing module, it is right for when data reach the second preset threshold in the non-volatile memory device
The data stored in the non-volatile memory device merge processing, and are written in lower layer of the merging tree construction,
Wherein, it is arranged in solid state hard disk for lower layer of the merging tree construction, the non-volatile memory device and the solid state hard disk
In all accumulation layers constitute the merging tree construction.
The another aspect of the embodiment of the present invention provides a kind of computer equipment, including memory, processor and storage
On a memory and the computer program that can run on a processor, the processor realize institute when executing the computer program
The step of stating date storage method.
The another aspect of the embodiment of the present invention provides a kind of computer readable storage medium, is stored thereon with computer
The step of program, the computer program realizes the data processing method when being executed by processor.
According to embodiments of the present invention, non-volatile by being stored in the data of the first floor in the storage system for merging tree construction
It stores in equipment, and data to be stored is converted into be stored in non-volatile memory device with the Multi-layer matrix array of index
In matrix storage organization accumulation layer in, when non-volatile memory device data reach threshold value, merging be written to merging tree
Other layers except the structure first floor, other layers are arranged in solid state hard disk.Since the L1 layer and knot later of tree construction will be merged
Structure is stored in the solid state hard disk with larger memory space (such as SSD), it is ensured that the readwrite performance of system increases each
The space size of layer writes amplification performance expense with this to handle the solution log-structured tree construction of multilayer to system bring, thus
Reduce system fluctuation and the delay that pauses.
On the other hand, since the read or write speed of non-volatile memory device is greater than solid state hard disk, so that data to be stored exists
During being written to the first floor L0 for merging tree construction from content, the read-write read or write speed of non-volatile memory device relative to
For solid state hard disk, it can directly be write data into from content admittedly closer to the read or write speed of memory so as to reduce
In state hard disk as equipment read or write speed it is not reciprocity caused by performance bottleneck, improve the storage efficiency of data.
Original L0 layers is replaced using the data structure L0-matrix that one kind can quickly carry out data directory lookup, benefit
Access with the random read-write performance acceleration outstanding with NVM of matrix structure index to L0-matrix internal data.
The present invention uses the L0-matrix of larger space, and the space of other levels is also corresponding in proportion in LSM tree expands,
In the case where gross space is fixed, every sheaf space increases, so that total number of plies is reduced, merges number and accordingly reduces, meanwhile, the present invention
Using L0-matrix high speed index search characteristic, avoid traditional structure expand meeting of L0 sheaf space and bring reading expense ask
Topic.
The present invention carries out L0- in merging process using cutting formation column block is carried out to it by column as unit of column block
Original big key value array of coverage of a big data quantity is merged, is changed into the small key value model of multiple intermediate data amounts by matrix
Original primary serious performance inconsistency is changed into repeatedly slight performance inconsistency, carrys out lifting system with this by the merging for enclosing column block
Average behavior and user usage experience.
Detailed description of the invention
It, below will be to specific in order to illustrate more clearly of the specific embodiment of the invention or technical solution in the prior art
Embodiment or attached drawing needed to be used in the description of the prior art be briefly described, it should be apparent that, it is described below
Attached drawing is some embodiments of the present invention, for those of ordinary skill in the art, before not making the creative labor
It puts, is also possible to obtain other drawings based on these drawings.
Fig. 1 is the architecture diagram of data-storage system in the embodiment of the present invention；
Fig. 2 is the flow chart of date storage method in the embodiment of the present invention；
Fig. 3 is that Multi-layer matrix array L0-matrix creates procedure chart in the embodiment of the present invention；
Fig. 4 is L0 layers of overall space figure in LSM tree construction in the embodiment of the present invention；
Fig. 5 is to carry out search procedure figure by L0-matrix segmental structure in the embodiment of the present invention；
Fig. 6 is L0-matrix union operation flow chart in the embodiment of the present invention；
Fig. 7 is union operation flow chart in NVM in L0-matrix in the embodiment of the present invention；
Fig. 8 is column block dicing process figure in the embodiment of the present invention；
Fig. 9 is the schematic diagram of data storage device in the embodiment of the present invention；
Figure 10 is the hardware structural diagram of computer equipment of the embodiment of the present invention.
Specific embodiment
Technical solution of the present invention is clearly and completely described below in conjunction with attached drawing, it is clear that described implementation
Example is a part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill
Personnel's every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
In the description of the present invention, it should be noted that term " first ", " second ", " third " are used for description purposes only,
It is not understood to indicate or imply relative importance.
As long as in addition, the non-structure each other of technical characteristic involved in invention described below different embodiments
It can be combined with each other at conflict.
Before introducing the embodiment of the present invention, first technical term involved in the embodiment of the present invention is explained:
Dynamic random access memory (Dynamic Random Access Memory, referred to as DRAM), it is most commonly seen
Installed System Memory.Data can only be kept for the very short time by DRAM.In order to keep data, DRAM is stored using capacitor, so must
(refresh) must be refreshed once every a period of time, if storage unit is not refreshed, the information of storage will lose.
Solid state hard disk, (Solid State Drive, referred to as SSD), solid state hard disk, solid state hard disk is to use solid-state electronic
Storage chip array and manufactured hard disk.
Non-volatile memory apparatus (Non-volatile Memory, referred to as NVM).
Date storage method described in the embodiment of the present invention is the storage new based on one kind provided by the embodiment of the present invention
System architecture, the storage system framework can be based on existing LSM-Tree structure, pass through what is made improvements.
The memory system architecture that the embodiment of the present invention is based on specifically includes that memory, non-volatile memory device and solid-state
Hard disk.Specifically, memory is used to handle the write request of data to be stored, and caches the data to be stored；It is non-volatile to deposit
Storage equipment be used for configure merge tree mechanism the first floor, and execute emptying buffer operation when, be written described in the memory to
Storing data, wherein described to merge the data structure that tree construction is key assignments storage；Solid state hard disk is for configuring the merging tree knot
Other layers of structure, and when data reach preset threshold in the non-volatile memory device, the non-volatile memories are written
The reading and writing data speed of data after merging in equipment, the non-volatile memory device is greater than the solid state hard disk.It is non-volatile
Property storage equipment and accumulation layers all in solid state hard disk composition is above-mentioned merges tree construction.
The readwrite performance of non-volatile memory device is higher than solid state hard disk, that is to say that read or write speed is higher than solid state hard disk.?
The data structure of storage system is using merging tree construction, such as LSM-Tree structure.The first floor L0 for merging tree construction is arranged non-
In volatile storage devices, other layers are arranged in solid state hard disk.Memory then is used to handle the write request on upper layer.
In the embodiment of the present invention, a kind of optional storage system in the embodiment of the present invention is introduced by taking LSM-Tree structure as an example
System framework.As shown in Figure 1, the storage system be based on memory, NVM with tri- layers of SSD mix storage architecture, it is whole to retain LSM tree
Structure do not change, overall structure is divided into three parts: 1) DRAM: memtable and immutable structure are designed in DRAM
To handle the write request on upper layer；2) NVM: L0 layers are arranged in NVM, a kind of data structure L0- of matrix form is devised
Matrix replaces traditional L0 layer structure；3) SSD: since the space of NVM limits, the structure after the L1 layer of LSM tree remains unchanged
It is stored in SSD.
It should be noted that structure shown in FIG. 1 is only that one kind of the embodiment of the present invention is optional in the embodiment of the present invention
Storage system framework, other are also applied for the technical solution of the application using the storage system for merging tree construction, and the present invention is not
There is improper restriction.
Based on above-mentioned storage system framework, the embodiment of the invention provides a kind of date storage methods, as shown in Fig. 2, should
Method includes:
Step S201, handles the write request of data to be stored in memory, and the data to be stored is buffered in institute
It states in memory.
Memory dram handles the write request on upper layer, and data to be stored writes memtable first, when memtable writes Man Shizhuan
Become immutable, then completes to operate L0 layers in log-structured tree of Flush by immutable, that is to say clearing buffers
Area's operation.When data in EMS memory reaches threshold value, ordered data content in memory is written to the storing framework for merging tree construction
In.
Step S202, when the data cached in the memory reach the first preset threshold, by described in the memory to
Storing data is converted into being stored in and being set in advance in the non-volatile memory device with the Multi-layer matrix array of index
In the accumulation layer of matrix storage organization, the accumulation layer of the matrix storage organization is the first floor for merging tree construction, the merging tree
Structure is the data structure of key assignments storage.
It can be after the data of caching reach the first preset threshold, emptying buffer operation executed, by the memory
Described in data to be stored be written to merge tree construction the first floor.It specifically, will among the above by executing Flush operation
Data to be stored is stored in non-volatile memory device by the data Flush in immutable to first floor L0, such as NVM
In.
Step S203, when data reach the second preset threshold in the non-volatile memory device, to described non-volatile
Property storage equipment in the data that store merge processing, and be written in lower layer of the merging tree construction, wherein the conjunction
And lower layer of tree construction is arranged in solid state hard disk, all accumulation layers in the non-volatile memory device and the solid state hard disk
Constitute the merging tree construction.
It, will when certain layer (accumulation layer in solid state hard disk) data capacity reaches corresponding threshold value in merging tree construction
Trigger to it is next laminated and operation.
Specifically, secondary storage framework is constructed with NVM and SSD, preposition storage equipment of the NVM as SSD is set in NVM
The accumulation layer of the special matrix storage organization of level-one is set as the first floor for merging tree construction, several file storages are set in SSD
The accumulation layer of structure, NVM is built into storage hierarchy all in SSD completely merges tree.
When data volume reaches the second preset threshold in non-volatile memory device, system starts merging process, will be non-easy
After data in the property lost storage equipment merge, it is written in next layer of L1 for merging tree construction, namely be written to solid state hard disk
In.In the embodiment of the present invention, the reading and writing data speed of non-volatile memory device is greater than solid state hard disk, but solid state hard disk is deposited
It stores up capacity and is greater than non-volatile memory device, solid state hard disk is used to store other layers other than L0 layers of the first floor that merge tree construction
Data.
According to embodiments of the present invention, non-volatile by being stored in the data of the first floor in the storage system for merging tree construction
It stores in equipment, and data to be stored is converted into be stored in non-volatile memory device with the Multi-layer matrix array of index
In matrix storage organization accumulation layer in, when non-volatile memory device data reach threshold value, merging be written to merging tree
Other layers except the structure first floor, other layers are arranged in solid state hard disk.Since the L1 layer and knot later of tree construction will be merged
Structure is stored in the solid state hard disk with larger memory space (such as SSD), it is ensured that the readwrite performance of system increases each
The space size of layer writes amplification performance expense with this to handle the solution log-structured tree construction of multilayer to system bring, thus
Reduce system fluctuation and the delay that pauses.
On the other hand, since the read or write speed of non-volatile memory device is greater than solid state hard disk, so that data to be stored exists
During being written to the first floor L0 for merging tree construction from content, the read-write read or write speed of non-volatile memory device relative to
For solid state hard disk, it can directly be write data into from content admittedly closer to the read or write speed of memory so as to reduce
In state hard disk as equipment read or write speed it is not reciprocity caused by performance bottleneck, improve the storage efficiency of data.
For the key assignments storage system of traditional LSM-Tree structure, using the NVM conduct relative to SSD more high speed
L0 layers of storage equipment in LSM tree construction improves the speed that immutable carries out Flush to L0 layers, so that Flush is operated
It can more quickly finish, avoid insertion process that the delay that pauses occurs, handle data I/O to store equipment faster, with
To reduce, device rate is not reciprocity and bring performance bottleneck for this.
In order to quickly carry out data directory lookup and access, data in EMS memory is being written to conjunction by the embodiment of the present invention
And during the first floor L0 of tree construction, there is the Multi-layer matrix array of index, the lookup and visit of Lai Shixian data by constructing
It asks.
In the embodiment of the present invention, data to be stored described in the memory is converted into having index by above-mentioned steps S201
Multi-layer matrix array, be stored in the accumulation layer for the matrix storage organization being set in advance in the non-volatile memory device
In, comprising:
The data to be stored is written to the matrix storage organization by capable mode by step S2011.
It include multiple data cells in every a line of data to be stored, which can be the lattice of character string dimension
Formula.
Step S2012 establishes the corresponding meta-data unit of each data cell for the data of every a line.
Step S2013 is constructed the Multi-layer matrix array with index and is stored as unit of each meta-data unit.
Further, the data cell of each data line can be built into a meta-data unit group, multiple metadata lists
Tuple is built into a two-dimensional array, and all meta-data units carry out storage and management with the two-dimensional array.
Specifically, it by the data to be stored of immutable structure in memory at Flush to L0 layers, is converted into saving data
Data structure, be denoted as table (that is to say above-mentioned data cell), then, in L0 layers with table be superimposed accumulation combination
Form carries out data storage, is denoted as L0-tables, that is to say array of data structure sheaf.By all key assignments in L0-tables
Data establish corresponding meta-data unit, and L0 layers of data management constructs multilayer index array as unit of each metadata
Data structure that is to say Multi-layer matrix array, be denoted as L0-matrix, carry out rapidly access positioning.
In the embodiment of the present invention, as unit of each meta-data unit, the Multi-layer matrix array with index is constructed, specifically
Following steps can be successively executed according to the sequence of the data cell, to construct the Multi-layer matrix array with index, wherein
The data cell sorts according to the numerical values recited of key:
S1, meta-data unit corresponding to current data unit, building and top layer in a upper Multi-layer matrix array
The index relative of meta-data unit, a upper Multi-layer matrix array are the upper data to the current character string data
The Multi-layer matrix array obtained after cell processing.A upper Multi-layer matrix array described here refers to be obtained when the last time writes data
The matrix array arrived, top layer that is to say its L1 layers.
The corresponding meta-data unit of the current data unit is added in a upper Multi-layer matrix array by S2,
Obtain new Multi-layer matrix array.
Specifically, meta-data unit is obtained from orderly immutable data structure obtain orderly meta-data unit
Array；The meta-data unit array of top layer in the building orderly meta-data unit array and the Multi-layer matrix array
Index relative；Meta-data unit array after the sequence is added in the Multi-layer matrix array, new multilayer square is obtained
Battle array array.A kind of example is as shown in Figure 3.
In the embodiment of the present invention, the data structure L0-matrix generation of data directory lookup can be quickly carried out using one kind
For L0 layers original, accelerated with NVM outstanding random read-write performance to L0-matrix internal data using matrix structure index
Access.
For the embodiment of the present invention in the Multi-layer matrix array, the index of each meta-data unit is directed toward its downward adjacent layer
The numerical value of middle key is more than or equal to the minimum value of all meta-data units of the meta-data unit.
Specific example is as shown in figure 4, the index of each meta-data unit is directed toward the adjacent table of its lower layer in the present embodiment
In or key numerical value be more than or equal to the metadata all units in minimum unit.I.e. for each meta-data unit,
The key value of all metadata after the corresponding position of the index of metadata is all larger than equal to the metadata, the institute before position
There is the key value of metadata to be respectively less than the metadata.
Between the adjacent layer of above-mentioned Multi-layer matrix array, the meta-data unit on upper layer has direction according to set principle
The index of lower level metadata unit, the index construct form longitudinal unit relationship, the metadata array with each layer
Lateral cell composition together, building forms the array of matrix structure.
The established principle refers to each meta-data unit, according to certain size rule, is directed toward institute in lower layer
There is the smallest meta-data unit of meta-data unit intermediate value greater than itself, due to the order inside each metadata array,
All units i.e. before lower level metadata unit pointed by each meta-data unit are respectively less than itself, all lists later
Member is all larger than itself.When carrying out data search to the array, can use between the array indexing confirmation each unit
Size relation accelerates search speed to reduce seeking scope.
In the embodiment of the present invention, after constructing the Multi-layer matrix array with index, further includes: to the Multi-layer matrix
Array carries out segment processing by layer, and the array of the first preset quantity is formed an array segment.
It specifically can be and segment processing carried out to the metadata array in Multi-layer matrix array among the above, as shown in figure 5,
Table in L0-matrix structure is segmented, continuous a certain number of table form a section, are arranged as unit of section
Bloom filter come quick garbled data, thus solve L0-matrix structured data level it is excessive caused by system read property
It can problem.
As shown in fig. 6, in the above embodiment of the present invention step S203, to the number stored in the non-volatile memory device
It is handled according to merging, comprising:
Step S2031, when the array layer of the Multi-layer matrix array in non-volatile memory device reaches the default number of plies
When threshold value, the metadata array of the second preset quantity is formed one since the bottom Multi-layer matrix array L0-matrix
Group, when the quantity in L0-matrix reaches certain threshold value by union operation in triggering group.
Step S2032, the invalid data in each group after removal merging, the array segment after being merged.
Step S2033 carries out cutting according to default times of capacity of former array capacity to the array segment after merging, obtains cutting
Array afterwards.Cutting can be carried out according to 150% times of capacity of former table size, row is at new larger capacity table, then group
New section is synthesized, is added in L0-matrix.
In the embodiment of the present invention, by merging metadata array, it is processed into the array of larger capacity, it in this way can be with
The level in index of metadata matrix structure is reduced, thus carrying out that read-write can be reduced during carrying out reading and writing data
Number, to reduce the performance cost of system.
Data in non-volatile memory device are merged, actually the data of the first floor L0 of storage are closed
And handle, in order to reduce the performance inconsistency of system, the average behavior of storage system is promoted, the present embodiment cuts Multi-layer matrix array
The mode for being divided into column block is handled.Correspondingly, processing is merged to the data stored in the non-volatile memory device,
It include: that tables of data all in the non-volatile memory device are deposited into the pre-set space marked off；By the pre-set space
It is longitudinally cut into several column blocks according to preset rules, merging treatment is executed respectively to the tables of data in each column block.
Specifically, as shown in fig. 7, overall space is divided into two portions in non-volatile memory device (being NVM in figure)
Divide group1 and group2；Wherein, group2 that is to say pre-set space among the above.Existing all table are added
Group2 starts union operation.Group2 is longitudinally cut into the similar column block of several sizes according to certain rules, completes to close
And column block in all data will be recovered, the space of recycling is used to store the new data of Flush to L0-matrix；?
During system starts simultaneously, the array table that all Flush enter non-volatile memory device NVM will be added to group1
In, form a new L0-matrix structure.
In the embodiment of the present invention, the pre-set space is longitudinally cut into several column blocks according to preset rules, comprising: base
Divide the pre-set space in next layer of file extent of the first floor, obtain several described column blocks, the column block with
Next layer of file boundaries of the first floor are boundary, the corresponding column block of next layer each file of the first floor；Its
In, only file identical with next stratose block range of the first floor is closed the array in each column block of the pre-set space
And.
Specifically as shown in figure 8, in the present embodiment, it is each to need to follow following principle-for the selection of column block in merging process
The data area of merging process is as small as possible, to guarantee to will not relate to excessive underlying file in merging process.From L1 layers (
Be next layer of first floor L0, be stored in solid state hard disk) range carry out the division of column block, with file in L1 layers
The range of sstable is divided, and the boundary divided every time is the boundary of sstable, i.e. a column block may correspond to multiple
Sstable, and a sstable only corresponds to a column block.Merging each time only can be identical with L0 stratose block range
Sstable is merged, and is avoided repeatedly reading same data space and is caused to write scale-up problem.
The embodiment of the invention also provides a kind of data storage device, which can be used for executing the above-mentioned implementation of the present invention
The date storage method of example, as shown in figure 9, the device includes:
Cache module 901 delays for handling the write request of data to be stored in memory, and by the data to be stored
It deposits in the memory.
First writing module 902, for when the data cached in the memory reach the first preset threshold, will it is described in
Data to be stored described in depositing is converted into the Multi-layer matrix array with index, is stored in and is set in advance in described non-volatile deposit
In the accumulation layer for storing up the matrix storage organization in equipment, the accumulation layer of the matrix storage organization is the first floor for merging tree construction,
It is described to merge the data structure that tree construction is key assignments storage.
Second writing module 903, it is right for when data reach the second preset threshold in the non-volatile memory device
The data stored in the non-volatile memory device merge processing, and are written in lower layer of the merging tree construction,
Wherein, it is arranged in solid state hard disk for lower layer of the merging tree construction, the non-volatile memory device and the solid state hard disk
In all accumulation layers constitute the merging tree construction.
According to embodiments of the present invention, non-volatile by being stored in the data of the first floor in the storage system for merging tree construction
It stores in equipment, and data to be stored is converted into be stored in non-volatile memory device with the Multi-layer matrix array of index
In matrix storage organization accumulation layer in, when non-volatile memory device data reach threshold value, merging be written to merging tree
Other layers except the structure first floor, other layers are arranged in solid state hard disk.Since the L1 layer and knot later of tree construction will be merged
Structure is stored in the solid state hard disk with larger memory space (such as SSD), it is ensured that the readwrite performance of system increases each
The space size of layer writes amplification performance expense with this to handle the solution log-structured tree construction of multilayer to system bring, thus
Reduce system fluctuation and the delay that pauses.
The specific descriptions of data storage device about the embodiment of the present invention may refer to above method embodiment, here not
It repeats again.
The present embodiment also provides a kind of computer equipment, can such as execute the desktop computer of program, rack-mount server,
Blade server, tower server or Cabinet-type server are (including composed by independent server or multiple servers
Server cluster) etc..The computer equipment 20 of the present embodiment includes, but is not limited to: that company can be in communication with each other by system bus
Memory 21, the processor 22 connect, as shown in Figure 10.It should be pointed out that Figure 10 illustrates only the calculating with component 21-22
Machine equipment 20, it should be understood that being not required for implementing all components shown, the implementation that can be substituted is more or less
Component.
In the present embodiment, memory 21 (i.e. readable storage medium storing program for executing) includes flash memory, hard disk, multimedia card, card-type memory
(for example, SD or DX memory etc.), random access storage device (RAM), static random-access memory (SRAM), read-only memory
(ROM), electrically erasable programmable read-only memory (EEPROM), programmable read only memory (PROM), magnetic storage, magnetic
Disk, CD etc..In some embodiments, memory 21 can be the internal storage unit of computer equipment 20, such as the calculating
The hard disk or memory of machine equipment 20.In further embodiments, memory 21 is also possible to the external storage of computer equipment 20
The plug-in type hard disk being equipped in equipment, such as the computer equipment 20, intelligent memory card (Smart Media Card, SMC), peace
Digital (Secure Digital, SD) card, flash card (Flash Card) etc..Certainly, memory 21 can also both include meter
The internal storage unit for calculating machine equipment 20 also includes its External memory equipment.In the present embodiment, memory 21 is commonly used in storage
It is installed on the operating system and types of applications software of computer equipment 20, such as the program of data storage device described in embodiment
Code etc..In addition, memory 21 can be also used for temporarily storing the Various types of data that has exported or will export.
Processor 22 can be in some embodiments central processing unit (Central Processing Unit, CPU),
Controller, microcontroller, microprocessor or other data processing chips.The processor 22 is commonly used in control computer equipment
20 overall operation.In the present embodiment, program code or processing data of the processor 22 for being stored in run memory 21,
Such as operation data storage device, to realize the date storage method of embodiment.
The present embodiment also provides a kind of computer readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory
(for example, SD or DX memory etc.), random access storage device (RAM), static random-access memory (SRAM), read-only memory
(ROM), electrically erasable programmable read-only memory (EEPROM), programmable read only memory (PROM), magnetic storage, magnetic
Disk, CD, server, App are stored thereon with computer program, phase are realized when program is executed by processor using store etc.
Answer function.The computer readable storage medium of the present embodiment storage device for storing data is realized real when being executed by processor
Apply the date storage method of example.
Obviously, the above embodiments are merely examples for clarifying the description, and does not limit the embodiments.It is right
For those of ordinary skill in the art, can also make on the basis of the above description it is other it is various forms of variation or
It changes.There is no necessity and possibility to exhaust all the enbodiments.And it is extended from this it is obvious variation or
It changes among still in the protection scope of the application.
Claims (10)
1. a kind of date storage method characterized by comprising
The write request of data to be stored is handled in memory, and in the memory by data to be stored caching；
When the data cached in the memory reach the first preset threshold, data to be stored described in the memory is converted into
Multi-layer matrix array with index is stored in the matrix storage organization being set in advance in the non-volatile memory device
In accumulation layer, the accumulation layer of the matrix storage organization is the first floor for merging tree construction, and the merging tree construction is key assignments storage
Data structure；
When data reach the second preset threshold in the non-volatile memory device, to being deposited in the non-volatile memory device
The data of storage merge processing, and are written in lower layer of the merging tree construction, wherein lower layer of the merging tree construction
It is arranged in solid state hard disk, the non-volatile memory device and accumulation layers all in the solid state hard disk composition are described to merge tree
Structure.
2. date storage method according to claim 1, which is characterized in that turn data to be stored described in the memory
It changes the Multi-layer matrix array with index into, is stored in the matrix storage knot being set in advance in the non-volatile memory device
In the accumulation layer of structure, comprising:
The data to be stored is written to the matrix storage organization by capable mode；
For the data of every a line, the corresponding meta-data unit of each data cell is established；
As unit of each meta-data unit, constructs the Multi-layer matrix array with index and store.
3. date storage method according to claim 1, which is characterized in that as unit of each meta-data unit, building
Multi-layer matrix array with index, comprising:
Following steps are successively executed according to the sequence of the data cell, to construct the Multi-layer matrix array with index, wherein
The data cell sorts according to the numerical values recited of key:
Meta-data unit corresponding to current data unit, the metadata list of building and top layer in a upper Multi-layer matrix array
The index relative of member, a upper Multi-layer matrix array are the upper data cell processing to the current character string data
The Multi-layer matrix array obtained afterwards；
The corresponding meta-data unit of the current data unit is added in a upper Multi-layer matrix array, is obtained new
Multi-layer matrix array.
4. date storage method according to claim 2, which is characterized in that in the Multi-layer matrix array, Mei Geyuan
The numerical value that the index of data cell is directed toward key in its downward adjacent layer is more than or equal to all meta-data units of the meta-data unit
Minimum value.
5. date storage method according to claim 1, which is characterized in that there is the Multi-layer matrix array of index in building
Later, further includes:
Segment processing is carried out by layer to the array in the Multi-layer matrix array, the array of the first preset quantity is formed into an array
Section.
6. date storage method according to claim 5, which is characterized in that when data in the non-volatile memory device
When reaching the second preset threshold, processing is merged to the data stored in the non-volatile memory device, comprising:
It, will be from described when the array layer of the Multi-layer matrix array in non-volatile memory device reaches default number of plies threshold value
The bottom of Multi-layer matrix array starts to form the array segment of the second preset quantity into a group, and to the array segment in each group into
Row merges；
The invalid data in each group after removal merging, the array segment after being merged；
Cutting is carried out according to default times of capacity of former array capacity to the array segment after merging, the array after obtaining cutting.
7. date storage method according to claim 1, which is characterized in that being stored in the non-volatile memory device
Data merge processing, comprising:
Arrays all in the non-volatile memory device are deposited into the pre-set space marked off in advance；
The pre-set space is longitudinally cut into several column blocks according to preset rules, the array in each column block is executed respectively
It is merged into the operation for merging lower layer of tree construction.
8. date storage method according to claim 7, which is characterized in that indulge the pre-set space according to preset rules
To being cut into several column blocks, comprising:
The pre-set space is divided based on next layer of file extent of the first floor, obtains several described column blocks, it is described
Column block is using next layer of file boundaries of the first floor as boundary, the corresponding column of next layer each file of the first floor
Block；
Wherein, the array in each column block of the pre-set space file identical with next stratose block range of the first floor
It merges.
9. a kind of data storage device characterized by comprising
Cache module is buffered in institute for handling the write request of data to be stored in memory, and by the data to be stored
It states in memory；
First writing module, for when the data cached in the memory reach the first preset threshold, by institute in the memory
Data to be stored is stated to be converted into being stored in the Multi-layer matrix array of index and being set in advance in the non-volatile memory device
In matrix storage organization accumulation layer in, the accumulation layer of the matrix storage organization be merge tree construction the first floor, the conjunction
And tree construction is the data structure of key assignments storage；
Second writing module, for when data reach the second preset threshold in the non-volatile memory device, to described non-
The data stored in volatile storage devices merge processing, and are written in lower layer of the merging tree construction, wherein institute
It states lower layer of tree construction of merging to be arranged in solid state hard disk, the non-volatile memory device is deposited with all in the solid state hard disk
Reservoir constitutes the merging tree construction.
10. a kind of computer equipment, which is characterized in that including memory, processor and store on a memory and can locate
The computer program run on reason device, the processor realize any one of claim 1 to 8 institute when executing the computer program
The step of stating method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910494495.0A CN110188108B (en) | 2019-06-10 | 2019-06-10 | Data storage method, device, system, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910494495.0A CN110188108B (en) | 2019-06-10 | 2019-06-10 | Data storage method, device, system, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110188108A true CN110188108A (en) | 2019-08-30 |
| CN110188108B CN110188108B (en) | 2021-03-02 |
Family
ID=67720849
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910494495.0A Active CN110188108B (en) | 2019-06-10 | 2019-06-10 | Data storage method, device, system, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110188108B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110910975A (en) * | 2019-10-10 | 2020-03-24 | 江苏盖睿健康科技有限公司 | Data processing method and device for medical information |
| CN111221922A (en) * | 2019-12-31 | 2020-06-02 | 苏州浪潮智能科技有限公司 | RocksDB database data writing method and RocksDB database |
| CN111399777A (en) * | 2020-03-16 | 2020-07-10 | 北京平凯星辰科技发展有限公司 | Differentiated key value data storage method based on data value classification |
| CN111858759A (en) * | 2020-07-08 | 2020-10-30 | 平凯星辰(北京)科技有限公司 | HTAP database based on consensus algorithm |
| CN112115126A (en) * | 2020-09-29 | 2020-12-22 | 北京人大金仓信息技术股份有限公司 | Processing method and device for set data types and database management system |
| CN112732191A (en) * | 2021-01-08 | 2021-04-30 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for merging tree merging data based on log structure |
| CN113392089A (en) * | 2021-06-25 | 2021-09-14 | 瀚高基础软件股份有限公司 | Database index optimization method and readable storage medium |
| CN113821177A (en) * | 2021-10-11 | 2021-12-21 | 中山大学 | Storage structure of LSM tree based on NVM and data storage method thereof |
| CN117009090A (en) * | 2023-09-28 | 2023-11-07 | 北京云枢创新软件技术有限公司 | Method for storing information of sub-nodes of design hierarchical tree, electronic equipment and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763310A (en) * | 2008-12-23 | 2010-06-30 | 康佳集团股份有限公司 | Data storage control method |
| CN105224237A (en) * | 2014-05-26 | 2016-01-06 | 华为技术有限公司 | A kind of date storage method and device |
| CN105989129A (en) * | 2015-02-15 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Real-time data statistic method and device |
| CN106021560A (en) * | 2016-05-30 | 2016-10-12 | 浙江工业大学 | Data-flow-oriented construction method for low-time-delay memory B+ tree index |
| US20170032005A1 (en) * | 2015-07-31 | 2017-02-02 | Netapp, Inc. | Snapshot and/or clone copy-on-write |
| CN109522951A (en) * | 2018-11-09 | 2019-03-26 | 上海智瞳通科技有限公司 | A kind of method of environment and the multidimensional information Data acquisition and storage of target |
| CN109828975A (en) * | 2019-03-01 | 2019-05-31 | 深圳市一航网络信息技术有限公司 | A kind of extensive quick account book access system based on block chain |
-
2019
- 2019-06-10 CN CN201910494495.0A patent/CN110188108B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763310A (en) * | 2008-12-23 | 2010-06-30 | 康佳集团股份有限公司 | Data storage control method |
| CN105224237A (en) * | 2014-05-26 | 2016-01-06 | 华为技术有限公司 | A kind of date storage method and device |
| CN105989129A (en) * | 2015-02-15 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Real-time data statistic method and device |
| US20170032005A1 (en) * | 2015-07-31 | 2017-02-02 | Netapp, Inc. | Snapshot and/or clone copy-on-write |
| CN106021560A (en) * | 2016-05-30 | 2016-10-12 | 浙江工业大学 | Data-flow-oriented construction method for low-time-delay memory B+ tree index |
| CN109522951A (en) * | 2018-11-09 | 2019-03-26 | 上海智瞳通科技有限公司 | A kind of method of environment and the multidimensional information Data acquisition and storage of target |
| CN109828975A (en) * | 2019-03-01 | 2019-05-31 | 深圳市一航网络信息技术有限公司 | A kind of extensive quick account book access system based on block chain |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110910975A (en) * | 2019-10-10 | 2020-03-24 | 江苏盖睿健康科技有限公司 | Data processing method and device for medical information |
| CN111221922A (en) * | 2019-12-31 | 2020-06-02 | 苏州浪潮智能科技有限公司 | RocksDB database data writing method and RocksDB database |
| CN111399777A (en) * | 2020-03-16 | 2020-07-10 | 北京平凯星辰科技发展有限公司 | Differentiated key value data storage method based on data value classification |
| CN111399777B (en) * | 2020-03-16 | 2023-05-16 | 平凯星辰(北京)科技有限公司 | Differential key value data storage method based on data value classification |
| CN111858759A (en) * | 2020-07-08 | 2020-10-30 | 平凯星辰(北京)科技有限公司 | HTAP database based on consensus algorithm |
| CN112115126A (en) * | 2020-09-29 | 2020-12-22 | 北京人大金仓信息技术股份有限公司 | Processing method and device for set data types and database management system |
| CN112732191B (en) * | 2021-01-08 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for merging tree merging data based on log structure |
| CN112732191A (en) * | 2021-01-08 | 2021-04-30 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for merging tree merging data based on log structure |
| CN113392089A (en) * | 2021-06-25 | 2021-09-14 | 瀚高基础软件股份有限公司 | Database index optimization method and readable storage medium |
| CN113392089B (en) * | 2021-06-25 | 2023-02-24 | 瀚高基础软件股份有限公司 | Database index optimization method and readable storage medium |
| CN113821177A (en) * | 2021-10-11 | 2021-12-21 | 中山大学 | Storage structure of LSM tree based on NVM and data storage method thereof |
| CN117009090A (en) * | 2023-09-28 | 2023-11-07 | 北京云枢创新软件技术有限公司 | Method for storing information of sub-nodes of design hierarchical tree, electronic equipment and medium |
| CN117009090B (en) * | 2023-09-28 | 2023-12-15 | 北京云枢创新软件技术有限公司 | Method for storing information of sub-nodes of design hierarchical tree, electronic equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110188108B (en) | 2021-03-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110188108A (en) | Date storage method, device, system, computer equipment and storage medium | |
| EP2735978B1 (en) | Storage system and management method used for metadata of cluster file system | |
| CN104899156B (en) | A kind of diagram data storage and querying method towards extensive social networks | |
| CN104809237B (en) | The optimization method and device of LSM-tree index | |
| CN107728937B (en) | Key value pair persistent storage method and system using nonvolatile memory medium | |
| CN109299113B (en) | Range query method with storage-aware mixed index | |
| CN105117415B (en) | A kind of SSD data-updating methods of optimization | |
| US11249664B2 (en) | File system metadata decoding for optimizing flash translation layer operations | |
| US9959054B1 (en) | Log cleaning and tiering in a log-based data storage system | |
| CN107066393A (en) | The method for improving map information density in address mapping table | |
| CN109800185B (en) | Data caching method in data storage system | |
| CN107870970B (en) | A kind of data store query method and system | |
| CN103890856B (en) | Support the removable bit memory of store data inside structure | |
| CN103530387A (en) | Improved method aimed at small files of HDFS | |
| US11061788B2 (en) | Storage management method, electronic device, and computer program product | |
| CN107153707A (en) | A kind of Hash table construction method and system for nonvolatile memory | |
| CN108733306A (en) | A kind of Piece file mergence method and device | |
| CN103365926A (en) | Method and device for storing snapshot in file system | |
| Carniel et al. | A generic and efficient framework for flash-aware spatial indexing | |
| US8732404B2 (en) | Method and apparatus for managing buffer cache to perform page replacement by using reference time information regarding time at which page is referred to | |
| CN106294189B (en) | Memory defragmentation method and device | |
| KR20160121819A (en) | Apparatus for data management based on hybrid memory | |
| CN110515897B (en) | Method and system for optimizing reading performance of LSM storage system | |
| KR20180135390A (en) | Data journaling method for large solid state drive device | |
| CN116204130A (en) | Key value storage system and management method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210910 Address after: 100192 207, floor 2, building C-1, Zhongguancun Dongsheng science and Technology Park, No. 66, xixiaokou Road, Haidian District, Beijing Patentee after: Pingkai star (Beijing) Technology Co., Ltd Address before: 100192 room 413, 4 / F, building 16, 66 xixiaokou Road, Haidian District, Beijing Patentee before: Beijing Pingkai Star Technology Development Co.,Ltd. |
|
| TR01 | Transfer of patent right |