CN107943867A - High-performance hierarchical storage system supporting heterogeneous storage - Google Patents
High-performance hierarchical storage system supporting heterogeneous storage Download PDFInfo
- Publication number
- CN107943867A CN107943867A CN201711106687.7A CN201711106687A CN107943867A CN 107943867 A CN107943867 A CN 107943867A CN 201711106687 A CN201711106687 A CN 201711106687A CN 107943867 A CN107943867 A CN 107943867A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- storage
- data block
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims abstract description 101
- 238000007726 management method Methods 0.000 claims abstract description 27
- 238000013500 data storage Methods 0.000 claims abstract description 11
- 238000012544 monitoring process Methods 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 6
- 238000012800 visualization Methods 0.000 claims abstract description 6
- 230000008901 benefit Effects 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 16
- 238000013517 stratification Methods 0.000 claims description 12
- 238000012217 deletion Methods 0.000 claims description 9
- 230000037430 deletion Effects 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 8
- 238000007792 addition Methods 0.000 claims description 6
- 238000011084 recovery Methods 0.000 claims description 6
- 238000009825 accumulation Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 230000014759 maintenance of location Effects 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000013523 data management Methods 0.000 claims description 2
- 238000013508 migration Methods 0.000 description 4
- 230000005012 migration Effects 0.000 description 4
- 230000033228 biological regulation Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000007599 discharging Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013028 medium composition Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/185—Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof
Abstract
The invention provides a high-performance hierarchical storage system supporting heterogeneous storage, which comprises: the file system module is used for providing uniform access to data files on different storage media; the data block storage module is used for dispersedly storing the complete data to different nodes in a data slicing mode; the metadata management module combines the advantages of centralized management and decentralized management and adopts a multi-copy distributed redundant storage method; the storage scheduling module realizes the scheduling of the data copy among all nodes on one hand and realizes the scheduling of the data block among three different storage media, namely an internal memory, an SSD and an HDD on the other hand; and the visualization module is mainly used for providing the service condition of the whole storage system, displaying the real-time monitoring and visualizing operation of dynamic adjustment. The method and the device effectively solve the problem of low efficiency of massive large data access, and improve the data storage and access efficiency of the platform.
Description
Technical field
The present invention relates to a kind of storage system, and in particular, to a kind of high-performance Hierarchical storage for supporting isomery to store
System.
Background technology
Under the demand of magnanimity big data, the data distribution and data access speed of data storage are huge on data processing influence
Greatly, rational big data storage capacity and distribution capability decide data accessibility, and efficient data access capabilities decide
Calculate processing speed.
In image data processing platform, data are mainly using centrally stored, in face of high-resolution, the shadow of big data quantity
As data file, if platform storage processing is inefficient, when being pressed for time, system can not complete the number of scale
According to processing.
The content of the invention
For in the prior art the defects of, the object of the present invention is to provide it is a kind of support isomery store high-performance stratification
Storage system, it carries out the bedding storage of big data by that can be stored using isomery, coordinates rational data hierarchyization storage to calculate
Method, can effectively solve magnanimity big data and access inefficiency problem, lift the data storage and access efficiency of platform.
According to an aspect of the present invention, there is provided a kind of high-performance stratification storage system for supporting isomery to store, it is special
Sign is, including：
File system module, there is provided the unified access of the data file on different storage mediums；File system be responsible for tissue,
Manage, safeguard distributed file system storage all data files, file in file system with master file, catalogue file,
Meta file is stored；
Data block memory module, disperses storage by partial data by data slicer mode and arrives each different node, number
Access speed can be improved in a manner of concurrent according to accessing；Data block memory module carries out data block using fixed length block algorithm
Division；The algorithm carries out cutting using the good block size of justice in advance to file, and carries out the weak strong check value of check value and md5；It is weak
Check value first calculates weak check value and carries out hash lookups, if it find that then counting primarily to the performance of lifting difference coding
Calculate the strong check values of md5 and make further hash and search；, can be effective since weak check value calculation amount is more much smaller than md5
Improve coding efficiency；
Metadata management module, metadata are for recording the corresponding informance of file and data block in file system, first number
According to the advantage of management module combination centralization and decentralization management, using the method for more copy distribution redundant storages；Provide first number
According to more back-up processings and Version Control, to realize the fault-tolerance of metadata and high availability；
Scheduler module is stored, on the one hand it realizes scheduling of the data copy between each node, and on the other hand it realizes number
According to scheduling of the block between memory, SSD and the different storage mediums of tri- kinds of HDD；
Visualization model, mainly provides the service condition of whole storage system, the displaying monitored in real time, and dynamic adjusts
Visualized operation.
Preferably, the file system module includes two submodules again, and file describes submodule and file body submodule；
File describes submodule and records description to file size, type, file mark, access limit etc.；File body submodule is remembered
Record the actual data information of file.
Preferably, the file system module includes following functions：
Document creation, all metadata operations, and the read operation for File Open and write operation, pass through file system
System object returns to file flow object to perform；
File is write, and file system obtains fileinfo, mould is dispatched in conjunction with file system by accessing meta data block
Block, selects data block storage location, and file write data block can be selected the accumulation layer of storage by file system；
File is read, and file metadata can be changed by file system object, or by obtain one by one inlet flow come
Read file.
Preferably, the data block memory module includes following functions：
Data block is distributed, and all files are stored with block, and block memory module is according to the number of file size allocation fixed length
File data storage is carried out according to block, while data block is distributed on different machine nodes；
Data chunk redundancy, in order to ensure the high reliability of data, bottom ensures the more copies of data by data chunk redundancy,
Number of copies can carry out pre-selection setting；
Data block galassing weighs, and during ensureing file system use, data block is evenly distributed, and will not be concentrated because of data block
And data access performance is caused to lose.
Preferably, the metadata management module includes following functions：
Central repositories：It is centrally stored to preserve all metadata, is responsible for Metadata Service when system runs well and accesses；
Distributed redundancy backup：Metadata is carried out according to different nodes in system and the network distance of central repositories standby
Part so that it can recover in time when metadata meets with and destroys；Under extreme case, when some storage medium is damaged,
It can still be backed up by metadata and access data block；
The Version Control of metadata：The old version of certain amount is preserved using tree structure, so that user is to different
Recovery between version, reduce the change of mistake influences caused by system.
Preferably, the allocation schedule of the storage scheduler module between the individual nodes has following three kinds of policy selections：
Greedy allocation strategy：Data block is distributed to the first memory node for having sufficient space；
Maximum residual space allocation policy：Data block is distributed to the memory node for having maximum residual space；
Polling dispatching allocation strategy：Distribution data block selects node to by polling dispatching.
Preferably, the visualization model includes following functions：
Additions and deletions data：Additions and deletions data provide the basic pipes such as upload and the deletion to data at all levels in storage system
Science and engineering is made, and facilitates the operation of user；
System demonstration：System demonstration module shows the use overview of whole system, the loading condition of each storage hierarchy
Deng；
Monitoring management：Monitoring management is the real-time monitoring to the operation conditions of whole system, so as to find in time overload or
Data therein are backed up and migrated by the storage medium that person is damaged；
Storage configuration：Storage configuration is then the storage configuration management module of system, realizes the visual configuration to system,
Easy to the management of cluster；The function of the Version Control of configuration file is provided at the same time, to realize the fast quick-recovery between version.
Compared with prior art, the present invention has following beneficial effect：
(1) since data hierarchy stores, hot spot data copy is preferentially loaded into the high storage of performance, and system is special according to data
Property give full play to isomery storage high-performance.
(2) data dynamic adjusts, and existing distributed storage scheme is all data copy static storage, and data do not migrate,
This programme can realize the dynamic memory of data according to the timeliness of data.
(3) it is data cached using distributed memory, relative to centrally stored, hot spot data only have common copy can including
Deposit or high speed storing, other copies can be persisted to disk or low speed storage, on the one hand accelerate Distributed Calculation, on the one hand carry
High disk utilization.
Brief description of the drawings
Upon reading the detailed description of non-limiting embodiments with reference to the following drawings, further feature of the invention,
Objects and advantages will become more apparent upon：
Fig. 1 is the functional block diagram for the high-performance stratification storage system that the present invention supports isomery storage.
Embodiment
With reference to specific embodiment, the present invention is described in detail.Following embodiments will be helpful to the technology of this area
Personnel further understand the present invention, but the invention is not limited in any way.It should be pointed out that the ordinary skill to this area
For personnel, without departing from the inventive concept of the premise, various modifications and improvements can be made.These belong to the present invention
Protection domain.
As shown in Figure 1, the present invention supports the high-performance stratification storage system of isomery storage to include：
File system module, there is provided the unified access of the data file on different storage mediums.File system be responsible for tissue,
Manage, safeguard distributed file system storage all data files, file in file system with master file, catalogue file,
Meta file is stored.
Data block memory module, disperses storage by partial data by data slicer mode and arrives each different node, number
Access speed can be improved in a manner of concurrent according to accessing.Data block memory module carries out data block using fixed length block algorithm
Division.The algorithm carries out cutting using the good block size of justice in advance to file, and carries out the weak strong check value of check value and md5.It is weak
Check value first calculates weak check value and carries out hash lookups, if it find that then counting primarily to the performance of lifting difference coding
Calculate the strong check values of md5 and make further hash and search., can be effective since weak check value calculation amount is more much smaller than md5
Improve coding efficiency；
Metadata management module, metadata are for recording the corresponding informance of file and data block in file system, first number
According to the advantage of management module combination centralization and decentralization management, using the method for more copy distribution redundant storages.Provide first number
According to more back-up processings and Version Control, to realize the fault-tolerance of metadata and high availability；
Scheduler module is stored, on the one hand it realizes scheduling of the data copy between each node, and on the other hand it realizes number
According to scheduling of the block between memory, SSD and the different storage mediums of tri- kinds of HDD；
Visualization model, mainly provides the service condition of whole storage system, the displaying monitored in real time, and dynamic adjusts
Visualized operation.
File system module includes two submodules again, and file describes submodule and file body submodule；File description
Module records the description to file size, type, file mark, access limit etc.；File body submodule records file
Actual data information.
File system module includes following functions：
Document creation, all metadata operations, and the read operation for File Open and write operation, pass through file system
System object returns to file flow object to perform；
File is write, and file system obtains fileinfo, mould is dispatched in conjunction with file system by accessing meta data block
Block, selects data block storage location, and file write data block can be selected the accumulation layer of storage by file system；
File is read, and file metadata can be changed by file system object, or by obtain one by one inlet flow come
Read file.
Data block memory module includes following functions：
Data block is distributed, and all files are stored with block, and block memory module is according to the number of file size allocation fixed length
File data storage is carried out according to block, while data block is distributed on different machine nodes；
Data chunk redundancy, in order to ensure the high reliability of data, bottom ensures the more copies of data by data chunk redundancy,
Number of copies can carry out pre-selection setting；
Data block galassing weighs, and during ensureing file system use, data block is evenly distributed, and will not be concentrated because of data block
And data access performance is caused to lose.
Metadata management module includes following functions：
Central repositories：It is centrally stored to preserve all metadata, is responsible for Metadata Service when system runs well and accesses；
Distributed redundancy backup：Metadata is carried out according to different nodes in system and the network distance of central repositories standby
Part so that it can recover in time when metadata meets with and destroys；Under extreme case, when some storage medium is damaged,
It can still be backed up by metadata and access data block；
The Version Control of metadata：The old version of certain amount (user can voluntarily be set) is preserved using tree structure,
So that user is the recovery different versions, reduce the change of mistake influences caused by system.
The allocation schedule of storage scheduler module between the individual nodes has three kinds of strategies to select：
(1) greedy allocation strategy：Data block is distributed to the first memory node for having sufficient space；
(2) maximum residual space allocation policy：Data block is distributed to the memory node for having maximum residual space；
(3) polling dispatching allocation strategy：Distribution data block selects node to by polling dispatching.
Visualization model includes following functions：
Additions and deletions data：Additions and deletions data provide the basic pipes such as upload and the deletion to data at all levels in storage system
Science and engineering is made, and facilitates the operation of user；
System demonstration：System demonstration module shows the use overview of whole system, the loading condition of each storage hierarchy
Deng；
Monitoring management：Monitoring management is the real-time monitoring to the operation conditions of whole system, so as to find in time overload or
Data therein are backed up and migrated by the storage medium that person is damaged；
Storage configuration：Storage configuration is then the storage configuration management module of system, realizes the visual configuration to system,
Easy to the management of cluster；The function of the Version Control of configuration file is provided at the same time, to realize the fast quick-recovery between version.
Storage scheduling of the data between memory, SSD and HDD, then mainly to the greatest extent may be used according to data temperature evaluation mechanism
Can be so that the high data copy of cache access frequency as far as possible in memory, SSD preserves the data copy that visiting frequency takes second place, and HDD
Then preserve cold data copy.On the one hand the temperature of data can be set by system for each user, can also upload number in user
According to when specify a kind of temperature.Temperature is divided into 0 to 3 four kinds of ranks, sets the accumulation layer of three copies on different nodes
It is secondary, as shown in table 1.Such a storage strategy can play high access efficiency, and storage is improved while data high reliability is ensured
The utilization rate of capacity.
1 data temperature storage strategy of table
| Temperature rank | Primary copy | From copy | From copy |
| 0 | Memory | SSD | SSD |
| 1 | Memory | SSD | HDD |
| 2 | SSD | HDD | HDD |
| 3 | HDD | HDD | HDD |
Since distributed file system uses the strategy of copy redundancy to realize data high availability, but reality is using
In, it is not that each copy possesses very high accessed property.System realizes the dynamic migration of dsc data copy, hot copy loading
To memory, cold copy is persisted to disk.
Storage migration ensure that data according to the actual requirements or actually use situation, different between a node to store
Migration between level.Generally, the average I/O of some file nearest a period of time can reflect the temperature of this file；But
Even if being that some files infrequently access, but it is needed to be called in time when accessing.So we have provided one to the user
The parameter of a configurable file importance.
If regulation file is vital document, retains a copy and never degrade in memory, if regulation file is time weight
Want file；Then retain a copy to determine whether to move back in SSD, the I/O that is averaged in the recent period according to file；If regulation file is not weigh
Will, then retain copy and determine whether to move back in HDD, the I/O that is averaged in the recent period also according to file.The problem of in view of memory capacity,
Final storage tier is finally determined according to file importance, average I/O and remaining three parameters of disk size.
Determine that the strategy of file level is as follows based on average I/O：
Average temperature approximate formula such as following formula (1)：
I/Oavg=(Anew-Aold)/(ST) ... ... (1)
Wherein, Anew is the statistical value for the nearest I/O activities of file read according to record log, and Aold is earliest statistics
Value, S represent file size, and T represents default time value.
Workflow is as follows：In certain time interval T, the I/Oavg of All Files under the level is calculated, and finds out maximum
Max and minimum value min.Certain I/O threshold values are set, will be stored less than the file destination of this value to migration queue.For sublevel
The file hierarchy of level, sets threshold value of moving back, if more than the value, then puts files into queue of moving back.
The strategy of file level is determined according to disk space：The speed and appearance of tri- memory, SSD and HDD storage hierarchys
Pyramid is presented in amount, in order to solve the problems, such as that the memory space of upper level more easily takes, to memory layers and SSD layers points
Not She Zhi a headspace utilization rate threshold value, maximum Cmax and minimum value Cmin.If practical efficiency is more than Cmax,
File system forces progress file to be moved out, if practical efficiency between Cmin and Cmax, suspends data write operation, it is allowed to
Data are moved out operation；If practical efficiency is less than Cmin, pause data, which are moved out, operates the write-in of permission data.
The preferentially file of never labeled severity level, and select which data to move out in small significance data.
Select the strategy moved out, there is provided following three kinds of modes：Greedy take-back strategy：Arbitrary block is removed until discharging required size
Space.LRU take-back strategies：Remove space of the least recently used data block until discharging required size.Part LRU is returned
Receive strategy：Based on least recently used removal, but selection has the storage catalogue in maximum residual space, only removes number from the catalogue
According to block.
The present invention is the high efficiency storage technologies of a dynamic adjustment, and efficient data-storage system can be built by it.
Data-storage system is by memory, the difference storage medium composition such as SSD and HDD, between different storage mediums by software definition into
Row data are managed collectively.Data-storage system mainly includes file system module, data block memory module, metadata management module
With storage scheduler module, the functions such as data communication and network interface, data management, data storage operations are provided to outside.
The specific embodiment of the present invention is described above.It is to be appreciated that the invention is not limited in above-mentioned
Particular implementation, those skilled in the art can make various deformations or amendments within the scope of the claims, this not shadow
Ring the substantive content of the present invention.
Claims (7)
- A kind of 1. high-performance stratification storage system for supporting isomery to store, it is characterised in that including：File system module, there is provided the unified access of the data file on different storage mediums；File system is responsible for tissue, pipe All data files of distributed file system storage are managed, safeguard, file is in file system with master file, catalogue file, member File is stored；Data block memory module, disperses storage by partial data by data slicer mode and arrives each different node, data are visited Access speed can be improved in a manner of concurrent by asking；Data block memory module draws data block using fixed length block algorithm Point；The algorithm carries out cutting using the good block size of justice in advance to file, and carries out the weak strong check value of check value and md5；Weak school Value is tested primarily to the performance of lifting difference coding, first calculates weak check value and carry out hash lookups, if it find that then calculating The strong check values of md5 simultaneously make further hash lookups；Since weak check value calculation amount is more much smaller than md5, can effectively carry High coding efficiency；Metadata management module, metadata are for recording the corresponding informance of file and data block in file system, metadata pipe The advantage of module combination centralization and decentralization management is managed, using the method for more copy distribution redundant storages；It is more to provide metadata Back-up processing and Version Control, to realize the fault-tolerance of metadata and high availability；Scheduler module is stored, on the one hand it realizes scheduling of the data copy between each node, and on the other hand it realizes data block Scheduling between memory, SSD and the different storage mediums of tri- kinds of HDD；Visualization model, mainly provides the service condition of whole storage system, the displaying monitored in real time, and dynamic adjustment can Operated depending on changing.
- 2. the high-performance stratification storage system according to claim 1 for supporting isomery storage, it is characterised in that the text Part system module includes two submodules again, and file describes submodule and file body submodule；File describes submodule and records Description to file size, type, file mark, access limit etc.；File body submodule record the actual number of file it is believed that Breath.
- 3. the high-performance stratification storage system according to claim 1 for supporting isomery storage, it is characterised in that the text Part system module includes following functions：Document creation, all metadata operations, and the read operation for File Open and write operation, pass through file system pair Performed as returning to file flow object；File is write, and file system obtains fileinfo, in conjunction with file system scheduler module, choosing by accessing meta data block Data block storage location is selected, file write data block can be selected the accumulation layer of storage by file system；File is read, file metadata can be changed by file system object, or is read by obtaining inlet flow one by one File.
- 4. the high-performance stratification storage system according to claim 1 for supporting isomery storage, it is characterised in that the number Include following functions according to block memory module：Data block is distributed, and all files are stored with block, and block memory module is according to the data block of file size allocation fixed length File data storage is carried out, while data block is distributed on different machine nodes；Data chunk redundancy, in order to ensure the high reliability of data, bottom ensures the more copies of data, copy by data chunk redundancy Number can carry out pre-selection setting；Data block galassing weighs, and during ensureing file system use, data block is evenly distributed, and will not be made because data block is concentrated Lost into data access performance.
- 5. the high-performance stratification storage system according to claim 1 for supporting isomery storage, it is characterised in that the member Data management module includes following functions：Central repositories：It is centrally stored to preserve all metadata, is responsible for Metadata Service when system runs well and accesses；Distributed redundancy backup：Metadata is backed up according to different nodes in system and the network distance of central repositories, Enable and recover in time when metadata meets with and destroys；Under extreme case, when some storage medium is damaged, still It can be backed up by metadata and access data block；The Version Control of metadata：The old version of certain amount is preserved using tree structure, so that user is to different versions Between recovery, reduce mistake change influenced caused by system.
- 6. the high-performance stratification storage system according to claim 1 for supporting isomery storage, it is characterised in that described to deposit The allocation schedule of storage scheduler module between the individual nodes has following three kinds of policy selections：Greedy allocation strategy：Data block is distributed to the first memory node for having sufficient space；Maximum residual space allocation policy：Data block is distributed to the memory node for having maximum residual space；Polling dispatching allocation strategy：Distribution data block selects node to by polling dispatching.
- 7. it is according to claim 1 support isomery storage high-performance stratification storage system, it is characterised in that it is described can Include following functions depending on changing module：Additions and deletions data：Additions and deletions data provide the basic management works such as upload and the deletion to data at all levels in storage system Make, facilitate the operation of user；System demonstration：System demonstration module shows the use overview of whole system, loading condition of each storage hierarchy etc.；Monitoring management：Monitoring management is the real-time monitoring to the operation conditions of whole system, so as to find in time overload or by Data therein are backed up and migrated by the storage medium of damage；Storage configuration：Storage configuration is then the storage configuration management module of system, realizes the visual configuration to system, is easy to The management of cluster；The function of the Version Control of configuration file is provided at the same time, to realize the fast quick-recovery between version.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711106687.7A CN107943867B (en) | 2017-11-10 | 2017-11-10 | High-performance hierarchical storage system supporting heterogeneous storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711106687.7A CN107943867B (en) | 2017-11-10 | 2017-11-10 | High-performance hierarchical storage system supporting heterogeneous storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107943867A true CN107943867A (en) | 2018-04-20 |
| CN107943867B CN107943867B (en) | 2021-11-23 |
Family
ID=61934815
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711106687.7A Active CN107943867B (en) | 2017-11-10 | 2017-11-10 | High-performance hierarchical storage system supporting heterogeneous storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107943867B (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109032809A (en) * | 2018-08-13 | 2018-12-18 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Heterogeneous parallel scheduling system based on remote sensing image storage position |
| CN109062505A (en) * | 2018-07-13 | 2018-12-21 | 南瑞集团有限公司 | A kind of write performance optimization method under cache policy write-in layering hardware structure |
| CN109271871A (en) * | 2018-08-22 | 2019-01-25 | 平安科技（深圳）有限公司 | Sample store path generation method, device, computer equipment and storage medium |
| CN109284072A (en) * | 2018-09-28 | 2019-01-29 | 北京航空航天大学 | A kind of self-adapting data dynamic dispatching method and system for heterogeneous storage system |
| CN109714402A (en) * | 2018-12-12 | 2019-05-03 | 胡书恺 | A kind of redundant data acquisition system and its operation application method |
| CN110659249A (en) * | 2019-09-25 | 2020-01-07 | 浪潮电子信息产业股份有限公司 | Metadata subtree migration method, device and equipment and readable storage medium |
| CN111176584A (en) * | 2019-12-31 | 2020-05-19 | 曙光信息产业(北京)有限公司 | Data processing method and device based on hybrid memory |
| CN111309270A (en) * | 2020-03-13 | 2020-06-19 | 清华大学 | Persistent memory key value storage system |
| CN111367639A (en) * | 2020-03-06 | 2020-07-03 | 中国科学技术大学 | I/O scheduling method based on structure perception and oriented to heterogeneous burst data caching |
| CN111400318A (en) * | 2020-03-09 | 2020-07-10 | 北京易华录信息技术股份有限公司 | Method and device for generating scheduling strategy of data storage |
| CN111427844A (en) * | 2020-04-15 | 2020-07-17 | 成都信息工程大学 | Data migration system and method for file hierarchical storage |
| CN111866038A (en) * | 2019-04-25 | 2020-10-30 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
| CN111881107A (en) * | 2020-08-05 | 2020-11-03 | 北京计算机技术及应用研究所 | Distributed storage method supporting mounting of multi-file system |
| CN111970520A (en) * | 2020-08-13 | 2020-11-20 | 北京中电兴发科技有限公司 | Heterogeneous node streaming data distributed storage method |
| CN112948343A (en) * | 2021-03-25 | 2021-06-11 | 兴业数字金融服务（上海）股份有限公司 | Mass content storage system and method based on distributed big data block |
| CN112947860A (en) * | 2021-03-03 | 2021-06-11 | 成都信息工程大学 | Hierarchical storage and scheduling method of distributed data copies |
| WO2023070935A1 (en) * | 2021-10-28 | 2023-05-04 | 华为云计算技术有限公司 | Data storage method and apparatus, and related device |
| CN117056303A (en) * | 2023-10-13 | 2023-11-14 | 中国电子科技集团公司第十五研究所 | Data storage method and device suitable for military operation big data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040225688A1 (en) * | 2003-05-06 | 2004-11-11 | Drew Thaler | File creation API |
| CN105243130A (en) * | 2015-09-29 | 2016-01-13 | 中国电子科技集团公司第三十二研究所 | Text processing system and method for data mining |
| CN106446126A (en) * | 2016-09-19 | 2017-02-22 | 哈尔滨航天恒星数据系统科技有限公司 | Massive space information data storage management method and storage management device |
| CN106682235A (en) * | 2017-01-18 | 2017-05-17 | 济南浪潮高新科技投资发展有限公司 | System and method for isomerous data mapping |
| CN107153588A (en) * | 2017-05-12 | 2017-09-12 | 成都优孚达信息技术有限公司 | data encoding storage method |
| CN107291889A (en) * | 2017-06-20 | 2017-10-24 | 郑州云海信息技术有限公司 | A kind of date storage method and system |
-
2017
- 2017-11-10 CN CN201711106687.7A patent/CN107943867B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040225688A1 (en) * | 2003-05-06 | 2004-11-11 | Drew Thaler | File creation API |
| CN105243130A (en) * | 2015-09-29 | 2016-01-13 | 中国电子科技集团公司第三十二研究所 | Text processing system and method for data mining |
| CN106446126A (en) * | 2016-09-19 | 2017-02-22 | 哈尔滨航天恒星数据系统科技有限公司 | Massive space information data storage management method and storage management device |
| CN106682235A (en) * | 2017-01-18 | 2017-05-17 | 济南浪潮高新科技投资发展有限公司 | System and method for isomerous data mapping |
| CN107153588A (en) * | 2017-05-12 | 2017-09-12 | 成都优孚达信息技术有限公司 | data encoding storage method |
| CN107291889A (en) * | 2017-06-20 | 2017-10-24 | 郑州云海信息技术有限公司 | A kind of date storage method and system |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109062505A (en) * | 2018-07-13 | 2018-12-21 | 南瑞集团有限公司 | A kind of write performance optimization method under cache policy write-in layering hardware structure |
| CN109032809A (en) * | 2018-08-13 | 2018-12-18 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Heterogeneous parallel scheduling system based on remote sensing image storage position |
| CN109271871A (en) * | 2018-08-22 | 2019-01-25 | 平安科技（深圳）有限公司 | Sample store path generation method, device, computer equipment and storage medium |
| CN109284072A (en) * | 2018-09-28 | 2019-01-29 | 北京航空航天大学 | A kind of self-adapting data dynamic dispatching method and system for heterogeneous storage system |
| CN109284072B (en) * | 2018-09-28 | 2020-06-02 | 北京航空航天大学 | Adaptive data dynamic scheduling method and system for heterogeneous storage system |
| CN109714402A (en) * | 2018-12-12 | 2019-05-03 | 胡书恺 | A kind of redundant data acquisition system and its operation application method |
| CN111866038A (en) * | 2019-04-25 | 2020-10-30 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
| CN110659249A (en) * | 2019-09-25 | 2020-01-07 | 浪潮电子信息产业股份有限公司 | Metadata subtree migration method, device and equipment and readable storage medium |
| CN111176584A (en) * | 2019-12-31 | 2020-05-19 | 曙光信息产业(北京)有限公司 | Data processing method and device based on hybrid memory |
| CN111176584B (en) * | 2019-12-31 | 2023-10-31 | 曙光信息产业(北京)有限公司 | Data processing method and device based on hybrid memory |
| CN111367639A (en) * | 2020-03-06 | 2020-07-03 | 中国科学技术大学 | I/O scheduling method based on structure perception and oriented to heterogeneous burst data caching |
| CN111400318A (en) * | 2020-03-09 | 2020-07-10 | 北京易华录信息技术股份有限公司 | Method and device for generating scheduling strategy of data storage |
| CN111400318B (en) * | 2020-03-09 | 2023-09-15 | 北京易华录信息技术股份有限公司 | Method and device for generating scheduling policy of data storage |
| CN111309270A (en) * | 2020-03-13 | 2020-06-19 | 清华大学 | Persistent memory key value storage system |
| CN111427844B (en) * | 2020-04-15 | 2023-05-19 | 成都信息工程大学 | Data migration system and method for file hierarchical storage |
| CN111427844A (en) * | 2020-04-15 | 2020-07-17 | 成都信息工程大学 | Data migration system and method for file hierarchical storage |
| CN111881107A (en) * | 2020-08-05 | 2020-11-03 | 北京计算机技术及应用研究所 | Distributed storage method supporting mounting of multi-file system |
| CN111881107B (en) * | 2020-08-05 | 2022-09-06 | 北京计算机技术及应用研究所 | Distributed storage method supporting mounting of multi-file system |
| CN111970520B (en) * | 2020-08-13 | 2022-04-08 | 北京中电兴发科技有限公司 | Heterogeneous node streaming data distributed storage method |
| CN111970520A (en) * | 2020-08-13 | 2020-11-20 | 北京中电兴发科技有限公司 | Heterogeneous node streaming data distributed storage method |
| CN112947860A (en) * | 2021-03-03 | 2021-06-11 | 成都信息工程大学 | Hierarchical storage and scheduling method of distributed data copies |
| CN112948343A (en) * | 2021-03-25 | 2021-06-11 | 兴业数字金融服务（上海）股份有限公司 | Mass content storage system and method based on distributed big data block |
| WO2023070935A1 (en) * | 2021-10-28 | 2023-05-04 | 华为云计算技术有限公司 | Data storage method and apparatus, and related device |
| CN117056303A (en) * | 2023-10-13 | 2023-11-14 | 中国电子科技集团公司第十五研究所 | Data storage method and device suitable for military operation big data |
| CN117056303B (en) * | 2023-10-13 | 2024-01-16 | 中国电子科技集团公司第十五研究所 | Data storage method and device suitable for military operation big data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107943867B (en) | 2021-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107943867A (en) | High-performance hierarchical storage system supporting heterogeneous storage | |
| US8732217B2 (en) | Using a per file activity ratio to optimally relocate data between volumes | |
| CN103914516B (en) | A kind of method and system of storage system multi-zone supervision | |
| US9110919B2 (en) | Method for quickly identifying data residing on a volume in a multivolume file system | |
| US8909887B1 (en) | Selective defragmentation based on IO hot spots | |
| US9355112B1 (en) | Optimizing compression based on data activity | |
| CN110058822B (en) | Transverse expansion method for disk array | |
| CN100565512C (en) | Eliminate the system and method for redundant file in the document storage system | |
| CN104731864B (en) | A kind of date storage method of magnanimity unstructured data | |
| JP6056769B2 (en) | Storage system, data rebalancing program, and data rebalancing method | |
| CN107844269A (en) | A kind of layering mixing storage system and method based on uniformity Hash | |
| US9612758B1 (en) | Performing a pre-warm-up procedure via intelligently forecasting as to when a host computer will access certain host data | |
| CN103095805A (en) | Cloud storage system of data intelligent and decentralized management | |
| CN107291889A (en) | A kind of date storage method and system | |
| KR20110087192A (en) | Method for placement of virtual volume hot-spots in storage pools using ongoing load measurements and ranking | |
| WO2019085769A1 (en) | Tiered data storage and tiered query method and apparatus | |
| CN103455577A (en) | Multi-backup nearby storage and reading method and system of cloud host mirror image file | |
| CN103929500A (en) | Method for data fragmentation of distributed storage system | |
| JPWO2015015550A1 (en) | Computer system and control method | |
| CN107003814A (en) | Effective metadata in storage system | |
| CN101997911A (en) | Data migration method and system | |
| CN103488685B (en) | Fragmented-file storage method based on distributed storage system | |
| CN102904948A (en) | Super-large-scale low-cost storage system | |
| CN102694828A (en) | Method and apparatus for data access in distributed caching system | |
| CN109522283A (en) | A kind of data de-duplication method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |