CN102521419A - Hierarchical storage realization method and system - Google Patents
Hierarchical storage realization method and system Download PDFInfo
- Publication number
- CN102521419A CN102521419A CN201110460042XA CN201110460042A CN102521419A CN 102521419 A CN102521419 A CN 102521419A CN 201110460042X A CN201110460042X A CN 201110460042XA CN 201110460042 A CN201110460042 A CN 201110460042A CN 102521419 A CN102521419 A CN 102521419A
- Authority
- CN
- China
- Prior art keywords
- data
- level
- storage
- level memory
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a hierarchical storage realization method and a system. The method comprises the following step: when any of the servers needs to access the stored data, whether the accessed data is existed in an one-level memorizer or not is judged, if not, the data to be accessed is transferred to the one-level memorizer form a two-level memorizer; and the data access of the server is realized through the one-level memorizer. According to the invention, memorizers with different access rates are adopted, the data access is realized by the faster memorizer through data transfer, and the access efficiency of a storage system is improved without affecting the normal storage of the data, thereby optimizing the storage data structure of cluster servers.
Description
Technical field
The present invention relates to computer realm, and especially, relate to a kind of implementation method and system of classification storage.
Background technology
At present; In extensive HPCC; The user is numerous, workload is huge, the demanding situation of whole IO handling capacity owing to existing; Therefore present NFS based on the individual server end can not satisfy user's demand, particularly large-scale data center and the demand that concurrent quantity is big, the IO amount is big greatly.At this moment, the notion of parallel memory system (distributed memory system) has been given birth to regard to application.
Some parallel memory systems have appearred at present; These storage systems can realize metadata and general data (be meant the data outside the former data, for example, file data; These general datas are collectively referred to as data in this article) storage respectively, so that in read-write, improve performance.
For traditional storage system, the user often requires the speed of storage system faster, and the IO performance is higher, and storage system is bigger.But for the supvr, faster, performance is high more, capacity more greatly means that cost is high more, and along with the raising of these indexs, the rising that cost may the order of magnitude.For this reason, the notion of classification storage (HSM) has just been arisen at the historic moment.The purpose of classification storage is can be divided into the two-stage storage a large-scale storage system, and the capacity of one-level storage is less, but Performance And Reliability is very high; The capacity of secondary storage is very big, but Performance And Reliability is then lower.
But,, do not propose how to realize the solution of classification storage as yet for large-scale cluster server parallel memory system.
Summary of the invention
To lacking the problem that large-scale cluster server parallel memory system is lacked the classification storage scheme in the correlation technique; The present invention proposes a kind of implementation method and system of classification storage; Can in the cluster server parallel memory system, realize the classification storage; Effectively improve the access efficiency of storage system, optimized the store data structure of cluster server.
Technical scheme of the present invention is achieved in that
According to an aspect of the present invention, a kind of implementation method of classification storage is provided, has been used for realizing the classification storage of data in the cluster server storage system.
This method comprises: under the situation that any server in a plurality of servers need conduct interviews to the data of storage, judge whether the data of required visit are present in the single-level memory; The data of visit are not present in the single-level memory if desired, and the data that then will visit migrate to single-level memory from second-level storage; Realize of the visit of this server through single-level memory to data.
This method can further comprise: according to indication or predetermined migration strategy with the data migtation of preserving in the single-level memory to second-level storage, and in single-level memory, preserve the metadata that migrates to the data in the second-level storage.
And; This method can further comprise: for the data that migrate to second-level storage from single-level memory; This data metadata corresponding is taken up space in single-level memory, takes up space in second-level storage with these data, adopt in a like fashion and name.
In addition, alternatively, it is one of following that above-mentioned migration strategy comprises:
The user's data amount that belongs to of in single-level memory, preserving surpasses under the situation of first threshold, does not have used data migtation to second-level storage in the predetermined amount of time that this user is preserved in single-level memory;
The data volume of in single-level memory, preserving shared ratio in the single-level memory total volume surpasses under the situation of predetermined ratio threshold value, with not having used data migtation in the predetermined amount of time in the single-level memory to second-level storage.
The realization system of all kinds of a kind of classification storages is provided according to a further aspect in the invention.
This system comprises: a plurality of servers are used for the data of storage are conducted interviews; Single-level memory; Second-level storage; The data search module is used under the situation that any server of a plurality of servers need conduct interviews to the data of storage, judging whether the data of required visit are present in the single-level memory; Transferring module is used for not being present under the situation of single-level memory in the data of needs visit, and the data that needs are visited migrate to single-level memory from second-level storage; The visit execution module is used to make the server of initiating visit that single-level memory is realized data access.
Wherein, the data migtation that transferring module also is used for according to indication or predetermined migration strategy single-level memory being preserved and is preserved the metadata that migrates to the data in the second-level storage in single-level memory to second-level storage.
And for the data that migrate to second-level storage from single-level memory, transferring module also is used for this data metadata corresponding is taken up space at single-level memory, takes up space in second-level storage with these data, adopts in a like fashion and names.
In addition, alternatively, it is one of following that above-mentioned migration strategy comprises:
The user's data amount that belongs to of in single-level memory, preserving surpasses under the situation of first threshold, does not have used data migtation to second-level storage in the predetermined amount of time that this user is preserved in single-level memory;
The data volume of in single-level memory, preserving shared ratio in the single-level memory total volume surpasses under the situation of predetermined ratio threshold value, with not having used data migtation in the predetermined amount of time in the single-level memory to second-level storage.
In addition, the input and output performance of single-level memory is higher than second-level storage.
The present invention has adopted the storer of different access speed; Migration through data by speed faster storer realize the visit of data; Can under the prerequisite that does not influence the data normal storage, effectively improve the access efficiency of storage system, optimize the store data structure of cluster server.
Description of drawings
Fig. 1 is the process flow diagram according to the implementation method of the classification storage of the embodiment of the invention;
Fig. 2 is the block diagram according to the realization system of the classification storage of the embodiment of the invention.
Embodiment
According to embodiments of the invention, a kind of implementation method of classification storage is provided, be used for realizing the classification storage of data in the cluster server storage system.
As shown in Figure 1, the implementation method of storing according to the classification of the embodiment of the invention comprises:
Step S101 under the situation that any server in a plurality of servers need conduct interviews to the data of storage, judges whether the data of required visit are present in the single-level memory;
Step S103, the data of visit are not present in the single-level memory if desired, and the data that then will visit migrate to single-level memory from second-level storage;
Step S105 realizes the visit of this server to data through single-level memory.
Wherein, the input and output performance of single-level memory is higher than the input and output performance of second-level storage, that is to say that single-level memory has access speed faster than second-level storage.
This method may further include: according to indication or predetermined migration strategy with the data migtation of preserving in the single-level memory to second-level storage, and in single-level memory, preserve the metadata that migrates to the data in the second-level storage.
Wherein,, this data metadata corresponding is taken up space in single-level memory, takes up space in second-level storage with these data, adopt in a like fashion and name for the data that migrate to second-level storage from single-level memory.
In addition; Alternatively; It is one of following that above-mentioned migration strategy comprises: the user's data amount that belongs to of in single-level memory, preserving surpasses under the situation of first threshold, does not have used data migtation to second-level storage in the predetermined amount of time that this user is preserved in single-level memory; The data volume of in single-level memory, preserving shared ratio in the single-level memory total volume surpasses under the situation of predetermined ratio threshold value, with not having used data migtation in the predetermined amount of time in the single-level memory to second-level storage.
Certainly; Migration strategy of the present invention is not limited to above-mentioned strategy, as required, multiple migration strategy can be set control the execution of migration (for the data or the data of different types of different ownership; Different strategies can be set), this paper enumerates the strategy of migration no longer one by one.
A kind of realization system of classification storage is provided according to a further aspect in the invention.
As shown in Figure 2, the realization system that stores according to the classification of the embodiment of the invention comprises:
A plurality of servers (are in clearly purpose; Only show 2 servers 1 and 2 among Fig. 2, in practical application, the quantity of server can be more; According to actual needs; May reach hundreds and thousands of even more, other unshowned servers all are arranged in the system of the present invention's proposition in a similar fashion), be used for the data of storage are conducted interviews;
Single-level memory 21;
Second-level storage 22 is connected with single-level memory 21;
Data search module 23 is connected with second-level storage 22 with single-level memory 21, is used under the situation that any server of a plurality of servers need conduct interviews to the data of storage, judging whether the data of required visit are present in the single-level memory 21;
Visit execution module 25 is used to make the server of initiating visit that single-level memory is realized data access.
Wherein, transferring module also can be used for according to indication or predetermined migration strategy with the data migtation of preserving in the single-level memory to second-level storage, and in single-level memory, preserve the metadata that migrates to the data in the second-level storage.
And for the data that migrate to second-level storage from single-level memory, transferring module also is used for this data metadata corresponding is taken up space at single-level memory, takes up space in second-level storage with these data, adopts in a like fashion and names.
In addition, alternatively, it is one of following that migration strategy comprises:
The user's data amount that belongs to of in single-level memory, preserving surpasses under the situation of first threshold, does not have used data migtation to second-level storage in the predetermined amount of time that this user is preserved in single-level memory; The data volume of in single-level memory, preserving shared ratio in the single-level memory total volume surpasses under the situation of predetermined ratio threshold value, with not having used data migtation in the predetermined amount of time in the single-level memory to second-level storage.
In practical application, can set up database for the full content on one-level storage and the secondary storage; Its various attributes are carried out record, and, a policy management system can be set; File is distinguished with owner, catalogue etc., and the difference Provisioning Policy is when certain strategy satisfies; Can send instruction from the trend Migration tools; With the file migration (function of tactful arbitration functions and Migration tools just is equivalent to above-mentioned transferring module) on secondary storage in the one-level storage, the instruction that Migration tools can the response policy management system is stored file and to be moved on the secondary storage from one-level.
After file migration was on the secondary storage, the metadata information of document retaining was answered in the one-level storage, make the one-level storage have identical name space with secondary storage, and the process of moving can be to user transparent.
When data that user capture is moved away, can automatically the file migration of moving on the secondary storage be returned, make the user to visit.
File both can be moved away or moved back to according to the requirement of policy management system; Also can move away and move back to according to user's requirement; The user can inquire the state of certain file; For example, inquire about these data (file) in one-level storage, in secondary storage, move, move to finish or move and fail etc.
In sum; By means of technique scheme of the present invention; Adopt the storer of different access speed; Migration through data by speed faster storer realize the visit of data, can under the prerequisite that does not influence the data normal storage, effectively improve the access efficiency of storage system, optimize the store data structure of cluster server.The present invention can set up the classification storage in multiple storage system, for example, can on cluster parallel memory system parastor100, set up hierarchical stor.
The above is merely preferred embodiment of the present invention, and is in order to restriction the present invention, not all within spirit of the present invention and principle, any modification of being done, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (9)
1. the implementation method of a classification storage is used for realizing in the cluster server storage system classification storage of data, it is characterized in that, comprising:
Under the situation that any server in a plurality of servers need conduct interviews to the data of storage, judge whether the data of required visit are present in the single-level memory;
The data of visit are not present in the said single-level memory if desired, and the data that then will visit migrate to said single-level memory from second-level storage;
Realize of the visit of this server through said single-level memory to data.
2. implementation method according to claim 1 is characterized in that, further comprises:
With in the data migtation of preserving in the said single-level memory to the said second-level storage, and in said single-level memory, preserve the metadata that migrates to the data in the said second-level storage according to indication or predetermined migration strategy.
3. implementation method according to claim 2 is characterized in that, further comprises:
For the data that migrate to said second-level storage from said single-level memory, this data metadata corresponding is taken up space in said single-level memory, takes up space in said second-level storage with these data, adopt in a like fashion and name.
4. implementation method according to claim 1 is characterized in that, it is one of following that said migration strategy comprises:
The user's data amount that belongs to of in said single-level memory, preserving surpasses under the situation of first threshold, does not have used data migtation to said second-level storage in the predetermined amount of time that this user is preserved in said single-level memory;
The data volume of in said single-level memory, preserving shared ratio in said single-level memory total volume surpasses under the situation of predetermined ratio threshold value, with there not being used data migtation to said second-level storage in the predetermined amount of time in the said single-level memory.
5. the realization system of a classification storage is characterized in that, comprising:
A plurality of servers are used for the data of storage are conducted interviews;
Single-level memory;
Second-level storage;
The data search module is used under the situation that any server of said a plurality of servers need conduct interviews to the data of storage, judging whether the data of required visit are present in the said single-level memory;
Transferring module is used for not being present under the situation of said single-level memory in the data of needs visit, and the data that needs are visited migrate to said single-level memory from said second-level storage;
The visit execution module is used to make the server of initiating visit that said single-level memory is realized data access.
6. realization according to claim 5 system; It is characterized in that; In data migtation to the said second-level storage that said transferring module also is used for according to indication or predetermined migration strategy said single-level memory being preserved, and in said single-level memory, preserve the metadata that migrates to the data in the said second-level storage.
7. realization according to claim 6 system; It is characterized in that; For the data that migrate to said second-level storage from said single-level memory; Said transferring module also is used for this data metadata corresponding is taken up space at said single-level memory, takes up space in said second-level storage with these data, adopts in a like fashion and names.
8. realization according to claim 5 system is characterized in that, it is one of following that said migration strategy comprises:
The user's data amount that belongs to of in said single-level memory, preserving surpasses under the situation of first threshold, does not have used data migtation to said second-level storage in the predetermined amount of time that this user is preserved in said single-level memory;
The data volume of in said single-level memory, preserving shared ratio in said single-level memory total volume surpasses under the situation of predetermined ratio threshold value, with there not being used data migtation to said second-level storage in the predetermined amount of time in the said single-level memory.
9. according to each described realization system in the claim 5 to 8, it is characterized in that the input and output performance of said single-level memory is higher than said second-level storage.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110460042XA CN102521419A (en) | 2011-12-31 | 2011-12-31 | Hierarchical storage realization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110460042XA CN102521419A (en) | 2011-12-31 | 2011-12-31 | Hierarchical storage realization method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102521419A true CN102521419A (en) | 2012-06-27 |
Family
ID=46292332
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110460042XA Pending CN102521419A (en) | 2011-12-31 | 2011-12-31 | Hierarchical storage realization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102521419A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103091707A (en) * | 2012-12-31 | 2013-05-08 | 中国石油集团川庆钻探工程有限公司地球物理勘探公司 | Device and method of seismic data hierarchical storage |
| CN103106045A (en) * | 2012-12-20 | 2013-05-15 | 华为技术有限公司 | Data migration method, system and device at host machine end |
| CN103198027A (en) * | 2013-02-27 | 2013-07-10 | 天脉聚源(北京)传媒科技有限公司 | Method and device for storing and providing files |
| CN103324713A (en) * | 2013-06-19 | 2013-09-25 | 北京奇虎科技有限公司 | Data processing method and device in multistage server and data processing system |
| CN104035925A (en) * | 2013-03-04 | 2014-09-10 | 深圳市腾讯计算机系统有限公司 | Data storage method and device and storage system |
| CN104199784A (en) * | 2014-08-20 | 2014-12-10 | 浪潮(北京)电子信息产业有限公司 | Data migration method and device based on hierarchical storage |
| CN104407987A (en) * | 2014-10-30 | 2015-03-11 | 曙光信息产业股份有限公司 | Hierarchical storage method |
| CN104717531A (en) * | 2013-12-11 | 2015-06-17 | 宏正自动科技股份有限公司 | image log storage system and recording method thereof |
| CN104869140A (en) * | 2014-02-25 | 2015-08-26 | 阿里巴巴集团控股有限公司 | Multi-cluster system and method for controlling data storage of multi-cluster system |
| CN105447182A (en) * | 2015-12-11 | 2016-03-30 | 芜湖乐锐思信息咨询有限公司 | Data storage system based on database |
| CN106354431A (en) * | 2016-08-26 | 2017-01-25 | 浪潮(北京)电子信息产业有限公司 | Data storage method and device |
| CN107807798A (en) * | 2017-11-22 | 2018-03-16 | 郑州云海信息技术有限公司 | One kind classification storage performance optimization method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101055544A (en) * | 2006-04-14 | 2007-10-17 | 国际商业机器公司 | Method and apparatus for supporting multiple one-time table access operations in a hierarchical memory setting |
| CN101067822A (en) * | 2006-05-03 | 2007-11-07 | 国际商业机器公司 | Hierarchical storage management of metadata |
| CN101079902A (en) * | 2007-06-29 | 2007-11-28 | 清华大学 | A great magnitude of data hierarchical storage method |
| CN101101563A (en) * | 2007-07-23 | 2008-01-09 | 清华大学 | Migration management based on massive data classified memory system |
| CN101311911A (en) * | 2007-05-23 | 2008-11-26 | 株式会社日立制作所 | Staging memory system and data migration method thereof |
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
-
2011
- 2011-12-31 CN CN201110460042XA patent/CN102521419A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101055544A (en) * | 2006-04-14 | 2007-10-17 | 国际商业机器公司 | Method and apparatus for supporting multiple one-time table access operations in a hierarchical memory setting |
| CN101067822A (en) * | 2006-05-03 | 2007-11-07 | 国际商业机器公司 | Hierarchical storage management of metadata |
| CN101311911A (en) * | 2007-05-23 | 2008-11-26 | 株式会社日立制作所 | Staging memory system and data migration method thereof |
| CN101079902A (en) * | 2007-06-29 | 2007-11-28 | 清华大学 | A great magnitude of data hierarchical storage method |
| CN101101563A (en) * | 2007-07-23 | 2008-01-09 | 清华大学 | Migration management based on massive data classified memory system |
| CN102291450A (en) * | 2011-08-08 | 2011-12-21 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103106045A (en) * | 2012-12-20 | 2013-05-15 | 华为技术有限公司 | Data migration method, system and device at host machine end |
| CN103091707B (en) * | 2012-12-31 | 2016-06-22 | 中国石油集团川庆钻探工程有限公司地球物理勘探公司 | Geological data hierachical storing device and method |
| CN103091707A (en) * | 2012-12-31 | 2013-05-08 | 中国石油集团川庆钻探工程有限公司地球物理勘探公司 | Device and method of seismic data hierarchical storage |
| CN103198027A (en) * | 2013-02-27 | 2013-07-10 | 天脉聚源(北京)传媒科技有限公司 | Method and device for storing and providing files |
| CN104035925B (en) * | 2013-03-04 | 2018-07-17 | 深圳市腾讯计算机系统有限公司 | Date storage method, device and storage system |
| CN104035925A (en) * | 2013-03-04 | 2014-09-10 | 深圳市腾讯计算机系统有限公司 | Data storage method and device and storage system |
| CN103324713B (en) * | 2013-06-19 | 2017-04-12 | 北京奇安信科技有限公司 | Data processing method and device in multistage server and data processing system |
| CN103324713A (en) * | 2013-06-19 | 2013-09-25 | 北京奇虎科技有限公司 | Data processing method and device in multistage server and data processing system |
| CN104717531A (en) * | 2013-12-11 | 2015-06-17 | 宏正自动科技股份有限公司 | image log storage system and recording method thereof |
| CN104717531B (en) * | 2013-12-11 | 2018-03-09 | 宏正自动科技股份有限公司 | image log storage system and recording method thereof |
| CN104869140A (en) * | 2014-02-25 | 2015-08-26 | 阿里巴巴集团控股有限公司 | Multi-cluster system and method for controlling data storage of multi-cluster system |
| CN104869140B (en) * | 2014-02-25 | 2018-05-22 | 阿里巴巴集团控股有限公司 | The method of the data storage of multi-cluster system and control multi-cluster system |
| CN104199784A (en) * | 2014-08-20 | 2014-12-10 | 浪潮(北京)电子信息产业有限公司 | Data migration method and device based on hierarchical storage |
| CN104199784B (en) * | 2014-08-20 | 2017-12-08 | 浪潮(北京)电子信息产业有限公司 | A kind of data migration method and device based on classification storage |
| CN104407987A (en) * | 2014-10-30 | 2015-03-11 | 曙光信息产业股份有限公司 | Hierarchical storage method |
| CN104407987B (en) * | 2014-10-30 | 2018-10-23 | 曙光信息产业股份有限公司 | A kind of classification storage method |
| CN105447182A (en) * | 2015-12-11 | 2016-03-30 | 芜湖乐锐思信息咨询有限公司 | Data storage system based on database |
| CN106354431A (en) * | 2016-08-26 | 2017-01-25 | 浪潮(北京)电子信息产业有限公司 | Data storage method and device |
| CN107807798A (en) * | 2017-11-22 | 2018-03-16 | 郑州云海信息技术有限公司 | One kind classification storage performance optimization method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102521419A (en) | Hierarchical storage realization method and system | |
| CN102332029B (en) | Hadoop-based mass classifiable small file association storage method | |
| CN100541454C (en) | A kind of data cache method and system | |
| CN101854388B (en) | Method and system concurrently accessing a large amount of small documents in cluster storage | |
| CN107168657B (en) | Virtual disk hierarchical cache design method based on distributed block storage | |
| CN102117248A (en) | Caching system and method for caching data in caching system | |
| CN100452046C (en) | Storage method and system for mass file | |
| CN102541985A (en) | Organization method of client directory cache in distributed file system | |
| US20130006993A1 (en) | Parallel data processing system, parallel data processing method and program | |
| CN102136003A (en) | Large-scale distributed storage system | |
| CN107291889A (en) | A kind of date storage method and system | |
| CN103455577A (en) | Multi-backup nearby storage and reading method and system of cloud host mirror image file | |
| CN107436813A (en) | A kind of method and system of meta data server dynamic load leveling | |
| CN101582076A (en) | Data de-duplication method based on data base | |
| CN104317742A (en) | Automatic thin-provisioning method for optimizing space management | |
| CN103916459A (en) | Big data filing and storing system | |
| CN102915340A (en) | Expanded B+ tree-based object file system | |
| CN104536903A (en) | Mixed storage method and system for conducting classified storage according to data attributes | |
| CN103077197A (en) | Data storing method and device | |
| CN104462389A (en) | Method for implementing distributed file systems on basis of hierarchical storage | |
| CN109800185A (en) | A kind of data cache method in data-storage system | |
| CN104679442A (en) | Method and device for improving performance of disk array | |
| CN103761059A (en) | Multi-disk storage method and system for mass data management | |
| CN107832423A (en) | A kind of file read/write method for distributed file system | |
| CN102937964A (en) | Intelligent data service method based on distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120627 |