CN102523258A - Data storage framework facing cloud operation system and load balancing method thereof - Google Patents
Data storage framework facing cloud operation system and load balancing method thereof Download PDFInfo
- Publication number
- CN102523258A CN102523258A CN2011103912246A CN201110391224A CN102523258A CN 102523258 A CN102523258 A CN 102523258A CN 2011103912246 A CN2011103912246 A CN 2011103912246A CN 201110391224 A CN201110391224 A CN 201110391224A CN 102523258 A CN102523258 A CN 102523258A
- Authority
- CN
- China
- Prior art keywords
- data
- metadata
- catalogue
- storage framework
- load
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention relates to the field of cloud computing, in particular to a data storage framework facing a cloud operation system and a load balancing method thereof. A distributing type storage framework is adopted to separate metadata and data, a special metadata server is utilized to store the metadata and process request of the client end to the metadata, and the catalogue is utilized as the smallest operation unit. Catalogue division is conducted on the large catalogue, namely the larger catalogue is divided into catalogues with proper size, and then the Hash mode is adopted to conduct even distribution of load. The framework and the method effectively achieve the effect that the cloud operation system is high in efficiency and stable in storage and can be applied to data storage of the cloud operation system.
Description
Technical field
The present invention relates to the cloud computing field, especially a kind of storage framework and load-balancing method thereof towards cloud operating system.
Background technology
Cloud operating system can be carried out unified management to the hardware resource that comprises processor, storage, network.In current cloud operating system application deployment, storage mainly contains two kinds of purposes: the storage of (1) user interface promptly offers the employed storage of user; (2) the required storage of system, for example virtual machine image storage, current, the storage of cloud operating system exists that performance is not high, autgmentability is bad, unstable, problem such as fail safe is not enough.
Summary of the invention
One of technical problem that the present invention solves is to provide a kind of storage framework towards cloud operating system, can solve availability, the extensibility of metadata service.
Two of the technical problem that the present invention solves is to provide a kind of data load balance method towards cloud operating system, can make full use of the computational resource of meta data server.
The technical scheme that the present invention one of solves the problems of the technologies described above is:
Adopt the distributed storage framework that metadata is separated with data, use special meta data server storing metadata and handle the request of client metadata;
When client desires to obtain certain file data, will at first communicate by letter with meta data server, obtain the metadata of describing this document data, promptly obtain memory location and other information of data at the data server cluster; The data that it is wanted are obtained in client and data server trunking communication then.
Described metadata service system framework has adopted many meta data servers.
Described metadata refers to the data of data of description and environment thereof, refers in particular to the data of description document information or file directory information, and these information comprise file size, deposit position etc.
The metadata store medium is solid state hard disc, high-speed cache.
Two the technical scheme that the present invention solves the problems of the technologies described above is:
With the catalogue is the least unit of operation; To big catalogue, then carry out catalogue and divide, be about to the catalogue that big catalogue is divided into suitable size; Then, the mode of employing Hash is carried out the uniform distribution of load.
To hot spot data, metadata copy mechanism is provided, promptly according to the popularity of file or catalogue, the number of copies of respective numbers is set.
The polynary framework of the present invention has following advantage: (1) no Single Point of Faliure, and when the part meta data server lost efficacy, other meta data servers can be taken over the metadata service, guaranteed the high availability of metadata service; (2) extensibility is good, and storage size needs constantly expansion, possibly expand to PB even EB level memory space, when memory space expands, can corresponding increase meta data server quantity, and to adapt to the rate request that metadata is served.Metadata service application scenarios to cloud operating system storage proposes unique method of on a plurality of meta data servers, carrying out load balancing, divides load, thereby makes full use of the computational resource of meta data server.The advantage that metadata is separated with data is: (1) function is divided clear, and logic is simple; Let data server store data and processes data requests specially, and will storing with process metadata of task is given special server, Each performs its own functions, lets simply clear that processing logic becomes.(2) the metadata service accounts for 30% to 70% of reading and writing data total amount, and the overwhelming majority is the small data quantity random read-write in the metadata read-write, will account for hyperbaric metadata service and data, services and be separated, and can improve processing speed.
Though it is high with the more traditional SATA hard disk of the mode storing metadata cost of solid state hard disc and high-speed cache; But because solid state hard disc does not have seek time and rotational time when reading and writing data; Be particularly suitable for carrying out the reading and writing data of frequent small data quantity; And the data volume of metadata is little, is desirable so adopt the higher solid state hard disc of cost that is fit to application scenarios at key performance point.For further improving the metadata read or write speed, to the focus higher data, adopt the mode of high-speed cache to carry out buffer memory, thereby further improve the reading performance of metadata service.
Description of drawings
Below in conjunction with accompanying drawing the present invention is further specified:
Fig. 1 is a metadata service system configuration diagram of the present invention;
Fig. 2 is a multivariate data server load balancing sketch map of the present invention.
Embodiment
As shown in Figure 1, the present invention has introduced the multivariate data server architecture, and the distributed storage framework separates metadata with data, uses special meta data server to come storing metadata and the request of processing client to metadata.When client desires to obtain certain file data, will at first communicate by letter with meta data server, obtain the metadata of describing this document data, promptly obtain memory location and other information of data at the data server cluster; The data that it is wanted are obtained in client and data server trunking communication then.To accounting up to 30% to 70% metadata flow of services; Propose to introduce a plurality of meta data servers and carry out load balancing; To critical data, adopt the mode of solid state hard disc and high-speed cache to store simultaneously, thereby solve the input and output speed of hot spot data.
Metadata refers to the data of data of description and environment thereof; Under storage background of the present invention, refer in particular to the data of description document information or file directory information, these information comprise: file size, deposit position etc.
The multivariate data server is the core of cloud operating system storage metadata service, is responsible for the response element data service request, carries out the corresponding metadata response, the metadata information of backspace file.
To critical data, the mode that adopts solid state hard disc to add the high speed buffer memory is stored, to improve the input and output speed of hot spot data.
In the metadata service, have the locality requirement usually, for example show the order of All Files information under a certain catalogue: ls, will obtain the fileinfo under a certain catalogue; Simultaneously, load balancing generally requires and can load be shared on the multiple servers through suitable mode.For taking into account this two kinds of requirements, the present invention is the least unit (Fig. 2) of operation with the catalogue, but to big catalogue, also need carry out catalogue and divide, and is about to the catalogue that big catalogue is divided into suitable size; Then; Adopt the mode of Hash to carry out the uniform distribution of load, accomplish that the load that takes into account locality shares, wherein as far as possible; It with the catalogue locality that the least unit of operation can guarantee file metadata under the same catalogue; Simultaneously, the introducing of hash function can be broken up the data allocations of corresponding different directories, thus the uniform distribution of proof load on many meta data servers.Simultaneously, metadata copy mechanism is provided, according to the popularity of file or catalogue, the number of copies of respective numbers is set, with the burst request of reply hot spot data.
Claims (7)
1. the storage framework towards cloud operating system is characterized in that: adopt the distributed storage framework that metadata is separated with data, use special meta data server storing metadata and handle the request of client to metadata;
When client desires to obtain certain file data, will at first communicate by letter with meta data server, obtain the metadata of describing this document data, promptly obtain memory location and other information of data at the data server cluster; The data that it is wanted are obtained in client and data server trunking communication then.
2. storage framework according to claim 1 is characterized in that: described metadata service system framework adopts many meta data servers.
3. storage framework according to claim 1 and 2 is characterized in that: described metadata refers to the data of data of description and environment thereof, refers in particular to the data of description document information or file directory information, and these information comprise file size, deposit position etc.
4. storage framework according to claim 1 and 2 is characterized in that: the metadata store medium is solid state hard disc, high-speed cache.
5. storage framework according to claim 4 is characterized in that: the metadata store medium is solid state hard disc, high-speed cache.
6. the load-balancing method of each storage framework of claim 1-5 is characterized in that: the least unit that with the catalogue is operation; To big catalogue, then carry out catalogue and divide, be about to the catalogue that big catalogue is divided into suitable size; Then, the mode of employing Hash is carried out the uniform distribution of load.
7. load-balancing method according to claim 6 is characterized in that: to hot spot data, metadata copy mechanism is provided, promptly according to the popularity of file or catalogue, the number of copies of respective numbers is set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103912246A CN102523258A (en) | 2011-11-30 | 2011-11-30 | Data storage framework facing cloud operation system and load balancing method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103912246A CN102523258A (en) | 2011-11-30 | 2011-11-30 | Data storage framework facing cloud operation system and load balancing method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102523258A true CN102523258A (en) | 2012-06-27 |
Family
ID=46294047
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2011103912246A Pending CN102523258A (en) | 2011-11-30 | 2011-11-30 | Data storage framework facing cloud operation system and load balancing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102523258A (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
| CN103002027A (en) * | 2012-11-26 | 2013-03-27 | 中国科学院高能物理研究所 | System and method for data storage on basis of key-value pair system tree-shaped directory achieving structure |
| CN103150394A (en) * | 2013-03-25 | 2013-06-12 | 中国人民解放军国防科学技术大学 | Distributed file system metadata management method facing to high-performance calculation |
| CN103685453A (en) * | 2013-09-11 | 2014-03-26 | 华中科技大学 | A method for obtaining metadata in a cloud storage system |
| CN103944997A (en) * | 2014-04-29 | 2014-07-23 | 上海交通大学 | Load balancing method with combination of random sampling and virtualization technology |
| CN104503708A (en) * | 2014-12-29 | 2015-04-08 | 成都致云科技有限公司 | Data hash storage method and device |
| CN104571952A (en) * | 2014-12-25 | 2015-04-29 | 华中科技大学 | Method for separately processing data reading and writing requests and metadata reading and writing requests |
| CN104657115A (en) * | 2015-03-12 | 2015-05-27 | 浪潮集团有限公司 | Cluster file system client-side multi-core concurrence and load implementation method |
| CN106302659A (en) * | 2016-08-02 | 2017-01-04 | 合肥奇也信息科技有限公司 | A kind of based on cloud storage system promotes access data quick storage method |
| CN106326012A (en) * | 2016-08-25 | 2017-01-11 | 中国农业银行股份有限公司 | Web application cluster buffer utilization method and system |
| CN106599102A (en) * | 2016-11-29 | 2017-04-26 | 郑州云海信息技术有限公司 | Metadata performance improvement method based on catalogue splitting mechanism |
| CN107122264A (en) * | 2017-05-15 | 2017-09-01 | 成都优孚达信息技术有限公司 | mass data disaster-tolerant backup method |
| CN109445694A (en) * | 2018-10-19 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of distributed memory system separated from meta-data method and apparatus |
| CN109739439A (en) * | 2018-12-28 | 2019-05-10 | 华北电力科学研究院有限责任公司 | The distributed storage method and system of large capacity energy-storage system mass data |
| US10372370B2 (en) | 2017-06-21 | 2019-08-06 | Western Digital Technologies, Inc. | Metadata load distribution management |
| CN116860564A (en) * | 2023-09-05 | 2023-10-10 | 山东智拓大数据有限公司 | Cloud server data management method and data management device thereof |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080061631A (en) * | 2006-12-28 | 2008-07-03 | (주)포스텍 | Intelligence form home network system |
| CN101854388A (en) * | 2010-05-17 | 2010-10-06 | 浪潮(北京)电子信息产业有限公司 | Method and system concurrently accessing a large amount of small documents in cluster storage |
| CN101866359A (en) * | 2010-06-24 | 2010-10-20 | 北京航空航天大学 | Small file storage and visit method in avicade file system |
| CN102193952A (en) * | 2010-03-19 | 2011-09-21 | 联想(北京)有限公司 | Metadata server, cluster system and file establishing method in cluster system |
-
2011
- 2011-11-30 CN CN2011103912246A patent/CN102523258A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080061631A (en) * | 2006-12-28 | 2008-07-03 | (주)포스텍 | Intelligence form home network system |
| CN102193952A (en) * | 2010-03-19 | 2011-09-21 | 联想(北京)有限公司 | Metadata server, cluster system and file establishing method in cluster system |
| CN101854388A (en) * | 2010-05-17 | 2010-10-06 | 浪潮(北京)电子信息产业有限公司 | Method and system concurrently accessing a large amount of small documents in cluster storage |
| CN101866359A (en) * | 2010-06-24 | 2010-10-20 | 北京航空航天大学 | Small file storage and visit method in avicade file system |
Non-Patent Citations (1)
| Title |
|---|
| 周功业，等: "一种基于对象存储系统的元数据缓存实现方法", 《计算机科学》, vol. 34, no. 10, 15 October 2007 (2007-10-15), pages 146 - 148 * |
Cited By (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102882983B (en) * | 2012-10-22 | 2015-06-10 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
| CN103002027A (en) * | 2012-11-26 | 2013-03-27 | 中国科学院高能物理研究所 | System and method for data storage on basis of key-value pair system tree-shaped directory achieving structure |
| CN103002027B (en) * | 2012-11-26 | 2015-09-02 | 中国科学院高能物理研究所 | Data-storage system and the method for tree directory structure is realized based on key-value pair system |
| CN103150394A (en) * | 2013-03-25 | 2013-06-12 | 中国人民解放军国防科学技术大学 | Distributed file system metadata management method facing to high-performance calculation |
| CN103150394B (en) * | 2013-03-25 | 2014-07-23 | 中国人民解放军国防科学技术大学 | Distributed file system metadata management method facing to high-performance calculation |
| CN103685453A (en) * | 2013-09-11 | 2014-03-26 | 华中科技大学 | A method for obtaining metadata in a cloud storage system |
| CN103685453B (en) * | 2013-09-11 | 2016-08-03 | 华中科技大学 | The acquisition methods of metadata in a kind of cloud storage system |
| CN103944997A (en) * | 2014-04-29 | 2014-07-23 | 上海交通大学 | Load balancing method with combination of random sampling and virtualization technology |
| CN103944997B (en) * | 2014-04-29 | 2015-10-07 | 上海交通大学 | In conjunction with the load-balancing method of random sampling and Intel Virtualization Technology |
| CN104571952A (en) * | 2014-12-25 | 2015-04-29 | 华中科技大学 | Method for separately processing data reading and writing requests and metadata reading and writing requests |
| CN104571952B (en) * | 2014-12-25 | 2017-08-01 | 华中科技大学 | A kind of method for separating processing data and metadata read-write requests |
| CN104503708A (en) * | 2014-12-29 | 2015-04-08 | 成都致云科技有限公司 | Data hash storage method and device |
| CN104503708B (en) * | 2014-12-29 | 2018-05-22 | 成都极驰科技有限公司 | The method and device of data hash storage |
| CN104657115A (en) * | 2015-03-12 | 2015-05-27 | 浪潮集团有限公司 | Cluster file system client-side multi-core concurrence and load implementation method |
| CN104657115B (en) * | 2015-03-12 | 2017-04-19 | 浪潮集团有限公司 | Cluster file system client-side multi-core concurrence and load implementation method |
| CN106302659A (en) * | 2016-08-02 | 2017-01-04 | 合肥奇也信息科技有限公司 | A kind of based on cloud storage system promotes access data quick storage method |
| CN106326012A (en) * | 2016-08-25 | 2017-01-11 | 中国农业银行股份有限公司 | Web application cluster buffer utilization method and system |
| CN106326012B (en) * | 2016-08-25 | 2019-09-24 | 中国农业银行股份有限公司 | Web application cluster caching utilizes method and system |
| CN106599102A (en) * | 2016-11-29 | 2017-04-26 | 郑州云海信息技术有限公司 | Metadata performance improvement method based on catalogue splitting mechanism |
| CN107122264A (en) * | 2017-05-15 | 2017-09-01 | 成都优孚达信息技术有限公司 | mass data disaster-tolerant backup method |
| CN107122264B (en) * | 2017-05-15 | 2020-06-09 | 成都优孚达信息技术有限公司 | Disaster-tolerant backup method for mass data |
| US10372370B2 (en) | 2017-06-21 | 2019-08-06 | Western Digital Technologies, Inc. | Metadata load distribution management |
| CN109445694A (en) * | 2018-10-19 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of distributed memory system separated from meta-data method and apparatus |
| CN109445694B (en) * | 2018-10-19 | 2022-02-18 | 郑州云海信息技术有限公司 | Metadata separation method and device for distributed storage system |
| CN109739439A (en) * | 2018-12-28 | 2019-05-10 | 华北电力科学研究院有限责任公司 | The distributed storage method and system of large capacity energy-storage system mass data |
| CN116860564A (en) * | 2023-09-05 | 2023-10-10 | 山东智拓大数据有限公司 | Cloud server data management method and data management device thereof |
| CN116860564B (en) * | 2023-09-05 | 2023-11-21 | 山东智拓大数据有限公司 | Cloud server data management method and data management device thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102523258A (en) | Data storage framework facing cloud operation system and load balancing method thereof | |
| CN101997918B (en) | Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment | |
| US20130036272A1 (en) | Storage engine node for cloud-based storage | |
| US20140189128A1 (en) | Cluster system with calculation and storage converged | |
| US8930501B2 (en) | Distributed data storage system and method | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN102855294A (en) | Intelligent hash data layout method, cluster storage system and method thereof | |
| CN103530388A (en) | Performance improving data processing method in cloud storage system | |
| CN102521063A (en) | Shared storage method suitable for migration and fault tolerance of virtual machine | |
| US9110820B1 (en) | Hybrid data storage system in an HPC exascale environment | |
| CN106534308B (en) | Method and device for solving data block access hot spot in distributed storage system | |
| CN102523105B (en) | Failure recovery method of data storage and applied data distribution framework thereof | |
| CN101916289A (en) | Method for establishing digital library storage system supporting mass small files and dynamic backup number | |
| US11347413B2 (en) | Opportunistic storage service | |
| CN105516313A (en) | Distributed storage system used for big data | |
| CN104410666A (en) | Method and system for implementing heterogeneous storage resource management under cloud computing | |
| US10057348B2 (en) | Storage fabric address based data block retrieval | |
| Islam et al. | Efficient data access strategies for Hadoop and Spark on HPC cluster with heterogeneous storage | |
| US10606478B2 (en) | High performance hadoop with new generation instances | |
| CN103209219A (en) | Distributed cluster file system | |
| CN101673288A (en) | Method and system for reading and writing files in IPTV system | |
| US11416156B2 (en) | Object tiering in a distributed storage system | |
| CN113472864B (en) | High-performance block chain distributed storage system, method, equipment and storage medium | |
| CN102833295A (en) | Data manipulation method and device in distributed cache system | |
| JP7318899B2 (en) | Systems and methods for storing content items in secondary storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120627 |