CN104104613A - Automatic re-equalizing method of cloud storage system - Google Patents
Automatic re-equalizing method of cloud storage system Download PDFInfo
- Publication number
- CN104104613A CN104104613A CN201410388381.5A CN201410388381A CN104104613A CN 104104613 A CN104104613 A CN 104104613A CN 201410388381 A CN201410388381 A CN 201410388381A CN 104104613 A CN104104613 A CN 104104613A
- Authority
- CN
- China
- Prior art keywords
- data
- data page
- hot spot
- back end
- frequency
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses an automatic re-equalizing method of a cloud storage system. The method comprises the steps of (a) saving data in a virtual disk, (b) dividing the virtual disk into a plurality of data blocks, each of which is composed of a plurality of data pages, and putting the data pages on a plurality of different data nodes dispersedly, (c) recording the times and frequency of accessing the data on the data pages, by a metadata server, (d) monitoring the frequency and times of accessing to each data page on each data node by the cloud storage system in real time and continuously, and (e) searching for and determining hot data pages by the metadata server, and determining the data nodes comprising more hot data pages, and shifting one part of the data pages in the data nodes comprising more hot data pages to the data nodes comprising less hot data pages. After the hot data pages are adjusted, the service life of the disk of the hot data cannot be affected due to data saturation and the data nodes do not crash and are high in usability; the performance of the whole cloud storage system is improved.
Description
Technical field
The invention belongs to cloud computing field, be specifically related to the automatic rebalancing method method of cloud storage system.
Background technology
For privately owned cloud or public cloud, cloud storage is all part important in cloud computing architecture.A outstanding cloud storage solution framework can help user realize efficiently, rationally, the use storage resources of automation, be also to weigh a cloud computing architecture whether successful mark greatly.But in the epoch of current this information expansion, the capacity of data and complexity than in the past more any moment all large.Therefore,, in the face of day by day complicated data environment, the cloud storage solution that builds high efficiency, autgmentability and a flexibility just becomes the essential of user's IT infrastructure construction.
Along with the continuous increase of data volume, the security requirement of data is also in continuous increase.Data not only will have enough volume space to go storage, also need to realize carrying out safety backup and the remote disaster tolerance of data.Not only to ensure the fail safe of local data, also will ensure, in the time that great disaster occurs in this locality, can carry out fast quick-recovery by remote backup or remote disaster allowable system.
The fast development of virtual, cloud computing, large data technique has changed the application model of conventional I T; when user is in the time disposing cloud computing; because calculating and the storage of cloud computing data are all carried out on cloud platform; the data in high in the clouds are by even more important; remove the reason of data-privacy and network attack, the data backup of complicated system configuration, the applied environment of multiple users, virtualized application model, huge memory device, magnanimity is also for the data protection in high in the clouds has brought new challenge.
Common solution stores data in physical disk, different back end is in different hard disks, but according to access request, some back end are often accessed, be hot spot data, but some back end basis is fewer accessed, the hard disk of depositing hot spot data node is often accessed, so just more easily damage than the hard disk of common store data node, the life-span is low.Therefore, common solution has two shortcomings: the disk life-span that 1, hot spot data exists is low, causes the back end of depositing hot spot data easily to collapse, and availability is low; 2, access request is many, and the back end of depositing for hot spot data is different with the access rate of general data node, and the performance of whole storage system is low.
Summary of the invention
For above-mentioned the deficiencies in the prior art, the invention provides the automatic rebalancing method method of cloud storage system, while having solved existing cloud storage, systematic function is low, the problem that the disk life-span is little.
To achieve these goals, the technical solution used in the present invention is as follows:
The automatic rebalancing method method of cloud storage system, comprises the steps:
(a) data are placed on virtual disk;
(b) virtual disk is divided into polylith data block, data block is made up of multi-disc data page, and data page is disperseed to be put on multiple different pieces of information nodes;
(c) on meta data server record data page data be subject to access times and frequency;
(d) be interviewed frequency and the number of times of every data page on cloud storage system real-time measurement back end;
(e) meta data server is according to be interviewed frequency and the number of times of every data page on Monitoring Data node in step (d), search and judge hot spot data page, and detect the back end many containing hot spot data page, by the data page adjustment member containing in the many back end of hot spot data page to containing in the less back end of hot spot data page.
In described step (e), described meta data server is searched and is judged that hot spot data page comprises step:
(i1) available back end in meta data server selective system;
(i2) judge the state of be interviewed number of times and the frequency of the data page on back end;
(i3) be interviewed number of times and frequency on data page are weighted, weighting is exactly to consider the ratio share of different variablees in overall, judges the data page that weight is large, is hot spot data page.
Compared with prior art, the present invention has following beneficial effect:
(1) the present invention is placed on data on virtual disk, virtual disk is divided into polylith data block, data block is made up of multi-disc data page, data page is disperseed to be put on multiple different pieces of information nodes, data page identifying recording layer be subject to access times and frequency, be interviewed frequency and the number of times of every data page on system real-time measurement back end, each like this data page will be accessed separately, and the number of times of being interviewed is also easily monitored and record;
(2) meta data server of the present invention is according to be interviewed frequency and the number of times of every data page on cloud storage system real-time measurement back end, search and judge hot spot data page, and detect the back end many containing hot spot data page, by the data page adjustment member containing in the many back end of hot spot data page to containing in the less back end of hot spot data page., hot spot data page is after adjusting, and the disk that hot spot data exists can be because of the saturated life-span that affects disk of data, and back end also can not collapse, and availability is high, and balanced after hot spot data page, and the speed of whole storage system is suitable, and performance improves;
(3) the present invention can also prevent the loss of hot spot data, has very high practical value.
Embodiment
Below in conjunction with embodiment, the invention will be further described, and embodiments of the present invention include but not limited to the following example.
Embodiment
The automatic rebalancing method method of cloud storage system, comprises step:
(a) data are placed on virtual disk;
(b) virtual disk is divided into polylith data block, data block is made up of multi-disc data page, and data page is disperseed to be put on multiple different pieces of information nodes;
(c) on meta data server record data page data be subject to access times and frequency;
(d) be interviewed frequency and the number of times of every data page on cloud storage system real-time measurement back end;
(e) meta data server is according to be interviewed frequency and the number of times of every data page on Monitoring Data node in step (d), search and judge hot spot data page, and detect the back end many containing hot spot data page, by the data page adjustment member containing in the many back end of hot spot data page to containing in the less back end of hot spot data page.
In described step (e), described meta data server is searched and is judged that hot spot data page comprises step:
(i1) available back end in meta data server selective system;
(i2) judge the state of be interviewed number of times and the frequency of the data page on back end;
(i3) be interviewed number of times and frequency on data page are weighted, weighting is exactly to consider the ratio share of different variablees in overall, judges the data page that weight is large, is hot spot data page.
By above-mentioned steps, hot spot data number of pages in all back end is more or less the same and realizes the object that the hot spot data page balancing in whole system distributes.
By above-mentioned setting, hot spot data page is after adjusting, and the disk that hot spot data exists can be because of the saturated life-span that affects disk of data, back end also can not collapse, and availability is high, and balanced after hot spot data page, the speed of whole storage system is suitable, and performance improves.Also prevent to a certain extent the loss of hot spot data.
According to above-described embodiment, just can realize well the present invention.What deserves to be explained is; under prerequisite based on said structure design, for solving same technical problem, even if some that make in the present invention are without substantial change or polishing; the essence of the technical scheme adopting is still the same with the present invention, therefore it also should be in protection scope of the present invention.
Claims (2)
1. the automatic rebalancing method method of cloud storage system, is characterized in that, comprises the steps:
(a) data are placed on virtual disk;
(b) virtual disk is divided into polylith data block, data block is made up of multi-disc data page, and data page is disperseed to be put on multiple different pieces of information nodes;
(c) on meta data server record data page data be subject to access times and frequency;
(d) be interviewed frequency and the number of times of every data page on cloud storage system real-time measurement back end;
(e) meta data server is according to be interviewed frequency and the number of times of every data page on Monitoring Data node in step (d), search and judge hot spot data page, and detect the back end many containing hot spot data page, by the data page adjustment member containing in the many back end of hot spot data page to containing in the less back end of hot spot data page.
2. the automatic rebalancing method method of cloud storage system according to claim 1, is characterized in that, in described step (e), described meta data server is searched and judged that hot spot data page comprises step:
(i1) available back end in meta data server selective system;
(i2) judge the state of be interviewed number of times and the frequency of the data page on back end;
(i3) be interviewed number of times and frequency on data page being weighted, judging the data page that weight is large, is hot spot data page.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410388381.5A CN104104613A (en) | 2014-08-08 | 2014-08-08 | Automatic re-equalizing method of cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410388381.5A CN104104613A (en) | 2014-08-08 | 2014-08-08 | Automatic re-equalizing method of cloud storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104104613A true CN104104613A (en) | 2014-10-15 |
Family
ID=51672421
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410388381.5A Pending CN104104613A (en) | 2014-08-08 | 2014-08-08 | Automatic re-equalizing method of cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104104613A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107783720A (en) * | 2016-08-24 | 2018-03-09 | 深圳市深信服电子科技有限公司 | A kind of data balancing method and storage device |
| CN110018879A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Delay loading method and device applied to distributed system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102104494A (en) * | 2009-12-18 | 2011-06-22 | 华为技术有限公司 | Metadata server, out-of-band network file system and processing method of system |
| WO2011107046A2 (en) * | 2011-04-19 | 2011-09-09 | 华为技术有限公司 | Memory access monitoring method and device |
-
2014
- 2014-08-08 CN CN201410388381.5A patent/CN104104613A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102104494A (en) * | 2009-12-18 | 2011-06-22 | 华为技术有限公司 | Metadata server, out-of-band network file system and processing method of system |
| WO2011107046A2 (en) * | 2011-04-19 | 2011-09-09 | 华为技术有限公司 | Memory access monitoring method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107783720A (en) * | 2016-08-24 | 2018-03-09 | 深圳市深信服电子科技有限公司 | A kind of data balancing method and storage device |
| CN110018879A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Delay loading method and device applied to distributed system |
| CN110018879B (en) * | 2018-01-09 | 2023-06-09 | 阿里巴巴集团控股有限公司 | Delay loading method and device applied to distributed system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6778704B2 (en) | Distributing data on a distributed storage system | |
| Liu et al. | A low-cost multi-failure resilient replication scheme for high-data availability in cloud storage | |
| CN102737127B (en) | Massive data storage method | |
| CN108829350A (en) | Data migration method and device based on block chain | |
| CN105892943B (en) | The access method and system of block storing data in a kind of distributed memory system | |
| CN112084098A (en) | Resource monitoring system and working method | |
| CN103124299A (en) | Distributed block-level storage system in heterogeneous environment | |
| CN103763383A (en) | Integrated cloud storage system and storage method thereof | |
| CN102170460A (en) | Cluster storage system and data storage method thereof | |
| CN103946846A (en) | Use of virtual drive as hot spare for RAID group | |
| CN104011717A (en) | Data selection for data storage backup | |
| CN106599308A (en) | Distributed metadata management method and system | |
| CN104008207B (en) | The data external storage system and date storage method of the database based on CD | |
| CN101986276B (en) | Methods and systems for storing and recovering files and server | |
| CN105404565B (en) | A kind of dual-active data guard method and device | |
| CN102820998B (en) | Realize the dual computer fault-tolerant service system towards office application and date storage method thereof | |
| US20090006501A1 (en) | Zone Control Weights | |
| CN104104613A (en) | Automatic re-equalizing method of cloud storage system | |
| EP3146442B1 (en) | Methods and apparatus for storing electronic documents | |
| CN111708601A (en) | Cloud computing-oriented intelligent storage gateway, system and control method thereof | |
| CN110674080A (en) | Method and system for collecting large-data-volume non-structural files based on NiFi | |
| CN109831502A (en) | Novel website cloud fictitious host computer framework | |
| CN104113439A (en) | Automatic data recovery method of cloud storage system | |
| Guo et al. | Research on data block storage strategy in cloud storage system | |
| Azeem et al. | Support-based replication algorithm for cloud storage systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20141015 |
|
| RJ01 | Rejection of invention patent application after publication |