CN106527995A - Data expansion and migration method for I/O equilibrium - Google Patents
Data expansion and migration method for I/O equilibrium Download PDFInfo
- Publication number
- CN106527995A CN106527995A CN201611029235.9A CN201611029235A CN106527995A CN 106527995 A CN106527995 A CN 106527995A CN 201611029235 A CN201611029235 A CN 201611029235A CN 106527995 A CN106527995 A CN 106527995A
- Authority
- CN
- China
- Prior art keywords
- data block
- data
- storage system
- migrated
- dilatation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A data expansion and migration method for I/O equilibrium relates to the field of computers and comprises twelve steps. The beneficial effects of the method provided by the present invention are as follows: the method enables hottest I/O data blocks to be uniformly distributed in all storage nodes, so as to achieve a smallest I/O data migration amount; and the best user experience is realized, that is a quick I/O user response.
Description
Technical field
The invention belongs to computer network and Computer Storage field, data expand in a balanced way to relate more specifically to a kind of I/O
Hold moving method.
Background technology
With depth integration of the network technology in social all trades and professions and development, enterprise data center's storage system faces magnanimity
The demand of data storage, thus the capacity of existing storage system can not meet growing data scale and capacity is proposed more
High request, in order to lift the capacity of data center's storage system, the volume space for expanding existing storage system must be adopted
Technology, i.e. capacity-enlargement technology.During storage cluster data dilatation, existing dilatation strategy mainly considers that new and old each disk is total
The equiblibrium mass distribution of data block number and the Data Migrating Strategy implemented, i.e., be simply uniformly distributed in dilatation by all data blocks
In new storage system afterwards.
However, at real Enterprise Data center, there is locality characteristic in data access, even if thus data volume each
Disk balanced distribution can not guarantee the harmony that storage system user I/O is accessed, and unbalanced I/O access will seriously shadow
Ring the readwrite performance of storage system.
The content of the invention
For above-mentioned technical problem the invention provides a kind of I/O data dilatation moving methods in a balanced way, the method exists
The uniform layout of I/O data blocks, low I/O Data Migration amounts after the dilation process of RAID0, can be realized, and is realized fast
The subscriber response time of speed.The present invention is achieved by the following technical solutions.
A kind of I/O data dilatation moving methods in a balanced way, comprise the following steps：
The first step：Initialization memory node, obtains existing storage system operational factor, node number, user data capacity and deposits
Storage system life parameter, and memory node each data block average daily access times are reset；
Second step：Monitoring memory capacity change, monitors existing storage system volume change situation, and obtain amount of user data and
Existing storage system residue free memory capacity；
3rd step：The daily I/O access times of each data block in memory node are counted, and calculates existing average daily I/O and deposited
Take number of times；
4th step：Whether data dilatation, according to second step, monitors existing storage system volume change situation, determines storage system
Whether need to implement dilatation, next step performed if amount of user data reaches the 90% of total memory capacity and implements data dilatation,
Otherwise return to step second step continues monitoring memory capacity situation of change；
5th step：According to total amount of user data, it is determined that new disk number is being added needed for dilation process, available data is in dilatation
Shared capacity ratio 40%～50% in new storage system afterwards,；
6th step：The new disk for adding is loaded, according to the new disk number of addition for determining；
7th step：According to the average daily I/O access times of data block, the data block for selecting front 30% is most hot data block；
8th step：According to the most dsc data block number for determining, its average block number in existing each memory node is calculated, in every node
Then it is defined as data block to be migrated more than the most hot data block of mean value；
9th step：According to data block to be migrated, the visitation frequency of non-migrating data block in single piece of transition process is calculated；According to true
Data block to be migrated in fixed memory node, calculates the visitation frequency of non-migrating data block during each data block migration；
Tenth step：Preferentially the non-migrating data block of mobile access frequency highest was migrated according to individual data block in new disk
The visitation frequency of the non-migrating data block calculated in journey, the non-migrating data block of preferential mobile access frequency highest is to new disk
In；
11st step：Whether data have migrated, and according to data block number to be migrated, hold if data block to be migrated movement is completed
Row next step, otherwise the 9th step of return to step continue to move to data block to be migrated；
12nd step：Whether monitoring storage system life cycle terminates, and completes if terminating, and otherwise returns the first step and continues prison
Survey the dilatation migration of storage system.
The invention discloses a kind of I/O data dilatation moving methods in a balanced way, its general principle is as shown in Fig. 2 storage system
Before system implements Data Migration, be stored with old disk storage node user data, according to every balance of each I/O user data blocks
Access times, are divided into dsc data data block respectively（Red block）And cold data（White blocks）, wherein hot data block is daily
User's visitation frequency highest data block, thus the I/O of system accesses major part and has focused on the access to hot data block, because
This basic thought of the invention is exactly only to migrate hot data block in old disk so as to after hot data block is uniformly distributed in dilatation
All memory nodes in, and the nearest visitation frequency highest data block of priority migration user during Data Migration,
It is achieved thereby that the uniform layout of dsc data block I/O and good user response.
The present invention has the advantages that：
（1）The uniform data layouts of I/O：Because there is the temporal locality and spatial locality of user I/O access in storage system,
By counting the average frequency that I/O is accessed daily of each data block, the data block that its forward part is frequently accessed is selected as to be migrated
Data block, make which uniformly be distributed in each memory node after dilatation, present invention achieves most hot I/O data blocks can be uniform
Be distributed in all memory nodes；
（2）Low I/O Data Migration amounts：During data dilatation, the partial data block of most frequent access has been only selected that
As data block to be migrated, therefore present invention achieves low I/O Data Migration amounts；
（3）Fast I/O subscriber response times：During Data Migration, during often migrating a data block, count each respectively
The Average visits of data block to be migrated, when migrating every time, all only migrate I/O user's visitation frequency highest data block,
Thus present invention achieves optimal Consumer's Experience, i.e., fast I/O subscriber response times.
Description of the drawings
Fig. 1 is a kind of I/O of the invention data dilatation moving method flow charts in a balanced way.
Fig. 2 is a kind of I/O of the invention data dilatation moving method structure charts in a balanced way.
Specific embodiment
Embodiment 1：The present invention is described in further detail below in conjunction with the accompanying drawings
As shown in figure 1, a kind of I/O of the invention data dilatation moving methods in a balanced way, including following implementation steps：
The first step：Initialization memory node, obtains existing storage system operational factor, node number, user data capacity and deposits
Storage system life parameter, and memory node each data block average daily access times are reset；
Second step：Monitoring memory capacity change, monitors existing storage system volume change situation, and obtain amount of user data and
Existing storage system residue free memory capacity；
3rd step：The daily I/O access times of each data block in memory node are counted, and calculates existing average daily I/O and deposited
Take number of times；
4th step：Whether data dilatation, according to second step, monitors existing storage system volume change situation, determines storage system
Whether need to implement dilatation, next step performed if amount of user data reaches the 90% of total memory capacity and implements data dilatation,
Otherwise return to step second step continues monitoring memory capacity situation of change；
5th step：According to total amount of user data, it is determined that new disk number is being added needed for dilation process, available data is in dilatation
Shared capacity ratio 40%～50% in new storage system afterwards,；
6th step：The new disk for adding is loaded, according to the new disk number of addition for determining；
7th step：According to the average daily I/O access times of data block, the data block for selecting front 30% is most hot data block；
8th step：According to the most dsc data block number for determining, its average block number in existing each memory node is calculated, in every node
Then it is defined as data block to be migrated more than the most hot data block of mean value；
9th step：According to data block to be migrated, the visitation frequency of non-migrating data block in single piece of transition process is calculated；According to true
Data block to be migrated in fixed memory node, calculates the visitation frequency of non-migrating data block during each data block migration；
Tenth step：Preferentially the non-migrating data block of mobile access frequency highest was migrated according to individual data block in new disk
The visitation frequency of the non-migrating data block calculated in journey, the non-migrating data block of preferential mobile access frequency highest is to new disk
In；
11st step：Whether data have migrated, and according to data block number to be migrated, hold if data block to be migrated movement is completed
Row next step, otherwise the 9th step of return to step continue to move to data block to be migrated；
12nd step：Whether monitoring storage system life cycle terminates, and completes if terminating, and otherwise returns the first step and continues prison
Survey the dilatation migration of storage system.
It is so far, of the present invention that for a kind of I/O, data dilatation moving method implementation steps all terminate in a balanced way.
The present invention is not only limited to above-mentioned specific embodiment, and persons skilled in the art are according to disclosed by the invention interior
Hold, the present invention can be implemented using other various specific embodiments, therefore, every design structure and think of using the present invention
Road, does some simple changes or the design changed, both falls within the scope of protection of the invention.
Claims (1)
1. a kind of I/O data dilatation moving methods in a balanced way, it is characterised in that：Comprise the following steps：
The first step：Initialization memory node, obtains existing storage system operational factor, node number, user data capacity and deposits
Storage system life parameter, and memory node each data block average daily access times are reset；
Second step：Monitoring memory capacity change, monitors existing storage system volume change situation, and obtain amount of user data and
Existing storage system residue free memory capacity；
3rd step：The daily I/O access times of each data block in memory node are counted, and calculates existing average daily I/O and deposited
Take number of times；
4th step：Whether data dilatation, according to second step, monitors existing storage system volume change situation, determines storage system
Whether need to implement dilatation, next step performed if amount of user data reaches the 90% of total memory capacity and implements data dilatation,
Otherwise return to step second step continues monitoring memory capacity situation of change；
5th step：According to total amount of user data, it is determined that new disk number is being added needed for dilation process, available data is in dilatation
Shared capacity ratio 40%～50% in new storage system afterwards,；
6th step：The new disk for adding is loaded, according to the new disk number of addition for determining；
7th step：According to the average daily I/O access times of data block, the data block for selecting front 30% is most hot data block；
8th step：According to the most dsc data block number for determining, its average block number in existing each memory node is calculated, in every node
Then it is defined as data block to be migrated more than the most hot data block of mean value；
9th step：According to data block to be migrated, the visitation frequency of non-migrating data block in single piece of transition process is calculated；According to true
Data block to be migrated in fixed memory node, calculates the visitation frequency of non-migrating data block during each data block migration；
Tenth step：Preferentially the non-migrating data block of mobile access frequency highest was migrated according to individual data block in new disk
The visitation frequency of the non-migrating data block calculated in journey, the non-migrating data block of preferential mobile access frequency highest is to new disk
In；
11st step：Whether data have migrated, and according to data block number to be migrated, hold if data block to be migrated movement is completed
Row next step, otherwise the 9th step of return to step continue to move to data block to be migrated；
12nd step：Whether monitoring storage system life cycle terminates, and completes if terminating, and otherwise returns the first step and continues prison
Survey the dilatation migration of storage system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611029235.9A CN106527995B (en) | 2016-11-22 | 2016-11-22 | A kind of data dilatation moving method of I/O equilibrium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611029235.9A CN106527995B (en) | 2016-11-22 | 2016-11-22 | A kind of data dilatation moving method of I/O equilibrium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106527995A true CN106527995A (en) | 2017-03-22 |
| CN106527995B CN106527995B (en) | 2019-03-29 |
Family
ID=58353162
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611029235.9A Active CN106527995B (en) | 2016-11-22 | 2016-11-22 | A kind of data dilatation moving method of I/O equilibrium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106527995B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107807796A (en) * | 2017-11-17 | 2018-03-16 | 北京联想超融合科技有限公司 | A kind of data hierarchy method, terminal and system based on super fusion storage system |
| CN109669641A (en) * | 2018-12-24 | 2019-04-23 | 深圳忆联信息系统有限公司 | Reduce the data deposit method and device of the SSD bit error rate |
| CN109753443A (en) * | 2019-01-12 | 2019-05-14 | 湖南国科微电子股份有限公司 | A kind of data processing method, device and electronic equipment |
| CN110737390A (en) * | 2018-07-20 | 2020-01-31 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for managing a storage system |
| CN111208952A (en) * | 2020-02-13 | 2020-05-29 | 河南文正电子数据处理有限公司 | Storage system capacity expansion method, readable storage medium and computing device |
| CN111736754A (en) * | 2019-03-25 | 2020-10-02 | 杭州海康威视系统技术有限公司 | Data migration method and device |
| CN115442388A (en) * | 2022-10-21 | 2022-12-06 | 四川省华存智谷科技有限责任公司 | Capacity expansion method, device and system for 100% utilization rate of distributed storage cluster |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102930062A (en) * | 2012-11-30 | 2013-02-13 | 南京富士通南大软件技术有限公司 | Rapid horizontal extending method for databases |
| CN103150263A (en) * | 2012-12-13 | 2013-06-12 | 深圳先进技术研究院 | Hierarchical storage method |
| US8935498B1 (en) * | 2011-09-29 | 2015-01-13 | Emc Corporation | Splitter based hot migration |
| CN105607868A (en) * | 2015-12-14 | 2016-05-25 | 华为数字技术（成都）有限公司 | Method and apparatus for capacity expansion of storage array |
| CN105630423A (en) * | 2015-12-25 | 2016-06-01 | 华中科技大学 | Erasure code cluster storage expansion method based on data caching |
-
2016
- 2016-11-22 CN CN201611029235.9A patent/CN106527995B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8935498B1 (en) * | 2011-09-29 | 2015-01-13 | Emc Corporation | Splitter based hot migration |
| CN102930062A (en) * | 2012-11-30 | 2013-02-13 | 南京富士通南大软件技术有限公司 | Rapid horizontal extending method for databases |
| CN103150263A (en) * | 2012-12-13 | 2013-06-12 | 深圳先进技术研究院 | Hierarchical storage method |
| CN105607868A (en) * | 2015-12-14 | 2016-05-25 | 华为数字技术（成都）有限公司 | Method and apparatus for capacity expansion of storage array |
| CN105630423A (en) * | 2015-12-25 | 2016-06-01 | 华中科技大学 | Erasure code cluster storage expansion method based on data caching |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107807796A (en) * | 2017-11-17 | 2018-03-16 | 北京联想超融合科技有限公司 | A kind of data hierarchy method, terminal and system based on super fusion storage system |
| CN110737390A (en) * | 2018-07-20 | 2020-01-31 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for managing a storage system |
| CN110737390B (en) * | 2018-07-20 | 2023-08-18 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for managing a storage system |
| CN109669641A (en) * | 2018-12-24 | 2019-04-23 | 深圳忆联信息系统有限公司 | Reduce the data deposit method and device of the SSD bit error rate |
| CN109669641B (en) * | 2018-12-24 | 2022-03-29 | 深圳忆联信息系统有限公司 | Data storage method and device for reducing SSD bit error rate |
| CN109753443A (en) * | 2019-01-12 | 2019-05-14 | 湖南国科微电子股份有限公司 | A kind of data processing method, device and electronic equipment |
| CN111736754A (en) * | 2019-03-25 | 2020-10-02 | 杭州海康威视系统技术有限公司 | Data migration method and device |
| CN111208952A (en) * | 2020-02-13 | 2020-05-29 | 河南文正电子数据处理有限公司 | Storage system capacity expansion method, readable storage medium and computing device |
| CN111208952B (en) * | 2020-02-13 | 2022-07-12 | 河南文正电子数据处理有限公司 | Storage system capacity expansion method, readable storage medium and computing device |
| CN115442388A (en) * | 2022-10-21 | 2022-12-06 | 四川省华存智谷科技有限责任公司 | Capacity expansion method, device and system for 100% utilization rate of distributed storage cluster |
| CN115442388B (en) * | 2022-10-21 | 2023-01-06 | 四川省华存智谷科技有限责任公司 | Capacity expansion method, device and system for 100% utilization rate of distributed storage cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106527995B (en) | 2019-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106527995B (en) | A kind of data dilatation moving method of I/O equilibrium | |
| CN104573119B (en) | Towards the Hadoop distributed file system storage methods of energy-conservation in cloud computing | |
| CN110196851B (en) | Data storage method, device, equipment and storage medium | |
| CN109690500A (en) | The elastic management of heterogeneous storage system is provided using Simulation spatial service quality (QoS) label in system based on by processor | |
| CN108810041A (en) | A kind of data write-in of distributed cache system and expansion method, device | |
| CN104699424B (en) | A kind of isomery EMS memory management process based on page temperature | |
| CN106406759B (en) | Data storage method and device | |
| CN102868763A (en) | Energy-saving dynamic adjustment method of virtual web application cluster in cloud computing environment | |
| CN102857560A (en) | Multi-service application orientated cloud storage data distribution method | |
| CN106534308A (en) | Method and device for solving data block access hotspot problem in distributed storage system | |
| CN109408590A (en) | Expansion method, device, equipment and the storage medium of distributed data base | |
| CN104346284A (en) | Memory management method and memory management equipment | |
| US11513854B1 (en) | Resource usage restrictions in a time-series database | |
| CN102981971A (en) | Quick-response phase change memory wear-leveling method | |
| CN104268099A (en) | Management data reading and writing method and device | |
| CN104035807B (en) | Metadata cache replacement method of cloud storage system | |
| CN111290699A (en) | Data migration method, device and system | |
| CN109358821A (en) | A kind of cold and hot data store optimization method of cloud computing of cost driving | |
| US9063667B2 (en) | Dynamic memory relocation | |
| Squillante et al. | Threshold-based priority policies for parallel-server systems with affinity scheduling | |
| CN113382074A (en) | Micro-service load balancing optimization method based on dynamic feedback | |
| CN112947860A (en) | Hierarchical storage and scheduling method of distributed data copies | |
| CN106126434B (en) | The replacement method and its device of the cache lines of the buffer area of central processing unit | |
| Ye et al. | A two-layer geo-cloud based dynamic replica creation strategy | |
| CN107920129A (en) | A kind of method, apparatus, equipment and the cloud storage system of data storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |