CN106527995B - A kind of data dilatation moving method of I/O equilibrium - Google Patents
A kind of data dilatation moving method of I/O equilibrium Download PDFInfo
- Publication number
- CN106527995B CN106527995B CN201611029235.9A CN201611029235A CN106527995B CN 106527995 B CN106527995 B CN 106527995B CN 201611029235 A CN201611029235 A CN 201611029235A CN 106527995 B CN106527995 B CN 106527995B
- Authority
- CN
- China
- Prior art keywords
- data block
- data
- storage system
- dilatation
- migrated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A kind of data dilatation moving method of I/O equilibrium, is related to computer field, is completed by 12 steps.The beneficial effects of the present invention are: the present invention realizes most hot I/O data block and can uniformly be distributed in all memory nodes, realizes low I/O Data Migration amount；Optimal user experience is realized, i.e., fast I/O subscriber response time.
Description
Technical field
The invention belongs to computer network and computer field of storage, the data for relating more specifically to a kind of I/O equilibrium expand
Hold moving method.
Background technique
With depth integration and development of the network technology in social all trades and professions, enterprise data center's storage system faces
The demand of mass data storage, thus the capacity of existing storage system is not able to satisfy growing data scale and proposes to capacity
Requirements at the higher level, in order to promote the capacity of data center's storage system, the volume space for expanding existing storage system is must to adopt
Technology, i.e. capacity-enlargement technology.During storage cluster data dilatation, existing dilatation strategy mainly considers new and old each magnetic
All data blocks, i.e., be simply uniformly distributed in by the equiblibrium mass distribution of disk total data block number and the Data Migrating Strategy implemented
In new storage system after dilatation.
However, there are locality characteristics for data access at true Enterprise Data center, thus even if data volume each
Disk balanced distribution can not ensure the harmony of storage system user I/O access, and unbalanced I/O access will seriously shadow
Ring the readwrite performance of storage system.
Summary of the invention
In view of the above technical problems the present invention provides a kind of data dilatation moving method of I/O equilibrium, this method exists
After the dilation process of RAID0, the uniform layout of I/O data block, low I/O Data Migration amount can be realized, and realize fast
The subscriber response time of speed.The present invention is achieved by the following technical solutions.
A kind of data dilatation moving method of I/O equilibrium, comprising the following steps:
Step 1: initialization memory node, obtains existing storage system operating parameter, node number, user data capacity
With storage system life parameter, and the average daily access times of each data block of memory node are reset；
Step 2: monitoring memory capacity variation, monitors existing storage system volume change situation, and obtain amount of user data
And existing storage system residue free memory capacity；
Step 3: the daily I/O access times of each data block in statistics memory node, and calculate existing average daily I/
O access times；
Step 4: whether data dilatation monitors existing storage system volume change situation according to second step, storage is determined
Whether system needs to implement dilatation, if amount of user data reach total memory capacity 90% if perform the next step and rapid implement data
Dilatation, otherwise return step second step continues to monitor memory capacity situation of change；
Step 5: determining that the required new disk number of addition, available data exist in dilation process according to total amount of user data
Shared capacity ratio 40%~50% in new storage system after dilatation,；
Step 6: the disk being newly added is loaded, according to the determining new disk number of addition；
Step 7: according to the average daily I/O access times of data block, the data block of choosing preceding 30% is most hot data block；
Step 8: calculating its average block number, every section in existing each memory node according to determining most dsc data block number
Most hot data block in point more than average value is then determined as data block to be migrated；
Step 9: calculating the visitation frequency of non-migrating data block in single block transition process according to data block to be migrated；Root
According to data block to be migrated in determining memory node, the access frequency of non-migrating data block during each data block migration is calculated
It is secondary；
Step 10: the preferential highest non-migrating data block of the mobile access frequency is moved into new disk according to individual data block
Move past the visitation frequency of the non-migrating data block calculated in journey, the preferential highest non-migrating data block of the mobile access frequency to new magnetic
In disk；
Step 11: whether data have migrated, according to data block number to be migrated, completed if data block to be migrated is mobile
It then performs the next step suddenly, otherwise the 9th step of return step continues to move to data block to be migrated；
Step 12: monitoring storage system life cycle whether terminate, if terminate if complete, otherwise return the first step after
The dilatation migration of continuous monitoring storage system.
The invention discloses a kind of data dilatation moving method of I/O equilibrium, basic principle is as shown in Fig. 2, storage system
Before system implements Data Migration, it is stored with user data in old disk storage node, according to every balance of each I/O user data block
Data block is divided into dsc data (red block) and cold data (white blocks) respectively by equal access times, and wherein hot data block is daily
The highest data block of user's visitation frequency, thus the most of access all focused on to hot data block of I/O access of system, because
This basic thought of the invention is exactly only to migrate hot data block in old disk, after so that its hot data block is uniformly distributed in dilatation
All memory nodes in, and the highest data block of priority migration user recent visit frequency during Data Migration,
To realize dsc data block I/O uniform layout and good user response.
The invention has the following beneficial effects:
(1) the uniform data layout of I/O: since there are the user I/O temporal localities accessed and space office for storage system
Portion's property, by counting the frequency of the average daily I/O access of each data block, select its front divide numerous access data block be used as to
The data block of migration, makes it uniformly be distributed in each memory node after dilatation, the present invention realizes the most hot equal energy of I/O data block
It is uniformly distributed in all memory nodes；
(2) low I/O Data Migration amount: during data dilatation, the part number of most frequent access has been only selected that
According to block as data block to be migrated, therefore the present invention realizes low I/O Data Migration amount；
(3) it fast I/O subscriber response time: during Data Migration, during one data block of every migration, counts respectively
The Average visits of each data block to be migrated all only migrate the highest data of I/O user's visitation frequency in each migration
Block, thus the present invention realizes optimal user experience, i.e., fast I/O subscriber response time.
Detailed description of the invention
Fig. 1 is a kind of data dilatation moving method flow chart of I/O equilibrium of the present invention.
Fig. 2 is a kind of data dilatation moving method structure chart of I/O equilibrium of the present invention.
Specific embodiment
Embodiment 1: the present invention is described in further detail with reference to the accompanying drawing
As shown in Figure 1, a kind of data dilatation moving method of I/O equilibrium of the present invention, including following implementation steps:
Step 1: initialization memory node, obtains existing storage system operating parameter, node number, user data capacity
With storage system life parameter, and the average daily access times of each data block of memory node are reset；
Step 2: monitoring memory capacity variation, monitors existing storage system volume change situation, and obtain amount of user data
And existing storage system residue free memory capacity；
Step 3: the daily I/O access times of each data block in statistics memory node, and calculate existing average daily I/
O access times；
Step 4: whether data dilatation monitors existing storage system volume change situation according to second step, storage is determined
Whether system needs to implement dilatation, if amount of user data reach total memory capacity 90% if perform the next step and rapid implement data
Dilatation, otherwise return step second step continues to monitor memory capacity situation of change；
Step 5: determining that the required new disk number of addition, available data exist in dilation process according to total amount of user data
Shared capacity ratio 40%~50% in new storage system after dilatation,；
Step 6: the disk being newly added is loaded, according to the determining new disk number of addition；
Step 7: according to the average daily I/O access times of data block, the data block of choosing preceding 30% is most hot data block；
Step 8: calculating its average block number, every section in existing each memory node according to determining most dsc data block number
Most hot data block in point more than average value is then determined as data block to be migrated；
Step 9: calculating the visitation frequency of non-migrating data block in single block transition process according to data block to be migrated；Root
According to data block to be migrated in determining memory node, the access frequency of non-migrating data block during each data block migration is calculated
It is secondary；
Step 10: the preferential highest non-migrating data block of the mobile access frequency is moved into new disk according to individual data block
Move past the visitation frequency of the non-migrating data block calculated in journey, the preferential highest non-migrating data block of the mobile access frequency to new magnetic
In disk；
Step 11: whether data have migrated, according to data block number to be migrated, completed if data block to be migrated is mobile
It then performs the next step suddenly, otherwise the 9th step of return step continues to move to data block to be migrated；
Step 12: monitoring storage system life cycle whether terminate, if terminate if complete, otherwise return the first step after
The dilatation migration of continuous monitoring storage system.
So far, the data dilatation moving method implementation steps of the present invention for a kind of I/O equilibrium all terminate.
The present invention is not only limited to above-mentioned specific embodiment, and persons skilled in the art are disclosed interior according to the present invention
Hold, the present invention can be implemented using other a variety of specific embodiments, it is therefore, all using design structure and think of of the invention
Road does the design of some simple variations or change, both falls within the scope of protection of the invention.
Claims (1)
1. a kind of data dilatation moving method of I/O equilibrium, it is characterised in that: the following steps are included:
Step 1: initialization memory node, obtains existing storage system operating parameter, node number, user data capacity and deposit
Storage system life parameter, and the average daily access times of each data block of memory node are reset；
Step 2: monitoring memory capacity variation, monitor existing storage system volume change situation, and obtain amount of user data and
Existing storage system residue free memory capacity；
Step 3: the daily I/O access times of each data block in statistics memory node, and calculate existing average daily I/O and deposit
Take number；
Step 4: whether data dilatation monitors existing storage system volume change situation, determines storage system according to second step
Whether need to implement dilatation, if amount of user data reach total memory capacity 90% if perform the next step it is rapid implement data dilatation,
Otherwise return step second step continues to monitor memory capacity situation of change；
Step 5: determining the required new disk number of addition in dilation process, available data is in dilatation according to total amount of user data
Shared capacity ratio 40%~50% in new storage system afterwards；
Step 6: loading the disk being newly added according to the determining new disk number of addition；
Step 7: according to the average daily I/O access times of data block, the data block of choosing preceding 30% is most hot data block；
Step 8: according to determining most dsc data block number, its average block number in existing each memory node is calculated, in every node
Most hot data block more than average value is then determined as data block to be migrated；
Step 9: calculating the visitation frequency of non-migrating data block in single block transition process according to data block to be migrated；Specific behaviour
As: according to data block to be migrated in determining memory node, calculate non-migrating data block during each data block migration
Visitation frequency；
Step 10: preferentially the highest non-migrating data block of the mobile access frequency is into new disk, concrete operations are as follows: according to single number
According to the visitation frequency of the non-migrating data block calculated in block transition process, the preferential highest non-migrating data block of the mobile access frequency
Into new disk；
Step 11: whether data have migrated, concrete operations are as follows: according to data block number to be migrated, if data block to be migrated
Mobile to complete then to perform the next step suddenly, otherwise the 9th step of return step continues to move to data block to be migrated；
Step 12: whether monitoring storage system life cycle terminates, completed if terminating, otherwise returns to the first step and continue to supervise
Survey the dilatation migration of storage system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611029235.9A CN106527995B (en) | 2016-11-22 | 2016-11-22 | A kind of data dilatation moving method of I/O equilibrium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611029235.9A CN106527995B (en) | 2016-11-22 | 2016-11-22 | A kind of data dilatation moving method of I/O equilibrium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106527995A CN106527995A (en) | 2017-03-22 |
| CN106527995B true CN106527995B (en) | 2019-03-29 |
Family
ID=58353162
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611029235.9A Active CN106527995B (en) | 2016-11-22 | 2016-11-22 | A kind of data dilatation moving method of I/O equilibrium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106527995B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107807796B (en) * | 2017-11-17 | 2021-03-05 | 北京联想超融合科技有限公司 | Data layering method, terminal and system based on super-fusion storage system |
| CN110737390B (en) * | 2018-07-20 | 2023-08-18 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for managing a storage system |
| CN109669641B (en) * | 2018-12-24 | 2022-03-29 | 深圳忆联信息系统有限公司 | Data storage method and device for reducing SSD bit error rate |
| CN109753443B (en) * | 2019-01-12 | 2021-05-18 | 湖南国科微电子股份有限公司 | Data processing method and device and electronic equipment |
| CN111736754A (en) * | 2019-03-25 | 2020-10-02 | 杭州海康威视系统技术有限公司 | Data migration method and device |
| CN111208952B (en) * | 2020-02-13 | 2022-07-12 | 河南文正电子数据处理有限公司 | Storage system capacity expansion method, readable storage medium and computing device |
| CN115442388B (en) * | 2022-10-21 | 2023-01-06 | 四川省华存智谷科技有限责任公司 | Capacity expansion method, device and system for 100% utilization rate of distributed storage cluster |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102930062A (en) * | 2012-11-30 | 2013-02-13 | 南京富士通南大软件技术有限公司 | Rapid horizontal extending method for databases |
| CN103150263A (en) * | 2012-12-13 | 2013-06-12 | 深圳先进技术研究院 | Hierarchical storage method |
| US8935498B1 (en) * | 2011-09-29 | 2015-01-13 | Emc Corporation | Splitter based hot migration |
| CN105607868A (en) * | 2015-12-14 | 2016-05-25 | 华为数字技术（成都）有限公司 | Method and apparatus for capacity expansion of storage array |
| CN105630423A (en) * | 2015-12-25 | 2016-06-01 | 华中科技大学 | Erasure code cluster storage expansion method based on data caching |
-
2016
- 2016-11-22 CN CN201611029235.9A patent/CN106527995B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8935498B1 (en) * | 2011-09-29 | 2015-01-13 | Emc Corporation | Splitter based hot migration |
| CN102930062A (en) * | 2012-11-30 | 2013-02-13 | 南京富士通南大软件技术有限公司 | Rapid horizontal extending method for databases |
| CN103150263A (en) * | 2012-12-13 | 2013-06-12 | 深圳先进技术研究院 | Hierarchical storage method |
| CN105607868A (en) * | 2015-12-14 | 2016-05-25 | 华为数字技术（成都）有限公司 | Method and apparatus for capacity expansion of storage array |
| CN105630423A (en) * | 2015-12-25 | 2016-06-01 | 华中科技大学 | Erasure code cluster storage expansion method based on data caching |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106527995A (en) | 2017-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106527995B (en) | A kind of data dilatation moving method of I/O equilibrium | |
| CN102981971B (en) | A kind of phase transition storage loss equalizing method of quick response | |
| CN108810041A (en) | A kind of data write-in of distributed cache system and expansion method, device | |
| CN104571954B (en) | A kind of date storage method and device | |
| CN104699424B (en) | A kind of isomery EMS memory management process based on page temperature | |
| CN106406759B (en) | Data storage method and device | |
| CN106598646A (en) | Program update method and terminal | |
| CN103714098B (en) | Method and system for carrying out subregion to database | |
| CN105975345B (en) | A kind of video requency frame data dynamic equalization memory management method based on distributed memory | |
| CN104268099A (en) | Management data reading and writing method and device | |
| CN109213699A (en) | A kind of metadata management method, system, equipment and computer readable storage medium | |
| CN106162680A (en) | Dilatation parameter determination method and device | |
| CN104951462B (en) | Method and system for managing database | |
| CN104035807B (en) | Metadata cache replacement method of cloud storage system | |
| CN107347039A (en) | A kind of management method and device in shared buffer memory space | |
| CN108733308A (en) | Method and apparatus for managing Pan Chi | |
| CN103605483A (en) | Feature processing method for block-level data in hierarchical storage system | |
| CN106534308A (en) | Method and device for solving data block access hotspot problem in distributed storage system | |
| CN110413224A (en) | Data storage method and device and memory | |
| CN109358821A (en) | A kind of cold and hot data store optimization method of cloud computing of cost driving | |
| CN103345519B (en) | Method and device for data distribution of shared-nothing distributed database | |
| CN107491272A (en) | A kind of method, apparatus of Data Migration, equipment and storage medium | |
| CN111290699A (en) | Data migration method, device and system | |
| CN102981972A (en) | Wear-leveling method for phase change memory | |
| CN105373350A (en) | Data management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |