CN104932835A - Erasure code based distributed storage system capacity expansion and reduction method - Google Patents
Erasure code based distributed storage system capacity expansion and reduction method Download PDFInfo
- Publication number
- CN104932835A CN104932835A CN201510289976.XA CN201510289976A CN104932835A CN 104932835 A CN104932835 A CN 104932835A CN 201510289976 A CN201510289976 A CN 201510289976A CN 104932835 A CN104932835 A CN 104932835A
- Authority
- CN
- China
- Prior art keywords
- disk
- ous
- dilatation
- nus
- reduction method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an erasure code based distributed storage system capacity expansion and reduction method. Before capacity expansion or capacity reduction, a disk array is identified, the free space of each disk in the disk array is checked, and relevant parameters m and n are acquired; the quantity of retained, remapped and reconstructed old used strips/new used strips (OUS/NUS) in the disk array is computed; the retained, remapped and reconstructed OUSs/NUSs are processed concurrently; and all verifying blocks are modified. The method can support bi-directional expansion of the disk array, the overhead generated during the expansion process is minimized, and the expansible process of a verifying array is accelerated.
Description
Technical field
The present invention relates to a kind of dilatation and capacity reduction method of the distributed memory system based on correcting and eleting codes.
Background technology
In different disk arrays, redundant array of inexpensive disk based on exclusive or check computing (is called for short " verification array " herein, XOR-based RAID) because it can provide highly reliable and high performance stores service simultaneously, become large-scale data center mainstream storage device.In recent years, the extensibility research verifying array receives much concern, mainly due to following reason:
1) in order to meet larger storage space, higher demand of handling up.Directly in the disk array used, adding disk is the scheme that a cost performance is high.
2) in order to meet energy-conservation demand.By removing the low disk of utilization rate in disk array, to reach the object reducing energy consumption.
3) in order to tackle the demand of more and more application on site.Usually, RAID is widely used in various online service, as cloud computing.Enhanced scalability can not only tackle the problem of the user data outburst that application on site faces, the expense huge servicing time that simultaneously it also avoid that mechanical disorder produces.
4) needs of data center.RAID framework is widely used in cluster and large memory system, and in such systems, extensibility is most important.
In different RAID structures, verification array (mainly RAID-4, RAID-5 and RAID-6) is of paramount importance a kind of form, and is widely used in large-scale data center.The expandable method of current main flow comprises wheel turn (Round-Robin, RR), half wheel turns Semi-RR, ALV and MDM etc.
But the expandable method of current main-stream has two problems.Problem is the high I/O expense in easily extensible process.At traditional expandable method---in wheel shifting method, nearly all data are all moved, thus cause all verifications all will to be recalculated and revise, and a large amount of metadata needs to upgrade.Half wheel turn has the unbalanced problem of Data distribution8, and this greatly will reduce the performance of verification array.ALV reduces I/O number by the method for assembling migration I/O, but it can not reduce the sum of the data block of access.Although MDM can reduce the activity of data and the verification amendment of quantity, it also can cause the performance after its storage efficiency and expansion lower.In addition, MDM changes the parity checking layout of original array, and this makes data-mapping more complicated, process read and write request more difficult.
Whether Second Problem supports two-way easily extensible (add disk and delete disk).Except wheel shifting method, other main stream approach all only support forward easily extensible (increase disk).
Summary of the invention
The object of the present invention is to provide a kind of dilatation and capacity reduction method of the distributed memory system based on correcting and eleting codes, the easily extensible process verifying array can be accelerated.
For solving the problem, the invention provides a kind of dilatation and capacity reduction method of the distributed memory system based on correcting and eleting codes, comprising:
Step S1, identify disk array before dilatation or capacity reducing, in chkdsk array each disk free space and obtain correlation parameter m and n, wherein, n is the number of disks before dilatation or capacity reducing, and m is the number of disks reduced after the data in magnetic disk or capacity reducing added after dilatation;
Step S2, calculates in disk array the OUS/NUS quantity retaining, remap and reconstruct, and wherein, OUS is the band of front the mistake of expansion, and NUS is the band for using after expanding;
Step S3, the OUS/NUS processing reservation concomitantly, remap and reconstruct;
Step S4, revises all check blocks.
Further, in the above-mentioned methods, the OUS/NUS that described process retains comprises:
All data blocks and check block are retained on same disk, if having data block to be moved into or remove corresponding check block, this check block are modified.
Further, in the above-mentioned methods, all data blocks and check block are retained on same disk, if having data block to be moved into or remove corresponding check block, during this check block is modified, comprise:
Upgrade the band numbering of OUS/NUS.
Further, in the above-mentioned methods, the described process OUS/NUS that remaps comprises:
By the new band that remaps, all data blocks are retained on same disk.
Further, in the above-mentioned methods, by the new band that remaps, it is upper that all data blocks are retained in same disk, comprising:
All data blocks that remaps also distribute new band numbering.
Further, in the above-mentioned methods, the OUS/NUS of described process reconstruct comprises:
All data blocks be migrated on another disk, in the OUS/NUS of each reconstruct, data block is migrated on new disk to expand or to remove disk.
Further, in the above-mentioned methods, be describedly migrated on another disk by all data blocks, in the OUS/NUS of each reconstruct, data block to be migrated on new disk with expansion or to remove disk, comprising:
Move all data blocks and distribute new band numbering.
Compared with prior art, the present invention mainly contains following three advantages:
1) support the two-way expansion of disk array (increase or reduce disk), when increasing new disk, the present invention can ensure that the array performance after expanding and storage efficiency get a promotion;
2) expense produced in expansion process is down to minimum, comprise Data Migration, parity block amendment and the expense of metadata operation, the present invention not only reduces Data Migration amount and verification index word to greatest extent, revise with the visual angle of the overall situation and calculate all verification chains, instead of any single data/verification is operated, also reduce the quantity of easily extensible expense and amendment metadata simultaneously;
3) present invention preserves data and the parity checking layout of original verification array, the performance after making system have better easily extensible.Array has higher dirigibility and availability, and can provide better service for application in real time, result shows, extended method of the present invention can accelerate the easily extensible process verifying array.
Accompanying drawing explanation
Fig. 1 is that the dilatation of the distributed memory system based on correcting and eleting codes of one embodiment of the invention and the RAID5 of capacity reduction method increase disk schematic diagram;
Fig. 2 is that the dilatation of the distributed memory system based on correcting and eleting codes of one embodiment of the invention and the RAID5 of capacity reduction method reduce disk schematic diagram.
Embodiment
For enabling above-mentioned purpose of the present invention, feature and advantage become apparent more, and below in conjunction with the drawings and specific embodiments, the present invention is further detailed explanation.
The invention provides a kind of dilatation and capacity reduction method of the distributed memory system based on correcting and eleting codes, comprising:
Step S1, step for identifying: dilatation (expanding disk array capacity) or capacity reducing (reducing disk array capacity) front identification disk array, in chkdsk array each disk free space (comprising new disk) and obtain correlation parameter m and n, wherein, n is the number of disks before dilatation or capacity reducing, and m is the number of disks reduced after the data in magnetic disk or capacity reducing added after dilatation;
Step S2, step for band reconstruct: calculate in disk array the OUS/NUS quantity retaining, remap and reconstruct, wherein, OUS (Old Used Stripe) is the front band crossed of expansion, and NUS (NewUsed Stripe) is the band used after expansion; Concrete, according to the difference of verification chain before and after expansion, some bands with short verification chain are retained on original disk, and other data can be deleted and move, and based on different functions, the band having shorter verification chain will be divided into 3 classes:
The OUS/NUS retained, the 0-2 band as before left side expansion in Fig. 1: all data blocks and check block are retained on same disk, if having data block to be moved into or remove corresponding check block, this check block are modified;
The OUS/NUS remapped, the 3-5 band as before left side expansion in Fig. 1: by the new band that remaps, all data blocks are retained on same disk;
The OUS/NUS of reconstruct, the 6-9 band as before left side expansion in Fig. 1: all data blocks be migrated on another disk, in the OUS/NUS of each reconstruct, data block is migrated on new disk to expand or to remove disk.
Step S3, the step for bar tape handling: the OUS/NUS processing reservation concomitantly, remap and reconstruct;
Step S4, the step for checking treatment: revise all check blocks.Concrete, basic thought of the present invention keeps existing band as far as possible, sacrifices sub-fraction bar simultaneously and bring the load balance realizing I/O, thus reach the object reducing Data Migration I/O.As Fig. 1, according to above-mentioned four steps, expand to 5 to illustrate (n=3, m=2) from 3 for RAID-5 disk number.The OUS number after identification completes, can calculate total reservation, remapping and reconstruct is respectively 3,3,4.Band processing stage, block 6-11 is remapped, and corresponding metadata is updated, and block 12-19 is migrated on new disk, and upgrades corresponding band numbering.Finally, amendment verification Q0 ~ Q4.As shown in Figure 2, inverse expansion (capacity reducing/minimizing disk) is the inverse operations of forward expansion (dilatation/increases disk), and when namely removing disk, data mobile direction is contrary with when increasing disk.
In the dilatation of the distributed memory system based on correcting and eleting codes of the present invention and a preferred embodiment of capacity reduction method, described all data blocks and check block are retained on same disk, if have data block to be moved into or remove corresponding check block, during this check block is modified, comprise the band numbering upgrading OUS/NUS.
In the dilatation of the distributed memory system based on correcting and eleting codes of the present invention and a preferred embodiment of capacity reduction method, by the new band that remaps, it is upper that all data blocks are retained in same disk, comprises and remap all data blocks and distribute new band numbering.
In the dilatation of the distributed memory system based on correcting and eleting codes of the present invention and a preferred embodiment of capacity reduction method, all data blocks are migrated on another disk, in the OUS/NUS of each reconstruct, data block to be migrated on new disk with expansion or to remove in disk, comprises all data blocks of migration and distributes new band numbering.
The present invention mainly contains following three advantages:
1) support the two-way expansion of disk array (increase or reduce disk), when increasing new disk, the present invention can ensure that the array performance after expanding and storage efficiency get a promotion;
2) expense produced in expansion process is down to minimum, comprise Data Migration, parity block amendment and the expense of metadata operation, the present invention not only reduces Data Migration amount and verification index word to greatest extent, revise with the visual angle of the overall situation and calculate all verification chains, instead of any single data/verification is operated, also reduce the quantity of easily extensible expense and amendment metadata simultaneously;
3) present invention preserves data and the parity checking layout of original verification array, the performance after making system have better easily extensible.Array has higher dirigibility and availability, and can provide better service for application in real time, result shows, extended method of the present invention can accelerate the easily extensible process verifying array.
In this instructions, each embodiment adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar portion mutually see.
Professional can also recognize further, in conjunction with unit and the algorithm steps of each example of embodiment disclosed herein description, can realize with electronic hardware, computer software or the combination of the two, in order to the interchangeability of hardware and software is clearly described, generally describe composition and the step of each example in the above description according to function.These functions perform with hardware or software mode actually, depend on application-specific and the design constraint of technical scheme.Professional and technical personnel can use distinct methods to realize described function to each specifically should being used for, but this realization should not thought and exceeds scope of the present invention.
Obviously, those skilled in the art can carry out various change and modification to invention and not depart from the spirit and scope of the present invention.Like this, if these amendments of the present invention and modification belong within the scope of the claims in the present invention and equivalent technologies thereof, then the present invention is also intended to comprise these change and modification.
Claims (7)
1., based on dilatation and the capacity reduction method of the distributed memory system of correcting and eleting codes, it is characterized in that, comprising:
Identify disk array before dilatation or capacity reducing, in chkdsk array each disk free space and obtain correlation parameter m and n, wherein, n is the number of disks before dilatation or capacity reducing, and m is the number of disks reduced after the data in magnetic disk or capacity reducing added after dilatation;
Calculate in disk array the OUS/NUS quantity retaining, remap and reconstruct, wherein, OUS is the band of front the mistake of expansion, and NUS is the band for using after expanding;
The OUS/NUS processing reservation concomitantly, remap and reconstruct;
Revise all check blocks.
2. as claimed in claim 1 based on dilatation and the capacity reduction method of the distributed memory system of correcting and eleting codes, it is characterized in that, the OUS/NUS that described process retains comprises:
All data blocks and check block are retained on same disk, if having data block to be moved into or remove corresponding check block, this check block are modified.
3. as claimed in claim 2 based on dilatation and the capacity reduction method of the distributed memory system of correcting and eleting codes, it is characterized in that, all data blocks and check block are retained on same disk, if having data block to be moved into or remove corresponding check block, during this check block is modified, comprising:
Upgrade the band numbering of OUS/NUS.
4. as claimed in claim 1 based on dilatation and the capacity reduction method of the distributed memory system of correcting and eleting codes, it is characterized in that, the described process OUS/NUS that remaps comprises:
By the new band that remaps, all data blocks are retained on same disk.
5., as claimed in claim 4 based on dilatation and the capacity reduction method of the distributed memory system of correcting and eleting codes, it is characterized in that, by the new band that remaps, it is upper that all data blocks are retained in same disk, comprising:
All data blocks that remaps also distribute new band numbering.
6. as claimed in claim 1 based on dilatation and the capacity reduction method of the distributed memory system of correcting and eleting codes, it is characterized in that, the OUS/NUS of described process reconstruct comprises:
All data blocks be migrated on another disk, in the OUS/NUS of each reconstruct, data block is migrated on new disk to expand or to remove disk.
7. as claimed in claim 6 based on dilatation and the capacity reduction method of the distributed memory system of correcting and eleting codes, it is characterized in that, describedly all data blocks are migrated on another disk, in the OUS/NUS of each reconstruct, data block be migrated on new disk with expansion or remove disk, comprising:
Move all data blocks and distribute new band numbering.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510289976.XA CN104932835A (en) | 2015-05-31 | 2015-05-31 | Erasure code based distributed storage system capacity expansion and reduction method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510289976.XA CN104932835A (en) | 2015-05-31 | 2015-05-31 | Erasure code based distributed storage system capacity expansion and reduction method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104932835A true CN104932835A (en) | 2015-09-23 |
Family
ID=54120016
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510289976.XA Pending CN104932835A (en) | 2015-05-31 | 2015-05-31 | Erasure code based distributed storage system capacity expansion and reduction method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104932835A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790408A (en) * | 2016-11-29 | 2017-05-31 | 中国空间技术研究院 | A kind of coding method repaired for distributed memory system node |
| CN109428913A (en) * | 2017-08-25 | 2019-03-05 | 浙江宇视科技有限公司 | A kind of storage expansion method and device |
| CN111831223A (en) * | 2020-06-19 | 2020-10-27 | 华中科技大学 | Fault-tolerant coding method, device and system for improving expandability of data deduplication system |
| CN112835533A (en) * | 2021-02-25 | 2021-05-25 | 上海交通大学 | Cloud storage array expansion method and device based on rack level |
| CN114546272A (en) * | 2022-02-18 | 2022-05-27 | 山东云海国创云计算装备产业创新中心有限公司 | Method, system, apparatus and storage medium for fast universal RAID demotion to RAID5 |
| CN115437578A (en) * | 2022-11-04 | 2022-12-06 | 苏州浪潮智能科技有限公司 | Data migration method, device, equipment and medium of disk array |
| CN115576500A (en) * | 2022-12-06 | 2023-01-06 | 苏州浪潮智能科技有限公司 | RAID array capacity expansion method and related device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103593253A (en) * | 2013-11-22 | 2014-02-19 | 华中科技大学 | Vertical RAID-6 coding method based on exclusive or |
| CN104035732A (en) * | 2014-06-17 | 2014-09-10 | 中国人民解放军国防科学技术大学 | Data placing method aiming at erasure codes |
-
2015
- 2015-05-31 CN CN201510289976.XA patent/CN104932835A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103593253A (en) * | 2013-11-22 | 2014-02-19 | 华中科技大学 | Vertical RAID-6 coding method based on exclusive or |
| CN104035732A (en) * | 2014-06-17 | 2014-09-10 | 中国人民解放军国防科学技术大学 | Data placing method aiming at erasure codes |
Non-Patent Citations (1)
| Title |
|---|
| CHENTAO WU, XUBIN HE: "GSR: A Global Stripe-based Redistribution Approach to Accelerate RAID-5 Scaling", 《2012 41ST INTERNATIONAL CONFERENCE ON PARALLEL PROCESSING》 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106790408B (en) * | 2016-11-29 | 2019-08-09 | 中国空间技术研究院 | A kind of coding method for distributed memory system node reparation |
| CN106790408A (en) * | 2016-11-29 | 2017-05-31 | 中国空间技术研究院 | A kind of coding method repaired for distributed memory system node |
| CN109428913B (en) * | 2017-08-25 | 2021-06-01 | 浙江宇视科技有限公司 | Storage capacity expansion method and device |
| CN109428913A (en) * | 2017-08-25 | 2019-03-05 | 浙江宇视科技有限公司 | A kind of storage expansion method and device |
| CN111831223B (en) * | 2020-06-19 | 2021-06-11 | 华中科技大学 | Fault-tolerant coding method, device and system for improving expandability of data deduplication system |
| CN111831223A (en) * | 2020-06-19 | 2020-10-27 | 华中科技大学 | Fault-tolerant coding method, device and system for improving expandability of data deduplication system |
| CN112835533A (en) * | 2021-02-25 | 2021-05-25 | 上海交通大学 | Cloud storage array expansion method and device based on rack level |
| CN112835533B (en) * | 2021-02-25 | 2023-02-17 | 上海交通大学 | Cloud storage array expansion method and device based on rack level |
| CN114546272A (en) * | 2022-02-18 | 2022-05-27 | 山东云海国创云计算装备产业创新中心有限公司 | Method, system, apparatus and storage medium for fast universal RAID demotion to RAID5 |
| CN114546272B (en) * | 2022-02-18 | 2024-04-26 | 山东云海国创云计算装备产业创新中心有限公司 | Method, system, device and storage medium for degrading RAID (redundant array of independent disks) to RAID5 (redundant array of independent disks) |
| CN115437578A (en) * | 2022-11-04 | 2022-12-06 | 苏州浪潮智能科技有限公司 | Data migration method, device, equipment and medium of disk array |
| CN115437578B (en) * | 2022-11-04 | 2023-03-24 | 苏州浪潮智能科技有限公司 | Data migration method, device, equipment and medium of disk array |
| WO2024093479A1 (en) * | 2022-11-04 | 2024-05-10 | 苏州元脑智能科技有限公司 | Data migration method and apparatus for redundant arrays of independent disks, and device and medium |
| CN115576500A (en) * | 2022-12-06 | 2023-01-06 | 苏州浪潮智能科技有限公司 | RAID array capacity expansion method and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104932835A (en) | Erasure code based distributed storage system capacity expansion and reduction method | |
| JP3587667B2 (en) | Parity storage and data repair method for external storage subsystem | |
| CN104765693A (en) | Data storage method, device and system | |
| CN101059751B (en) | Method and system for increasing parallelism of disk accesses when restoring data in a disk array system | |
| CN104471546B (en) | Effective redundant arrays of inexpensive disks skill and technique for improving solid state hard disc reliability | |
| US8966170B2 (en) | Elastic cache of redundant cache data | |
| CN102799533B (en) | Method and apparatus for shielding damaged sector of disk | |
| CN107484427B (en) | Method for processing stripes in a storage device and storage device | |
| CN107250975B (en) | Data storage system and data storage method | |
| US11218174B2 (en) | Method and system for storing data locally repairable and efficient multiple encoding | |
| US8838890B2 (en) | Stride based free space management on compressed volumes | |
| US8392813B2 (en) | Redundant file system | |
| CN1118503A (en) | RAID level 5 with free blocks parity cache | |
| CN104598162A (en) | Method and system for writing data across storage devices | |
| JPH04230512A (en) | Method and apparatus for updating record for dasd array | |
| CN102184079B (en) | Method for optimizing writing performance of redundant array of independent disk (RAID5)-grade disk array | |
| CN102722340A (en) | Data processing method, apparatus and system | |
| CN105302492A (en) | RAID reconstruction method and device | |
| CN112799604B (en) | N-Code-based RAID6 disk array capacity expansion method and data filling method | |
| CN102520890A (en) | RS (Reed-Solomon) - DRAID( D redundant array of independent disk) system based on GPUs (graphic processing units) and method for controlling data of memory devices | |
| CN104866244A (en) | RAID-6 I/O scheduling method for balancing strip writing | |
| CN105677508A (en) | Method and system for modifying erasure code data in cloud storage | |
| CN111782439A (en) | Double-disk cyclic verification method based on horizontal coding | |
| JP2002278707A (en) | Disk controller | |
| CN113687975A (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150923 |
|
| WD01 | Invention patent application deemed withdrawn after publication |