CN105373349A - Mass data storage system capable of deleting repetitive data - Google Patents
Mass data storage system capable of deleting repetitive data Download PDFInfo
- Publication number
- CN105373349A CN105373349A CN201510744661.XA CN201510744661A CN105373349A CN 105373349 A CN105373349 A CN 105373349A CN 201510744661 A CN201510744661 A CN 201510744661A CN 105373349 A CN105373349 A CN 105373349A
- Authority
- CN
- China
- Prior art keywords
- data
- size
- duplication
- deleting
- heavily
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention belongs to the technical field of information, and in particular relates to a mass data storage system capable of deleting repetitive data. A repetitive data deleting function is realized as follows: the size of a block is adjusted according to the size of a file system; the size of the block is set to 128 KB; the size of the block can be automatically configured; a repetitive data deleting technology can be selectively set; data with much repetition can be deleted while being stored, such that use of a hard disk is reduced; data with little repetition is not deleted; and thus, the data read-write speed is increased.
Description
Technical field
The invention belongs to areas of information technology, particularly a kind of can the large data-storage system of deleting duplicated data.
Background technology
Along with globalization ecommerce, the carrying out on a large scale of non-support cable and cloud computing, on the memory device of various application system, TB or even the PB level mode that information is just storing with data grows at top speed. as EMCCEO Qiao Tusi says shareholders " the most inundant two kinds of trend have appearred in IT industry: cloud computing and mass data ".Along with the fast development of cloud computing, the efficient storage demand of mass data and management become the emphasis of a research.
Summary of the invention
For above-mentioned prior art, the present invention proposes a kind of can the large data-storage system of deleting duplicated data, concrete technical scheme is as follows:
Can the large data-storage system of deleting duplicated data, comprise the setup of attribute of automatically simplifying and the setup of attribute heavily deleting technology, the NAS subregion that the setup of attribute of automatically simplifying is set up can create the logical volume being greater than physical store, each logical volume shares the size of whole storage pool, use setquota that the size of logical volume is set. the SAN subregion of foundation can create the logical volume being greater than physical store size, the partition size set up is oneself setting, outside storage pool size, but the size of the size of available memory pool or original storage pool, just when multi-section display, the utilization factor of storage pool is illusion or real situation, use zfscreate-s-b128K-V that the size of logical volume is set, the disk mapped out is exactly the size of automatically simplifying configuration setting, but the size of available capacity or original total storage pool, when LiveStor keeper obtains warning message by the storage pool alarm arranged, will store dilatation rear end as required, add hard disk, for user provides lasting available efficient storage,
The function of heavily deleting of heavily deleting the setup of attribute of technology is data de-duplication based on block level, and acquiescence uses SHA-256, does not verify; Utilize SHA-256 hash function to provide block level data de-duplication function, the LiveStor opening data de-duplication needs stronger processing power, and therefore raising processor ability and internal memory just can improve the speed of data de-duplication; 3 property values are provided with to data de-duplication technology, are respectively on, off, verify; When attribute is set on, this method is heavily to delete technology medium velocity the fastest, because it is undertaken deleting proportion by the cryptographic hash of block, but different data are deleted as identical cryptographic hash by this heavy possibility of deleting existence 2256; In order to ensure the security of hash data de-duplication, the property value heavily deleting function can be set to verify, allow storage data carry out whole byte contrast; For data de-duplication, also can use improvement, it and authentication function to reduce required processing power, and combines the bulk velocity improving data de-duplication by simple hashing algorithm; Data de-duplication function can carry out the size of adjustment block according to the size of file system, arranging block size is 128KB, the size of block can configure automatically, heavily the technology of deleting can be arranged selectively, for comprising the many data of repetition, just can carry out data de-duplication work when storing, saving the use of hard disk, the data few to repeating data are not heavily deleted, and improve the read or write speed of data.
Beneficial effect:
1. the large data-storage system that the present invention proposes has high-performance, high availability, easy-to-use, manageable feature.Can memory property be improved, reduce the pressure of environment, reduce and totally realize cost, reduce energy consumption and reduce CO2 emissions, meet the green requirement stored.
2. the large data-storage system that the present invention proposes not only provides efficient storage; can also based on snapping technique for user provides the continuous data protection of local logical partition; remote copy and restore funcitons is provided in conjunction with local CDP; meet the continuity of production run and the demand of disaster recovery, farthest protect secure user data.
Embodiment
Can the large data-storage system of deleting duplicated data, comprise the setup of attribute of automatically simplifying and the setup of attribute heavily deleting technology;
The NAS subregion that the setup of attribute of automatically simplifying is set up can create the logical volume being greater than physical store, each logical volume shares the size of whole storage pool, use setquota that the size of logical volume is set. the SAN subregion of foundation can create the logical volume being greater than physical store size, the partition size set up is oneself setting, outside storage pool size, but the size of the size of available memory pool or original storage pool, just when multi-section display, the utilization factor of storage pool is illusion or real situation, use zfscreate-s-b128K-V that the size of logical volume is set, the disk mapped out is exactly the size of automatically simplifying configuration setting, but the size of available capacity or original total storage pool, when LiveStor keeper obtains warning message by the storage pool alarm arranged, will store dilatation rear end as required, add hard disk, for user provides lasting available efficient storage,
The function of heavily deleting of heavily deleting the setup of attribute of technology is data de-duplication based on block level, and acquiescence uses SHA-256, does not verify; Utilize SHA-256 hash function to provide block level data de-duplication function, the LiveStor opening data de-duplication needs stronger processing power, and therefore raising processor ability and internal memory just can improve the speed of data de-duplication; 3 property values are provided with to data de-duplication technology, are respectively on, off, verify; When attribute is set on, this method is heavily to delete technology medium velocity the fastest, because it is undertaken deleting proportion by the cryptographic hash of block, but different data are deleted as identical cryptographic hash by this heavy possibility of deleting existence 2256; In order to ensure the security of hash data de-duplication, the property value heavily deleting function can be set to verify, allow storage data carry out whole byte contrast; For data de-duplication, also can use improvement, it and authentication function to reduce required processing power, and combines the bulk velocity improving data de-duplication by simple hashing algorithm; Data de-duplication function can carry out the size of adjustment block according to the size of file system, arranging block size is 128KB, the size of block can configure automatically, heavily the technology of deleting can be arranged selectively, for comprising the many data of repetition, just can carry out data de-duplication work when storing, saving the use of hard disk, the data few to repeating data are not heavily deleted, and improve the read or write speed of data.
Claims (1)
1. can the large data-storage system of deleting duplicated data, comprise the setup of attribute of automatically simplifying and the setup of attribute heavily deleting technology;
The NAS subregion that the setup of attribute of automatically simplifying is set up can create the logical volume being greater than physical store, each logical volume shares the size of whole storage pool, use setquota that the size of logical volume is set. the SAN subregion of foundation can create the logical volume being greater than physical store size, the partition size set up is oneself setting, outside storage pool size, but the size of the size of available memory pool or original storage pool, just when multi-section display, the utilization factor of storage pool is illusion or real situation, use zfscreate-s-b128K-V that the size of logical volume is set, the disk mapped out is exactly the size of automatically simplifying configuration setting, but the size of available capacity or original total storage pool, when LiveStor keeper obtains warning message by the storage pool alarm arranged, will store dilatation rear end as required, add hard disk, for user provides lasting available efficient storage,
The function of heavily deleting of heavily deleting the setup of attribute of technology is data de-duplication based on block level, and acquiescence uses SHA-256, does not verify; Utilize SHA-256 hash function to provide block level data de-duplication function, the LiveStor opening data de-duplication needs stronger processing power, and therefore raising processor ability and internal memory just can improve the speed of data de-duplication; 3 property values are provided with to data de-duplication technology, are respectively on, off, verify; When attribute is set on, this method is heavily to delete technology medium velocity the fastest, because it is undertaken deleting proportion by the cryptographic hash of block, but different data are deleted as identical cryptographic hash by this heavy possibility of deleting existence 2256; In order to ensure the security of hash data de-duplication, the property value heavily deleting function can be set to verify, allow storage data carry out whole byte contrast; For data de-duplication, also can use improvement, it and authentication function to reduce required processing power, and combines the bulk velocity improving data de-duplication by simple hashing algorithm; Data de-duplication function can carry out the size of adjustment block according to the size of file system, arranging block size is 128KB, the size of block can configure automatically, heavily the technology of deleting can be arranged selectively, for comprising the many data of repetition, just can carry out data de-duplication work when storing, saving the use of hard disk, the data few to repeating data are not heavily deleted, and improve the read or write speed of data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510744661.XA CN105373349A (en) | 2015-10-30 | 2015-10-30 | Mass data storage system capable of deleting repetitive data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510744661.XA CN105373349A (en) | 2015-10-30 | 2015-10-30 | Mass data storage system capable of deleting repetitive data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105373349A true CN105373349A (en) | 2016-03-02 |
Family
ID=55375583
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510744661.XA Pending CN105373349A (en) | 2015-10-30 | 2015-10-30 | Mass data storage system capable of deleting repetitive data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105373349A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105930223A (en) * | 2016-04-24 | 2016-09-07 | 湖南大学 | Method for reducing size of check point file |
| CN110018988A (en) * | 2017-11-08 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Snapshot delet method, processing method, apparatus and system |
| CN114556243A (en) * | 2019-09-20 | 2022-05-27 | 诺信公司 | Flexible mapping with application data identifiers for PLC communications |
-
2015
- 2015-10-30 CN CN201510744661.XA patent/CN105373349A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105930223A (en) * | 2016-04-24 | 2016-09-07 | 湖南大学 | Method for reducing size of check point file |
| CN110018988A (en) * | 2017-11-08 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Snapshot delet method, processing method, apparatus and system |
| CN110018988B (en) * | 2017-11-08 | 2023-04-04 | 阿里巴巴集团控股有限公司 | Snapshot deleting method, processing method, device and system |
| CN114556243A (en) * | 2019-09-20 | 2022-05-27 | 诺信公司 | Flexible mapping with application data identifiers for PLC communications |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10417202B2 (en) | Storage system deduplication | |
| US10387661B2 (en) | Data reduction with end-to-end security | |
| CN102629258B (en) | Repeating data deleting method and device | |
| KR102537119B1 (en) | Logical-to-physical map synchronization in a memory device | |
| CN104360914B (en) | Incremental snapshot method and apparatus | |
| CN102982122A (en) | Repeating data deleting method suitable for mass storage system | |
| WO2007049109A3 (en) | Method and system for compression of logical data objects for storage | |
| CN105373349A (en) | Mass data storage system capable of deleting repetitive data | |
| CN102737270A (en) | Security co-processor of bank smart card chip based on domestic algorithms | |
| CN102073808A (en) | Method for encrypting and storing information through SATA interface and encryption card | |
| CN104463020A (en) | Method for protecting data integrity of memory | |
| CN103955440A (en) | Nonvolatile storage equipment and method of carrying out data manipulation therethrough | |
| US20190073318A1 (en) | Secured Access Control In A Storage System | |
| CN105205416A (en) | Mobile hard disk password module | |
| CN105740733A (en) | Encrypted mobile hard disk and realization method thereof | |
| CN105450704A (en) | Network storage device for flash memories and processing method thereof | |
| CN104463510A (en) | Finance management system | |
| CN104317532A (en) | Multifunctional data destruction machine | |
| CN102999728B (en) | Based on date storage method and the device of safety desktop | |
| CN104036201A (en) | Application-layer file hiding method on Windows operating system | |
| US20220123932A1 (en) | Data storage device encryption | |
| CN204178355U (en) | A kind of multi-functional data destroyer | |
| CN103049388A (en) | Compression managing method and compression managing device of paging memory device | |
| CN204904279U (en) | Storage device with data are from destroying mechanism | |
| CN105573677A (en) | Implementation method of efficient storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160302 |