CN107193693A - A kind of online data storage optimization method based on storage system - Google Patents
A kind of online data storage optimization method based on storage system Download PDFInfo
- Publication number
- CN107193693A CN107193693A CN201710369766.0A CN201710369766A CN107193693A CN 107193693 A CN107193693 A CN 107193693A CN 201710369766 A CN201710369766 A CN 201710369766A CN 107193693 A CN107193693 A CN 107193693A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- volume
- group
- storage medium
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1479—Generic software techniques for error detection or fault masking
- G06F11/1489—Generic software techniques for error detection or fault masking through recovery blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses the online data storage optimization method based on storage system, comprise the following steps：1) initial data is issued in the caching of storage, is stored in by memory image copy function in NVRAM；After the completion of mirror image, store to main frame feedback information；2) storage system by LZ4 compression algorithms formed it is data-optimized after different length data block；3) multiple data blocks are merged into data group in logic；4) data group is sequentially written in final storage medium；5) by data in storage medium block M and with all data blocks are organized, deposited by read operation into Installed System Memory；6) by data block and with all data blocks are organized, data store optimization is released, the data during block enters are updated the data；Again data store optimization operation is carried out by the data block after renewal and with group data block by LZ4 compression algorithms；7) be logically combined into new data group, write in sequence in final storage medium and step 4) in the volume that creates.
Description
Technical field
The present invention relates to processing data information technical field, more particularly to a kind of online data storage based on storage system
Optimization method.
Background technology
As digitized multimedia messages especially digital video, the data volume of audio signal are especially huge, for number
According to the continuous magnanimity of amount.It is difficult to be on the actual application if without carrying out it effective data store optimization processing.Therefore,
Data store optimization technology turns into a crucial general character skill in current digital communication, broadcast, storage and multimedia recreation
Art.Various fields pair require more and more higher with memory space optimisation technique.Also high to space optimization technical requirements while
Ensure data storage performance.
The content of the invention
Existed for solving the above-mentioned problems of the prior art, it is an object of the present invention to provide a kind of based on storage system
Line number is according to storage optimization method；The present invention is optimized by effective online data storage, can be reduced magnetic disk media and be repeated
Storage consumption when data are deleted, so as to improve disk utilization rate, and does not influence storage performance.For user, save and deposit
Space is stored up, so as to reduce input cost.The present invention solve technical problem technical scheme be：
A kind of online data storage optimization method based on storage system, comprises the following steps：
1) initial data is issued in the caching of storage by main frame, preserves primary data information (pdi) in the buffer by depositing
Store up in image copying function, the NVRAM for being stored in two storage controls, to ensure the uniformity of data；After the completion of mirror image,
Store to main frame feedback information, show that write operation has been completed；
2) storage system carries out online data storage analysis optimization, the online data storage analysis optimization to initial data
Using LZ4 compression algorithms, by LZ4 compression algorithms formed it is multiple it is data-optimized after different length data block；
3) multiple data blocks of different length are merged into data group in logic；
4) finally data group is sequentially written in final storage medium；
5) by data in storage medium block M and with all data blocks are organized, deposited by read operation into Installed System Memory；
6) by data block and with all data blocks are organized, data store optimization is released, the data during block enters are updated the data；Again
Data store optimization operation is carried out by the data block after renewal and with group data block by LZ4 compression algorithms；
7) new data group is logically combined into, is write in sequence in final storage medium and step 4) middle wound
In the volume built.
The step 4) final storage medium be SSD or disk.
The management of the final storage medium includes herein below：
1) new volume is created in storage；
2) configuration volume essential information：Fill in file label, the description information of volume, volume performance strategy, confirmation application type, encryption work(
Energy configuration, access control policy (unrestricted access and limitation access two kinds), CHAP names and the multiple initiator of permission are visited
The configuration information asked；
3) dimension information of configuration volume：Volume group where the size of volume, volume, functional configuration, the hundred of volume space configuration are deleted again
It is the configuration information of reserved space size point than number；
4) configuration volume Preservation tactics information：Snapshot backup policy wouldn't not be done, existing uniformity group is added, creates one
New volume group, volume individually protected, need to select one of Preservation tactics；
5) operate new volume to create more than to complete, data group is write in storage medium in sequence；Data after optimization
Storage consumption when storage medium carries out data de-duplication can be reduced, so as to improve disk utilization rate.
Beneficial effects of the present invention：
1. the method for the present invention is write buffer power down guarantor safely using the mirror image NVRAM (nonvolatile cache) for having back-up source
Shield, data write common memory and NVRAM simultaneously, and after the completion of doing between the NVRAM of two controllers image copying, mirror image
Just completed to host response write operation, then storage system optimizes processing to the data in internal memory, and online data is deposited
The basis of storage optimisation technique is elongated data block layout, and the LZ4 compression algorithms that online data storage optimisation technique is selected are realized
The quick and low effect of resources costs, therefore the resource occupation such as CPU, internal memory to controller is seldom, the effect of data store optimization
Really good, system is integrally substantially unaffected, it is not necessary to consider influence of the online data optimization processing to performance.By implementing example
Middle step 3,6 effective online data storage optimizations, can reduce storage consumption when storage medium carries out data de-duplication,
So as to improve disk utilization rate, for user, memory space is saved, so as to reduce input cost.
2. the NVRAM benefits used in the present invention：After power is turned off, the data stored are not lost, and improve data
Security；NVRAM is the new storage level of one set up between DRAM and storage medium, excellent performance, reliability superelevation；
There is the processor of a special mobile data in NVRAM, can be moved data to instead of CPU in NVRAM, so as to reduce CPU moneys
The occupancy in source.
3. the medium that the method for the present invention is finally stored, such as SDD or disk, can not only be carried out substantially by managing
Configuration, but also can select related configuration information according to different application, be capable of the different need for meeting user of diversification
Ask, provide the user customized data store optimization method.
4. the present invention is in addition to virtualization desktop VDI environment, data store optimization efficiency in more application environments is better than
Data de-duplication, effective online data storage optimization, can reduce storage when magnetic disk media carries out data de-duplication
Consumption, so as to improve disk utilization rate.
Brief description of the drawings
Fig. 1 is the data store optimization flow chart of invention.
Embodiment
For a better understanding of the present invention, embodiments of the present invention are explained in detail below in conjunction with the accompanying drawings.
With reference to Fig. 1, a kind of online data storage optimization method based on storage system comprises the following steps：1) original number
It is issued to according to by main frame in the caching of storage, preserves primary data information (pdi) in the buffer by memory image copy function,
In the NVRAM for being stored in two storage controls, to ensure the uniformity of data；After the completion of mirror image, store and fed back to main frame
Information, shows that write operation has been completed；
2) storage system carries out online data storage analysis optimization, the online data storage analysis optimization to initial data
Using LZ4 compression algorithms, by LZ4 compression algorithms formed it is multiple it is data-optimized after different length data block.LZ4 compressions are calculated
Method is seldom to resource occupation such as CPU, the internal memories with the small compression algorithm for target of quick and resource overhead, therefore to controller,
It is also small influence, most 3% performance impact to performance；
3) multiple data blocks of different length are merged into data group, i.e. data strip in logic；
4) finally data group is sequentially written in final storage medium.
5) by data in storage medium block M and with all data blocks are organized, deposited by read operation into Installed System Memory；
6) by data block M and with all data blocks are organized, data store optimization is released, the data during block M enters are updated the data；Again
The secondary data block M ' by after renewal and same group data block carry out employing LZ4 pressures in data store optimization operation, same to step (2)
The data store optimization technology of compression algorithm.Storage consumption when storage medium carries out data de-duplication can be reduced, so as to carry
High disk utilization rate, saves memory space and reduces cost；
7) new data group, i.e. data strip are logically combined into.Write in sequence in final storage medium
And step 4) in create volume in.
The step 4) final storage medium be SSD or disk.
The management of the final storage medium includes herein below：
1) new volume is created in storage；
2) configuration volume essential information：Fill in file label, the description information of volume, volume performance strategy, confirmation application type, encryption work(
Energy configuration, access control policy (unrestricted access and limitation access two kinds), CHAP names and the multiple initiator of permission are visited
The configuration information asked；
3) dimension information of configuration volume：Volume group where the size of volume, volume, functional configuration, the hundred of volume space configuration are deleted again
It is the configuration information of reserved space size point than number；
4) configuration volume Preservation tactics information：Snapshot backup policy wouldn't not be done, existing uniformity group is added, creates one
New volume group, volume individually protected, need to select one of Preservation tactics；
5) operate new volume to create more than to complete, data group is write in storage medium in sequence；Data after optimization
Storage consumption when storage medium carries out data de-duplication can be reduced, so as to improve disk utilization rate.
Although above-mentioned the embodiment of invention is described with reference to accompanying drawing, not to the scope of the present invention
Limitation, on the basis of technical scheme, those skilled in the art need not pay creative work and can make
Various modifications or deform still within protection scope of the present invention.
Claims (3)
1. a kind of online data storage optimization method based on storage system, it is characterized in that, comprise the following steps：
1) initial data is issued in the caching of storage by main frame, preserves primary data information (pdi) in the buffer by storing mirror
As copy function, in the NVRAM for being stored in two storage controls, to ensure the uniformity of data；After the completion of mirror image, storage
To main frame feedback information, show that write operation has been completed；
2) storage system carries out online data storage analysis optimization to initial data, and the online data storage analysis optimization is used
LZ4 compression algorithms, by LZ4 compression algorithms formed it is multiple it is data-optimized after different length data block；
3) multiple data blocks of different length are merged into data group in logic；
4) finally data group is sequentially written in final storage medium；
5) by data in storage medium block M and with all data blocks are organized, deposited by read operation into Installed System Memory；
6) by data block and with all data blocks are organized, data store optimization is released, the data during block enters are updated the data；Again will more
Data block and same group data block after new pass through LZ4 compression algorithms and carry out data store optimization operation；
7) be logically combined into new data group, write in sequence in final storage medium and step 4) in create
In volume.
2. a kind of online data storage optimization method based on storage system as claimed in claim 1, it is characterized in that, the step
Rapid final storage medium 4) is SSD or disk.
3. a kind of online data storage optimization method based on storage system as claimed in claim 1, it is characterized in that, it is described most
The management of whole storage medium includes herein below：
1) new volume is created in storage；
2) configuration volume essential information：File label, the description information of volume, volume performance strategy, confirmation application type, encryption function is filled in match somebody with somebody
Put, access control policy (unrestricted access and limitation access two kinds), CHAP names and allow what multiple initiator were accessed
Configuration information；
3) dimension information of configuration volume：Volume group where the size of volume, volume, functional configuration, the percentage of volume space configuration are deleted again
The configuration information for the space size that number is reserved；
4) configuration volume Preservation tactics information：Wouldn't not do snapshot backup policy, add existing uniformity group, create one it is new
Volume group, volume individually protected, need to select one of Preservation tactics；
5) operate new volume to create more than to complete, data group is write in storage medium in sequence；Data after optimization can
Storage consumption when storage medium carries out data de-duplication is reduced, so as to improve disk utilization rate.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710369766.0A CN107193693A (en) | 2017-05-23 | 2017-05-23 | A kind of online data storage optimization method based on storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710369766.0A CN107193693A (en) | 2017-05-23 | 2017-05-23 | A kind of online data storage optimization method based on storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107193693A true CN107193693A (en) | 2017-09-22 |
Family
ID=59874519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710369766.0A Pending CN107193693A (en) | 2017-05-23 | 2017-05-23 | A kind of online data storage optimization method based on storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107193693A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268219A (en) * | 2018-02-01 | 2018-07-10 | 杭州宏杉科技股份有限公司 | A kind of method and device for handling I/O request |
| CN109445693A (en) * | 2018-10-19 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of data compression method and device |
| CN109840053A (en) * | 2019-02-13 | 2019-06-04 | 安徽大尺度网络传媒有限公司 | A kind of online data storage optimization method based on storage system |
| CN114357713A (en) * | 2021-12-03 | 2022-04-15 | 华南理工大学 | Elasticity evaluation method considering system performance influenced duration and water flow velocity |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778018A (en) * | 2015-04-23 | 2015-07-15 | 南京道熵信息技术有限公司 | Broad-strip disk array based on asymmetric hybrid type disk image and storage method of broad-strip disk array |
| CN105117246A (en) * | 2015-08-04 | 2015-12-02 | 晶晨半导体(上海)有限公司 | Method for rapidly booting electronic equipment |
| US20160259561A1 (en) * | 2015-03-04 | 2016-09-08 | Nimble Storage, Inc. | Fibre Channel Initiator Alias/Port Suggestion and Autocomplete |
-
2017
- 2017-05-23 CN CN201710369766.0A patent/CN107193693A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160259561A1 (en) * | 2015-03-04 | 2016-09-08 | Nimble Storage, Inc. | Fibre Channel Initiator Alias/Port Suggestion and Autocomplete |
| CN104778018A (en) * | 2015-04-23 | 2015-07-15 | 南京道熵信息技术有限公司 | Broad-strip disk array based on asymmetric hybrid type disk image and storage method of broad-strip disk array |
| CN105117246A (en) * | 2015-08-04 | 2015-12-02 | 晶晨半导体(上海)有限公司 | Method for rapidly booting electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 架构师技术联盟: "[技术] HPE收购很忙，不如一起来聊聊Nimble Storage", 《搜狐网 HTTPS://WWW.SOHU.COM/A/128369917_632967》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268219A (en) * | 2018-02-01 | 2018-07-10 | 杭州宏杉科技股份有限公司 | A kind of method and device for handling I/O request |
| CN108268219B (en) * | 2018-02-01 | 2021-02-09 | 杭州宏杉科技股份有限公司 | Method and device for processing IO (input/output) request |
| CN109445693A (en) * | 2018-10-19 | 2019-03-08 | 郑州云海信息技术有限公司 | A kind of data compression method and device |
| CN109840053A (en) * | 2019-02-13 | 2019-06-04 | 安徽大尺度网络传媒有限公司 | A kind of online data storage optimization method based on storage system |
| CN114357713A (en) * | 2021-12-03 | 2022-04-15 | 华南理工大学 | Elasticity evaluation method considering system performance influenced duration and water flow velocity |
| CN114357713B (en) * | 2021-12-03 | 2024-05-10 | 华南理工大学 | Elasticity evaluation method considering affected duration of system performance and water flow velocity |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102137761B1 (en) | Heterogeneous unified memory section and method for manaing extended unified memory space thereof | |
| US9921955B1 (en) | Flash write amplification reduction | |
| TWI506421B (en) | System, method, and computer program product for increasing spare space in memory to extend a lifetime of the memory | |
| US8533397B2 (en) | Improving performance in a cache mechanism by way of destaging data in partial strides | |
| TWI494761B (en) | Method of partitioning physical block and memory system thereof | |
| CN107193693A (en) | A kind of online data storage optimization method based on storage system | |
| CN102129434B (en) | Method and system for reading and writing separation database | |
| CN103077121B (en) | The method of data management in a kind of flash memory device and device | |
| CN101727395A (en) | Flash memory device and management system and method thereof | |
| CN103399823B (en) | The storage means of business datum, equipment and system | |
| CN109213420A (en) | Date storage method, apparatus and system | |
| US9569381B2 (en) | Scheduler for memory | |
| CN103049220A (en) | Storage control method, storage control device and solid-state storage system | |
| CN103677674A (en) | Data processing method and device | |
| CN104407936A (en) | Data snapshot method and device | |
| CN109582213A (en) | Data reconstruction method and device, data-storage system | |
| CN105677508A (en) | Method and system for modifying erasure code data in cloud storage | |
| CN103699429A (en) | Virtual machine migration method and virtual machine migration device | |
| CN112632069A (en) | Hash table data storage management method, device, medium and electronic equipment | |
| CN110007870A (en) | A kind of storage equipment write request processing method and relevant apparatus | |
| CN109375868B (en) | Data storage method, scheduling device, system, equipment and storage medium | |
| CN107506466A (en) | A kind of small documents storage method and system | |
| CN104408126B (en) | A kind of persistence wiring method of database, device and system | |
| CN102520879B (en) | Priority-based file information storage method, device and system | |
| US11861175B2 (en) | Latency in data storage systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170922 |
|
| RJ01 | Rejection of invention patent application after publication |