CN104133643A - Method for improving data transfer efficiency under automatic data hierarchical storage frame - Google Patents
Method for improving data transfer efficiency under automatic data hierarchical storage frame Download PDFInfo
- Publication number
- CN104133643A CN104133643A CN201410378871.7A CN201410378871A CN104133643A CN 104133643 A CN104133643 A CN 104133643A CN 201410378871 A CN201410378871 A CN 201410378871A CN 104133643 A CN104133643 A CN 104133643A
- Authority
- CN
- China
- Prior art keywords
- data block
- migration
- data
- module
- level
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000012544 monitoring process Methods 0.000 claims abstract description 34
- 239000000284 extract Substances 0.000 claims abstract description 18
- 238000013508 migration Methods 0.000 claims description 78
- 230000005012 migration Effects 0.000 claims description 78
- 238000000605 extraction Methods 0.000 claims description 24
- 238000013507 mapping Methods 0.000 claims description 10
- 230000003068 static effect Effects 0.000 claims description 3
- 238000007726 management method Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000002567 autonomic effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method for improving the data transfer efficiency under an automatic data hierarchical storage frame. The method involves a data block IO monitoring module, a data block transfer push module and a data block transfer extract module; the data block IO monitoring module is used for monitoring and counting the IO flow, from a user, of each hierarchy of a whole hierarchical storage pool and monitoring the maximum IO flow, set by the user, of each hierarchy of the hierarchical storage pool; the data block transfer push module is responsible for scanning data blocks of hierarchical equipment where the data block transfer push module is located and scanning counting information from the data block IO monitoring module; the data block transfer extract module is used for extracting IO from a transfer queue of the hierarchical equipment according to an IO quantity extracting threshold value from the data block IO monitoring module. Compared with the prior art, the method enables data block information which is accessed recently to be counted in a memory, the part of the counting information is scanned rapidly, the transfer real-time performance and the transfer efficiency of data are improved, the influence on normal IO is reduced, and the use efficiency of storage equipment is improved.
Description
Technical field
The present invention relates to computer communication technology field, specifically under a kind of automaticdata classification storing framework, improve the method for Data Migration efficiency.
Background technology
In the hierarchical stor being directed to based on piece DBMS, by the data Autonomic Migration Framework of asking without frequentation to lower equipment in carrying cost level, the storage space that discharges higher cost is to more frequently access or the data of higher priority, thereby greatly reduce non-importance data in the shared space of one-level local disk, accelerate the memory property of whole system, reduce the cost that has of whole storage system, and then obtain better cost performance.Classification storage based on data block is a kind of fine-grained data staging Managed Solution, the granularity refinement of data management can be arrived according to the defined extension blocks level of application demand, therefore more accurate for the management of data.In the time there is intensive IO access bottom data, will produce hot spot data, at this moment need the upgrading migration operation of hot spot data, can the better storage resources of usability to realize hot spot data, thus improve storage system overall performance.But, in the existing classification storage migration scheme based on data block, exist problem that the transport efficiency of data block is lower and the migration of data to produce certain negative effect to user's regular traffic, mainly there is following reason:
(1) in the time that certain moment reaches the migration cycle for data, there is the situation of intensive IO access, at this moment in order to ensure the consistance of data, carry out IO when operation can be to the processing that locks of whole classification storage pool, at this moment triggering migration operation can cause operation due to the locking mutual exclusion of data block cannot be completed in time, need the wait next one or multiple migration cycles just can attempt to carry out the updating operation of hot spot data, cause hot spot data to upgrade fast, affected the practical application effect of data staging management; In the time that Data Migration has obtained classification storage pool trivial, can carry out a large amount of Data Migrations, now Data Migration also can occupy certain IO bandwidth of storage pool, and user's regular traffic has been produced to negative performance impact;
(2) while carving the migration cycle reaching for data at a time, often all data blocks in whole storage pool are carried out to traverse scanning, in the time that the data volume in classification storage pool reaches ZB rank, under this kind of scene, the transport efficiency of whole storage pool data is very low.
Summary of the invention
The object of the invention is to overcome the deficiencies in the prior art, a kind of method that improves Data Migration efficiency under automaticdata classification storing framework is provided, the method can be used for, in the automaticdata hierarchical storage management of data intelligence management domain, especially carrying out automatic grading management for the data of block level.
Technical scheme of the present invention realizes in the following manner, and its structure is made up of data block IO monitoring module, data block migration pushing module and data block migration extraction module:
Data block IO monitoring module: whole each level of classification storage pool of monitoring statistics is from user's IO flow, and the maximum IO flow of each level of classification storage pool that supervisory user arranges, provides and extract IO quantity threshold values to data block migration extraction module; The access reference count of each level of monitoring statistics; To adding up from the data block of user IO access, and regular statistical information being notified to data block migration pushing module;
Data block migration pushing module: module instance is responsible for scanning to place level device data block and the scanning from the statistical information of data block IO monitoring module, according to data block recent visit time, reference count, from data block IO monitoring module extract threshold amount, three dimensions judge data block upwards move, move downwards, static, according to the result of judgement, data block is packaged into migration IO request, is pushed in the IO migration queue of corresponding upper layer device or lower floor's equipment;
Data block migration extraction module: according to the extraction IO quantity threshold values from data block IO monitoring module, IO is taken out from the migration queue of this level equipment, distribute the original user data being stored in lower/upper level equipment of new data block storage in this level, these stylish mapping relations are set up, and delete the mapping relations of legacy data piece; Accumulative total this IO flow is to total IO flow that extracts, and in the time that total extraction IO flow of accumulative total exceedes the extraction IO quantity threshold values from data block IO monitoring module, stops epicycle unit and extracts IO migration request, treats that next example operation continues to process; When this level equipment transportation IO queue does not exist migration request, stop epicycle unit and extract IO migration request, treat that next example operation continues to process.
The regular R/W IO flow to from user of above-mentioned data IO monitoring module is added up, the maximum IO flowmeter of each level of the classification storage pool arranging according to user calculates difference between the two, the maximum IO flow using this value as data block migration extraction module.
Above-mentioned data block is moved the data block of the regular scan module example place level equipment of pushing module and the statistical information from data block IO monitoring module, according to the extraction threshold amount of the cold and hot degree of data, data block IO monitoring module, data block is packaged into IO migration request, adds in the migration IO queue of corresponding device.
Above-mentioned data block migration extraction module example extracts the request being stored in the migration IO queue of this level equipment, distribute new data block in this level, original storage of subscriber data that request is carried sets up new mapping relations in newly assigned data block, deletes the mapping relations of legacy data piece.
Advantage of the present invention is:
Under a kind of automaticdata classification storing framework of the present invention, improve the method for Data Migration efficiency compared to the prior art, by the data block information of nearest time access is added up in internal memory, this partial statistical information is scanned fast, the real-time of raising Data Migration and transport efficiency, reduction are for the impact of normal IO, the service efficiency of raising memory device, and the present invention also has the features such as reasonable in design, simple in structure, easy to use, thereby, there is good use value.
Brief description of the drawings
Fig. 1 is the structural representation of automaticdata classification storing framework.
Fig. 2 is storage resources functional module-Data Migration process flow diagram.
Embodiment
Below in conjunction with accompanying drawing, the method that improves Data Migration efficiency under a kind of automaticdata classification storing framework of the present invention is described in detail below.
As shown in Figure 1-2, improve the method for Data Migration efficiency under a kind of automaticdata classification storing framework of the present invention, its structure is made up of data block IO monitoring module, data block migration pushing module and data block migration extraction module:
Data IO monitoring module: whole each level of classification storage pool of regular monitoring is from R/W IO flow in user's unit interval and make statistics, the IO flow of maximum in the unit interval of each level of classification storage pool that supervisory user arranges, these two IO flows are carried out to difference calculating, and using this difference as level equipment I O, migration queue IO flow threshold values is that IO flow threshold values is extracted in data block migration; Access reference count to each level equipment is added up; To adding up from the data block of user IO access, statistical information comprises access reference count, current accessed time, and statistical information is kept at a in internal memory and in disk, preserves a.
Data block migration pushing module: module instance regularly scans to place level device data block and from the statistical information of module (1), according to the access time of data block, access reference count, extract threshold amount, upwards migration of three dimensions judgement, migration, static downwards from data block IO monitoring module, according to the result of judgement, data block is packaged into migration IO request, is pushed in the migration queue of corresponding upper layer device or lower floor's equipment.
Data block migration extraction module: data block migration extraction module example periodic operation, when this level equipment transportation IO queue exists migration request, migration IO request is taken out from the migration queue of this level equipment, distribute the original user data being stored in lower/upper level equipment of new data block storage in this level, these stylish mapping relations are set up, and delete the mapping relations of legacy data piece; Accumulative total this IO flow is to total IO flow that extracts, and in the time that total extraction IO flow of accumulative total exceedes the extraction IO quantity threshold values from data block IO monitoring module, stops epicycle unit and extracts IO migration request, treats that next example operation continues to process.When this level equipment transportation IO queue does not exist migration request, stop epicycle unit and extract IO migration request, treat that next example operation continues to process.
By reference to the accompanying drawings, content of the present invention is described to the process that realizes this architecture with an instantiation.
As described in summary of the invention, main modular of the present invention comprises: data IO monitoring module; Data block migration pushing module; Data block migration extraction module;
These three modules belong to the storage resources subsystem in automaticdata classification storing framework; Embodiment is:
(1) automaticdata hierarchical stor software package is installed to storage system;
(2) above-mentioned three modules are installed in storage resources subsystem as submodule;
(3) activate respectively the example of three modules in each equipment level, provide service by whole automaticdata hierarchical stor for user.
Its processing and fabricating of method that improves Data Migration efficiency under a kind of automaticdata classification storing framework of the present invention is very simple and convenient, can process to specifications shown in accompanying drawing.
Except the technical characterictic described in instructions, be the known technology of those skilled in the art.
Claims (4)
1. under automaticdata classification storing framework, improve a method for Data Migration efficiency, it is characterized in that being formed by data block IO monitoring module, data block migration pushing module and data block migration extraction module:
Data block IO monitoring module: whole each level of classification storage pool of monitoring statistics is from user's IO flow, and the maximum IO flow of each level of classification storage pool that supervisory user arranges, provides and extract IO quantity threshold values to data block migration extraction module; The access reference count of each level of monitoring statistics; To adding up from the data block of user IO access, and regular statistical information being notified to data block migration pushing module;
Data block migration pushing module: module instance is responsible for scanning to place level device data block and the scanning from the statistical information of data block IO monitoring module, according to data block recent visit time, reference count, from data block IO monitoring module extract threshold amount, three dimensions judge data block upwards move, move downwards, static, according to the result of judgement, data block is packaged into migration IO request, is pushed in the IO migration queue of corresponding upper layer device or lower floor's equipment;
Data block migration extraction module: according to the extraction IO quantity threshold values from data block IO monitoring module, IO is taken out from the migration queue of this level equipment, distribute the original user data being stored in lower/upper level equipment of new data block storage in this level, these stylish mapping relations are set up, and delete the mapping relations of legacy data piece; Accumulative total this IO flow is to total IO flow that extracts, and in the time that total extraction IO flow of accumulative total exceedes the extraction IO quantity threshold values from data block IO monitoring module, stops epicycle unit and extracts IO migration request, treats that next example operation continues to process; When this level equipment transportation IO queue does not exist migration request, stop epicycle unit and extract IO migration request, treat that next example operation continues to process.
2. under a kind of automaticdata classification storing framework according to claim 1, improve the method for Data Migration efficiency, it is characterized in that the regular R/W IO flow to from user of data IO monitoring module adds up, the maximum IO flowmeter of each level of the classification storage pool arranging according to user calculates difference between the two, the maximum IO flow using this value as data block migration extraction module.
3. under a kind of automaticdata classification storing framework according to claim 1, improve the method for Data Migration efficiency, it is characterized in that the data block of the regular scan module example place level equipment of data block migration pushing module and the statistical information from data block IO monitoring module, according to the extraction threshold amount of the cold and hot degree of data, data block IO monitoring module, data block is packaged into IO migration request, adds in the migration IO queue of corresponding device.
4. under a kind of automaticdata classification storing framework according to claim 1, improve the method for Data Migration efficiency, it is characterized in that data block migration extraction module example extracts the request being stored in the migration IO queue of this level equipment, distribute new data block in this level, original storage of subscriber data that request is carried sets up new mapping relations in newly assigned data block, deletes the mapping relations of legacy data piece.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410378871.7A CN104133643A (en) | 2014-08-04 | 2014-08-04 | Method for improving data transfer efficiency under automatic data hierarchical storage frame |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410378871.7A CN104133643A (en) | 2014-08-04 | 2014-08-04 | Method for improving data transfer efficiency under automatic data hierarchical storage frame |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104133643A true CN104133643A (en) | 2014-11-05 |
Family
ID=51806334
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410378871.7A Pending CN104133643A (en) | 2014-08-04 | 2014-08-04 | Method for improving data transfer efficiency under automatic data hierarchical storage frame |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104133643A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI608359B (en) * | 2017-01-17 | 2017-12-11 | 關貿網路股份有限公司 | Data transfer system and method thereof |
| CN110209345A (en) * | 2018-12-27 | 2019-09-06 | 中兴通讯股份有限公司 | The method and device of data storage |
| CN112825023A (en) * | 2019-11-20 | 2021-05-21 | 上海商汤智能科技有限公司 | Cluster resource management method and device, electronic equipment and storage medium |
| CN114020828A (en) * | 2021-09-27 | 2022-02-08 | 南京云创大数据科技股份有限公司 | Distributed hierarchical storage system |
| CN114020828B (en) * | 2021-09-27 | 2024-05-31 | 南京云创大数据科技股份有限公司 | Distributed hierarchical storage system |
-
2014
- 2014-08-04 CN CN201410378871.7A patent/CN104133643A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI608359B (en) * | 2017-01-17 | 2017-12-11 | 關貿網路股份有限公司 | Data transfer system and method thereof |
| CN110209345A (en) * | 2018-12-27 | 2019-09-06 | 中兴通讯股份有限公司 | The method and device of data storage |
| CN112825023A (en) * | 2019-11-20 | 2021-05-21 | 上海商汤智能科技有限公司 | Cluster resource management method and device, electronic equipment and storage medium |
| CN114020828A (en) * | 2021-09-27 | 2022-02-08 | 南京云创大数据科技股份有限公司 | Distributed hierarchical storage system |
| CN114020828B (en) * | 2021-09-27 | 2024-05-31 | 南京云创大数据科技股份有限公司 | Distributed hierarchical storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103399942B (en) | A kind of data engine system supporting the many tenants of SaaS and method of work thereof | |
| WO2016165509A1 (en) | Big data statistics at data-block level | |
| US9569481B1 (en) | Efficient locking of large data collections | |
| US9043660B2 (en) | Data store capable of efficient storing of keys | |
| US8229916B2 (en) | Method for massively parallel multi-core text indexing | |
| AU2014212780A1 (en) | Data stream splitting for low-latency data access | |
| CN103886005A (en) | Method and system by using temporary performance objects for enhanced query performance | |
| CN109241178A (en) | The method and apparatus for managing memory space | |
| CN104050042A (en) | Resource allocation method and resource allocation device for ETL (Extraction-Transformation-Loading) jobs | |
| CN106060176A (en) | Cloud computing application architecture and cloud computing service method based on mixed cloud | |
| US9904577B2 (en) | Hybrid heterogeneous host system, resource configuration method and task scheduling method | |
| CN104133643A (en) | Method for improving data transfer efficiency under automatic data hierarchical storage frame | |
| CN102780574A (en) | Service-oriented office data collocation method and device as well as service-oriented office data inspection method and device | |
| CN110647392A (en) | Intelligent elastic expansion method based on container cluster | |
| US9104567B2 (en) | Memory-leak identification | |
| CN105138650A (en) | Hadoop data cleaning method and system based on outlier mining | |
| WO2021230964A1 (en) | Distributed histogram computation framework using data stream sketches and samples | |
| US10860579B2 (en) | Query planning and execution with reusable memory stack | |
| US9760836B2 (en) | Data typing with probabilistic maps having imbalanced error costs | |
| CN102609314A (en) | Quantification management method and quantification management system for virtual machine | |
| CN103150400A (en) | MapReduce-framework-based data screening method | |
| US20130013824A1 (en) | Parallel aggregation system | |
| CN107147531B (en) | CDM cluster website management system | |
| CN109739438A (en) | A kind of big data cloud storage system | |
| EP3201800A1 (en) | Systems and methods for processing graphs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20141105 |