CN103078933B - A kind of method and apparatus determining data migration time - Google Patents
A kind of method and apparatus determining data migration time Download PDFInfo
- Publication number
- CN103078933B CN103078933B CN201210590171.5A CN201210590171A CN103078933B CN 103078933 B CN103078933 B CN 103078933B CN 201210590171 A CN201210590171 A CN 201210590171A CN 103078933 B CN103078933 B CN 103078933B
- Authority
- CN
- China
- Prior art keywords
- data migration
- cycle
- described data
- migration
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Mobile Radio Communication Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention is applicable to field of Internet communication, provides a kind of method determining data migration time, comprises the following steps: start two-wire journey, monitors the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively; Described two-wire journey, regularly judges that whether described Data Migration condition is satisfied, if not, then judges after waiting for the identical time again, until judge that described Data Migration condition is satisfied, if so, then perform Data Migration.The present invention can according to the different situations of migration, the migration cycle is adjusted, the dynamic change of load in cluster can be adapted to, thus not only can make full use of the space of the one-level storage in hierarchical stor, also can ensure that system adjusts Data Position timely, make the access performance of system reach best.
Description
Technical field
The invention belongs to field of Internet communication, particularly relate to a kind of method and apparatus determining data migration time.
Background technology
Along with the explosive increase of data, the cluster storing mass data arises at the historic moment.Because the data volume stored in cluster is very large, what have reaches PB level, and synchronization may have a large amount of data accesses.Therefore, whether the deposit position of data is reasonable, directly has influence on the access delay of data.
Data Migration, refers to the data mobile between different storage mediums.Complete Data Migration, system needs to consume certain resource (comprising hardware resource, bandwidth etc.).
In the cluster storing mass data, need the situation of Data Migration, the backup/restoration of main systematic upgrading, data, and the dynamic conditioning etc. of data in hierarchical stor.Wherein the upgrading of system is large-scale, disposable; The backup/restoration of data is based on security consideration; Under normal circumstances, the number of times that the migration of this two class occurs is little, and be mostly at a time must to complete.
But the Data Dynamic adjustment in hierarchical stor, its objective is to make the position of data obtain the configuration rationalized.Classification stores, and is according to data activity degree " sixteen principles ": namely the data of 20% are in active state, and the data of 80% are in disabled state, and server is divided into different storage hierarchy by hardware performance, and storage hierarchy is higher, and performance is better.The target of hierarchical stor alive data is stored in the high storage of level, but the liveness of data can change along with the change of time, even if so certain data configuration is once rational, but after a period of time, this configuration also can factor data liveness change and seem outdated, therefore in order to reach the target of data rationalisation configuration, just need to carry out Data Migration.
But migration needs consumes resources, if migration number of times is very few, just cannot give full play to the advantage of hierarchical stor, if migration number of times is too much, system has too much resource for in-fighting, likely reduces system to the service quality of other business.The choose opportunities of therefore moving is very important, should guarantee to play the advantage of hierarchical stor, can not have again too much resource for in-fighting.
At present, the method that two kinds are determined migration opportunity is had.One is fixed-period planning, and namely hierarchical stor after some time, can carry out Data Migration, and do not consider other situations; Another kind is that remaining space monitors method, namely monitors and if remaining space is not enough, then starts migration by the remaining space that one-level stores.
But these two kinds of methods have deficiency: the cycle set by fixed-period planning, if long, be difficult to the advantage playing classification storage; If too short, the resource of system can be frequent for in-fighting.And the load of cluster is in dynamic change, the situation of different application access data is different, and the reasonable period of an all application scenarios of adaptation, be non-existent, that is, fixed-period planning is difficult to the dynamic change of adaptive system load.
Remaining space monitors method, can play effect when having mass data to write in the cluster, but in a lot of situation, can no longer include data write and have digital independent frequently, but the liveness of data is constantly changing in cluster.Under this situation, hierarchical stor just can not carry out Data Migration, does not also accomplish rational data configuration, does not play its due effect.
Therefore, the method need determining migration opportunity needs to rationalize, want the dynamic change of energy adaptive system load on the one hand, on the other hand, data can be made again to deposit on suitable node, the efficiency of system when processing alive data being improved, finally make the access performance of data improve, and the access delay of entirety reducing.
Summary of the invention
The embodiment of the present invention provides a kind of method and apparatus determining data migration time, be intended to solve current cannot the suitable data migration time of Dynamic Selection, both guaranteed to play the advantage of hierarchical stor, do not had again the problem of too much resource for in-fighting.
For this reason, following technical scheme is embodiments provided:
Determine a device for data migration time, comprising:
Monitoring module, is connected with judge module, for starting two-wire journey, monitors the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively;
Judge module, is connected with Data Migration module with described monitoring module respectively, for by described two-wire journey, regularly judges that whether described Data Migration condition is satisfied;
Data Migration module, for performing described Data Migration;
Adjusting module, is connected with described Data Migration module, for adjusting the described Data Migration cycle.
The embodiment of the present invention additionally provides a kind of method determining data migration time, comprising:
A: start two-wire journey, monitors the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively;
B: described two-wire journey, regularly judges that whether described Data Migration condition is satisfied, if not, then judges after waiting for the identical time again, until judge that described Data Migration condition is satisfied, if so, then performs step C;
C: perform described Data Migration;
D: according to the situation of described migration, adjusts the described Data Migration cycle;
E: after described Data Migration period modulation, described two-wire journey reads parameter again, continues next round and monitors.
Compared with prior art, embodiments of the invention tool has the following advantages:
The embodiment of the present invention is by starting two-wire journey, monitor the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively, and regularly judge that whether described Data Migration condition is satisfied, and according to meeting the reason of transition condition, adjust the described Data Migration cycle.Ensure that the data migration time of selection, thus not only can make full use of the space of the one-level storage in hierarchical stor, also can ensure that system adjusts Data Position timely, make the access performance of system reach best.
Accompanying drawing explanation
Fig. 1 is the method flow diagram that first embodiment of the invention provides the opportunity of given data migration really;
Fig. 2 is that second embodiment of the invention provides given data really to move the structure chart of the device on opportunity.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that described herein is only a part of embodiment of the present invention, instead of whole embodiments.Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain under the prerequisite not making creative work, all belongs to the scope of protection of the invention.
Fig. 1 is the method flow diagram that first embodiment of the invention provides the opportunity of given data migration really, for convenience of explanation, illustrate only the part relevant to the embodiment of the present invention.
As shown in Figure 1, the method comprises the following steps:
Step 101, starts two-wire journey, monitors the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively.
Step 102, described two-wire journey, regularly judges that whether described Data Migration condition is satisfied, if not, then judges after waiting for the identical time again, until judge that described Data Migration condition is satisfied, if so, then performs step 103.
Concrete, described described Data Migration condition, comprise space utilization rate that in described hierarchical stor, one-level stores exceed setting threshold or, the Data Migration cycle arrives.
Concrete, when judging that the space utilization rate that in hierarchical stor, one-level stores does not exceed setting threshold, when the Data Migration cycle does not arrive yet, then not trigger data migration, judges after the by the time identical time interval again.When judging that the space utilization rate that in hierarchical stor, one-level stores exceedes setting threshold, or, when the Data Migration cycle arrives, then trigger migration.
Step 103, performs described Data Migration.
Preferably, if trigger migration, it is the migration triggered by which kind of background thread that system can record, and chooses migrating objects, completes migration.
Step 104, according to the situation of described migration, adjusts the described Data Migration cycle.
Concrete, when the space utilization rate that the reason triggering migration is one-level storage in hierarchical stor exceedes setting threshold, that is: due to the migration of one-level storage fragmentation deficiency initiation, illustrate now there is a large amount of data writes, timely adjustment to be made to Data Position, shorten the described Data Migration cycle, as the new described Data Migration cycle.
Concrete, due to the migration that migration approach exhaustion cycle time causes, but but do not have actual data mobile to occur, illustrative system does not have significant read-write operation within this cycle, extends the migration cycle, as the new described Data Migration cycle.
Concrete, due to the migration that migration cycle approach exhaustion causes, have normal data mobile to occur, then original migration cycle is constant.
Preferably, be the described Data Migration cycle, higher limit and lower limit are set.When the result in shortening described Data Migration cycle is lower than described lower limit, using described lower limit as the new described Data Migration cycle, or, when the result in prolongation described Data Migration cycle is higher than described higher limit, using described higher limit as the new described Data Migration cycle.
Step 105, after described Data Migration period modulation, described two-wire journey reads parameter again, continues next round and monitors.
Preferably, after Data Migration period modulation, described two-wire journey reads the Data Migration cycle parameter after renewal again, again monitors whether Data Migration condition reaches.
Based on identical design, second embodiment of the invention also provides a kind of device determining data migration time, and as shown in Figure 2, this device comprises monitoring module 201, judge module 202, Data Migration module 203 and adjusting module 204.
Monitoring module 201 is connected with judge module 202, for starting two-wire journey, monitors the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively.
Judge module 202 is connected with Data Migration module 203 with described monitoring module 201 respectively, for by described two-wire journey, regularly judges that whether described Data Migration condition is satisfied.
Concrete, described Data Migration condition comprise space utilization rate that in described hierarchical stor, one-level stores exceed setting threshold or, the Data Migration cycle arrives.
Concrete, when judging that the space utilization rate that in hierarchical stor, one-level stores does not exceed setting threshold, when the Data Migration cycle does not arrive yet, then not trigger data migration, judges after the by the time identical time interval again.When judging that the space utilization rate that in hierarchical stor, one-level stores exceedes setting threshold, or, when the Data Migration cycle arrives, then trigger migration.
Data Migration module 203 is for performing described Data Migration.
Preferably, if trigger migration, it is the migration triggered by which kind of background thread that system can record, and chooses migrating objects, completes migration.
Adjusting module 204 is connected with described Data Migration module 203, for adjusting the described Data Migration cycle.
Concrete, when the space utilization rate that the reason triggering migration is one-level storage in hierarchical stor exceedes setting threshold, that is: due to the migration of one-level storage fragmentation deficiency initiation, illustrate now there is a large amount of data writes, timely adjustment to be made to Data Position, shorten the described Data Migration cycle, as the new described Data Migration cycle.
Concrete, due to the migration that migration approach exhaustion cycle time causes, but but do not have actual data mobile to occur, illustrative system does not have significant read-write operation within this cycle, extends the migration cycle, as the new described Data Migration cycle.
Concrete, due to the migration that migration cycle approach exhaustion causes, have normal data mobile to occur, then original migration cycle is constant.
Preferably, be the described Data Migration cycle, higher limit and lower limit are set.When the result in shortening described Data Migration cycle is lower than described lower limit, using described lower limit as the new described Data Migration cycle, or, when the result in prolongation described Data Migration cycle is higher than described higher limit, using described higher limit as the new described Data Migration cycle.
Preferably, described adjusting module 204 is connected with described monitoring module 201, after again reading parameter, starts two-wire journey for described two-wire journey, monitor the space utilization rate that in hierarchical stor, arbitrary grade stores and Data Migration cycle respectively, namely return the step 101 in the first embodiment.
Preferably, after Data Migration period modulation, described two-wire journey reads the Data Migration cycle parameter after renewal again, again monitors whether Data Migration condition reaches.
The embodiment of the present invention provides a kind of method and apparatus determining data migration time, be intended to solve current cannot the suitable data migration time of Dynamic Selection, both guaranteed to play the advantage of hierarchical stor, do not have again the problem of too much resource for in-fighting, not only can make full use of the space that the one-level in hierarchical stor stores, also can ensure that system adjusts Data Position timely, make the access performance of system reach best.
It will be appreciated by those skilled in the art that the module in the device in embodiment can carry out being distributed in the device of embodiment according to embodiment description, also can carry out respective change and be arranged in the one or more devices being different from the present embodiment.The module of above-described embodiment can merge into a module, also can split into multiple submodule further.
Through the above description of the embodiments, those skilled in the art can be well understood to the mode that the present invention can add required general hardware platform by software and realize, and can certainly pass through hardware, but in a lot of situation, the former is better execution mode.Based on such understanding, technical scheme of the present invention can embody with the form of software product the part that prior art contributes in essence in other words, this computer software product is stored in a storage medium, comprising some instructions in order to make a station terminal equipment (can be mobile phone, personal computer, server, or the network equipment etc.) perform method described in each embodiment of the present invention.
The above is only the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention; can also make some improvements and modifications, these improvements and modifications also should look protection scope of the present invention.
Claims (5)
1. determine a method for data migration time, it is characterized in that, comprise the following steps:
A: start two-wire journey, monitors the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively;
B: described two-wire journey, regularly judges that whether Data Migration condition is satisfied, if not, then judges after waiting for the identical time again, until judge that described Data Migration condition is satisfied, if so, then performs step C; Wherein, regular described in described step B judges that whether described Data Migration condition is satisfied, comprising:
B1: regularly judge whether the space utilization rate that in described hierarchical stor, one-level stores exceedes setting threshold;
With,
B2: regularly judge whether the described Data Migration cycle arrives;
C: perform described Data Migration;
D: according to the situation of described Data Migration, adjusts the described Data Migration cycle;
E: after described Data Migration period modulation, described two-wire journey reads parameter again, continues next round and monitors.
2. the method for claim 1, is characterized in that, described step D comprises:
During D1:B1, shorten the described Data Migration cycle, as the new described Data Migration cycle;
Or,
During D2:B2, if do not have actual data mobile to occur, then extend the described Data Migration cycle, as the new described Data Migration cycle, if there is normal data mobile to occur, then original described migration cycle is constant.
3. determine the method for data migration time as claimed in claim 2, it is characterized in that, described step D also comprises:
D3: for the described Data Migration cycle arranges higher limit and lower limit;
D4: when the result in shortening described Data Migration cycle is lower than described lower limit, using described lower limit as the new described Data Migration cycle, or, when the result in prolongation described Data Migration cycle is higher than described higher limit, using described higher limit as the new described Data Migration cycle.
4. determine a device for data migration time, it is characterized in that, comprising:
Monitoring module, is connected with judge module, for starting two-wire journey, monitors the space utilization rate that in hierarchical stor, one-level stores and Data Migration cycle respectively;
Judge module, is connected with Data Migration module with described monitoring module respectively, for by described two-wire journey, regularly judges that whether Data Migration condition is satisfied; Described Data Migration condition comprises the space utilization rate that in described hierarchical stor, one-level stores and exceedes setting threshold, or, the Data Migration cycle arrives, when judging that the space utilization rate that in hierarchical stor, one-level stores does not exceed setting threshold, the Data Migration cycle is not when arriving yet, then not trigger data migration, judges after waiting for the identical time again; When judging that the space utilization rate that in hierarchical stor, one-level stores exceedes setting threshold, or, when the Data Migration cycle arrives, then trigger data migration;
Data Migration module, for performing described Data Migration;
Adjusting module, is connected with described Data Migration module, for the situation according to described migration, adjusts the described Data Migration cycle.
5. determine the device of data migration time as claimed in claim 4, it is characterized in that, described adjusting module, be connected with described monitoring module, for after Data Migration period modulation, described two-wire journey reads parameter again, continues next round and monitors.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210590171.5A CN103078933B (en) | 2012-12-29 | 2012-12-29 | A kind of method and apparatus determining data migration time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210590171.5A CN103078933B (en) | 2012-12-29 | 2012-12-29 | A kind of method and apparatus determining data migration time |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103078933A CN103078933A (en) | 2013-05-01 |
| CN103078933B true CN103078933B (en) | 2015-12-02 |
Family
ID=48155335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210590171.5A Active CN103078933B (en) | 2012-12-29 | 2012-12-29 | A kind of method and apparatus determining data migration time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103078933B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657286B (en) * | 2013-11-19 | 2019-05-10 | 中兴通讯股份有限公司 | A kind of classification storage method and device |
| CN103744623A (en) * | 2014-01-10 | 2014-04-23 | 浪潮电子信息产业股份有限公司 | Method for realizing intelligent degradation of data cached in SSD (Solid State Disk) of storage system |
| CN104573020B (en) * | 2015-01-12 | 2017-11-10 | 浪潮电子信息产业股份有限公司 | Automatic data migration optimization method in hierarchical storage system |
| CN106502576B (en) * | 2015-09-06 | 2020-06-23 | 中兴通讯股份有限公司 | Migration strategy adjusting method and device |
| CN106502578B (en) * | 2015-09-06 | 2019-06-11 | 中兴通讯股份有限公司 | Capacity changes suggesting method and device |
| CN107239457A (en) * | 2016-03-28 | 2017-10-10 | 阿里巴巴集团控股有限公司 | Data archiving method and device |
| CN107291752A (en) * | 2016-04-01 | 2017-10-24 | 中国电信股份有限公司 | Data Migration control method and system based on service-aware |
| CN106598500A (en) * | 2016-12-16 | 2017-04-26 | 郑州云海信息技术有限公司 | Hierarchical storage dynamic migration method |
| CN108897707A (en) * | 2018-06-26 | 2018-11-27 | 郑州云海信息技术有限公司 | A kind of capacity statistical method, device and the equipment of multi-controller |
| CN111385328B (en) * | 2018-12-29 | 2024-04-05 | 三六零科技集团有限公司 | Service request processing method, system and electronic equipment |
| CN112860188A (en) * | 2021-02-09 | 2021-05-28 | 山东英信计算机技术有限公司 | Data migration method, system, device and medium |
| CN113741819A (en) * | 2021-09-15 | 2021-12-03 | 第四范式（北京）技术有限公司 | Method and device for hierarchical storage of data |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1751307A (en) * | 2002-01-18 | 2006-03-22 | 伊迪蒂克公司 | A design for storage and retrieval of arbitrary content and application data |
| CN102136003A (en) * | 2011-03-25 | 2011-07-27 | 上海交通大学 | Large-scale distributed storage system |
| CN102456049A (en) * | 2010-10-28 | 2012-05-16 | 无锡江南计算技术研究所 | Data migration method and device, and object-oriented distributed file system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060136525A1 (en) * | 2004-12-21 | 2006-06-22 | Jens-Peter Akelbein | Method, computer program product and mass storage device for dynamically managing a mass storage device |
| CN102291450B (en) * | 2011-08-08 | 2014-01-15 | 浪潮电子信息产业股份有限公司 | Data online hierarchical storage method in cluster storage system |
-
2012
- 2012-12-29 CN CN201210590171.5A patent/CN103078933B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1751307A (en) * | 2002-01-18 | 2006-03-22 | 伊迪蒂克公司 | A design for storage and retrieval of arbitrary content and application data |
| CN102456049A (en) * | 2010-10-28 | 2012-05-16 | 无锡江南计算技术研究所 | Data migration method and device, and object-oriented distributed file system |
| CN102136003A (en) * | 2011-03-25 | 2011-07-27 | 上海交通大学 | Large-scale distributed storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103078933A (en) | 2013-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103078933B (en) | A kind of method and apparatus determining data migration time | |
| CN106814970B (en) | Enhanced multi-stream operation | |
| CN108829352B (en) | User quota method and system for distributed storage system | |
| US9720727B1 (en) | Managing virtual machine migration | |
| CN106533723B (en) | Virtual resource scheduling method, device and system | |
| CN105302497B (en) | A kind of buffer memory management method and system | |
| CN105138276B (en) | Data storage method and data storage system | |
| CN103139302A (en) | Real-time copy scheduling method considering load balancing | |
| CN105245912A (en) | Methods and devices for caching video data and reading video data | |
| CN102752387B (en) | Data storage processing system and data storage handling method | |
| CN103812849A (en) | Local cache updating method and system, client and server | |
| CN104504147B (en) | A kind of resource coordination method of data-base cluster, apparatus and system | |
| CN110633325B (en) | Docker-based database cluster capacity expansion method and device | |
| CN103607428A (en) | Method of accessing shared memory and apparatus thereof | |
| CN103617007A (en) | Method and system for achieving multilevel intelligent storage | |
| CN105007193A (en) | Multi-layer information processing method, system thereof and cluster management node | |
| CN106445409A (en) | Distributed block storage data writing method and device | |
| CN106202082B (en) | Method and device for assembling basic data cache | |
| CN101827120A (en) | Cluster storage method and system | |
| CN105760391B (en) | Method, data node, name node and system for dynamically redistributing data | |
| CN110109865A (en) | A kind of date storage method, device, equipment and readable storage medium storing program for executing | |
| CN102520879B (en) | Priority-based file information storage method, device and system | |
| CN102779058A (en) | Interface data loading device and interface data loading method | |
| CN102455991A (en) | Computer state updating method and device as well as distributed system | |
| EP4379543A1 (en) | Cloud desktop data migration method, service node, management node, server, electronic device, and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230103 Address after: Room 301, No. 235, Kexue Avenue, Huangpu District, Guangzhou, Guangdong 510000 Patentee after: OURCHEM INFORMATION CONSULTING CO.,LTD. Address before: 1068 No. 518055 Guangdong city in Shenzhen Province, Nanshan District City Xili University School Avenue Patentee before: SHENZHEN INSTITUTES OF ADVANCED TECHNOLOGY Effective date of registration: 20230103 Address after: 510000 room 606-609, compound office complex building, No. 757, Dongfeng East Road, Yuexiu District, Guangzhou City, Guangdong Province (not for plant use) Patentee after: China Southern Power Grid Internet Service Co.,Ltd. Address before: Room 301, No. 235, Kexue Avenue, Huangpu District, Guangzhou, Guangdong 510000 Patentee before: OURCHEM INFORMATION CONSULTING CO.,LTD. |