CN105049508A - Cloud data migration method - Google Patents
Cloud data migration method Download PDFInfo
- Publication number
- CN105049508A CN105049508A CN201510429337.9A CN201510429337A CN105049508A CN 105049508 A CN105049508 A CN 105049508A CN 201510429337 A CN201510429337 A CN 201510429337A CN 105049508 A CN105049508 A CN 105049508A
- Authority
- CN
- China
- Prior art keywords
- node
- particle
- subregion
- data migration
- cloud data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/563—Data redirection of data network streams
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a cloud data migration method. The cloud data migration method comprises the following specific realization processes: data is partitioned in a partition form organization in each storage node, namely, data of one storage node can be partitioned into a plurality of partitions; all storage nodes are respectively classified in a migration-in node set In_set and a migration-out node set Out_set according to a relationship of a standardized load value and 1/n, wherein the In_set comprises all the nodes, the load values of which are less than 1/n, and the Out_set comprises all the nodes, the load values of which are more than 1/n; cloud data migration is carried out, namely, partial data in the nodes in the Out_set is migrated onto the nodes in the In_set. Compared with the prior art, the cloud data migration method disclosed by the invention has the beneficial effects that: the average shortest time for migrating each data partition between the nodes is calculated by utilizing a particle swarm optimization algorithm; therefore, global and local searches are controlled; the resource optimization of a system is finally realized; and the cloud data migration method is high in practicability and liable to popularize.
Description
Technical field
The present invention relates to field of data storage, specifically a kind of practical cloud data migration method.
Background technology
The mass data storage possessing high reliability and extensibility is concerning Internet firm being a huge challenge, traditional database is often difficult to meet this demand, and many times for the retrieval of the specific system overwhelming majority be all based on major key inquiry, use relevant database to make inefficiency in this case, and expansion also will become a following very large difficult problem.Under these circumstances, use storage will be one well to select.For the storage system being deployed in cloud environment, Data Migration realizes the node dynamic expansion key technology balanced with elastic load, mainly comprises that migration plan (migrationplan) is formulated, routing iinformation is synchronous, the core content such as user's request forward and data coherence management.A large amount of state synchronized adjoint in data migration process can bring certain influence to systematic function, and therefore, how effectively reducing migration overhead is the problem that cloud service provider need put forth effort to solve.But, storage system have strict low of state, new virtualized environment, user to postpone to require and the unpredictability of access load and time variation bring new challenge to Data Migration.
Existing many a lot of algorithms are in order to solve time and the system resources consumption problem of Data Migration, there is following several method: 1) for the QoS security problem in stores service data migration process, author proposes a kind of method based on FEEDBACK CONTROL, periodically solve the optimum migration rate meeting QoS constraint, the main control problem paying close attention to migration bandwidth.2) based on the Data Migrating Strategy of greedy method, mainly for the data migration problems of Key/Value storage system, basic thought is, adopts statistical method on-line monitoring focus subregion, preferentially the partial data of focus subregion is migrated to the lighter neighbor node of load.For simplifying the complexity of migration operation, hash algorithm can keep the sequencing between Key value.The main deficiency of this two parts work is not consider migration overhead.
For cloud storage system, Data Migration realizes the node dynamic expansion key technology balanced with elastic load.How reducing transit time, reducing Data Migration affects duration to system, and reducing overhead is the problem that cloud service provider need put forth effort to solve.Although existing research work is mostly for the data migration problems under non-virtualized environment, these methods for cloud storage system often and inapplicable.For tackling above-mentioned challenge, data migration problems being included in load-balancing scenario and solving.The invention provides a kind of cloud data migration method, the method works out Data Migrating Strategy based on equilibrium degree, chooses optimum data migration operation, to reach the load balancing of system.
Summary of the invention
Technical assignment of the present invention is for above weak point, provides a kind of practical, cloud data migration method.
A kind of cloud data migration method, its specific implementation process is:
In each memory node, data are split with the form tissue of subregion, and namely the data of a memory node can be divided into some subregions, and subregion is the base unit of Data Migration and load monitoring;
All memory nodes are included into according to the relation of the load value after standardization and 1/n the node set In_set and the node set Out_set that moves out that moves into respectively, In_set comprises the node of all load values lower than 1/n, and Out_set then comprises the node of all load values higher than 1/n;
Carry out cloud Data Migration, to move in In_set on each node by the partial data in node each in Out_set, and the average shortest time of moving between node by calculating each data partition controls the overall situation and Local Search, make the equilibrium being reached system load in cloud environment between each node by Data Migration.
The detailed process of described cloud Data Migration is:
1) first set following parameter, set up storage system model: population, dimension size, the detected value/entropy of the load balancing that maximum iteration time Tmax or needs reach, the size of inertia weight Wmax, Wmix and optimal solution set P;
2) subregion obtained using each node of moving out is as a population, then occur in system that each population of multiple population is relatively independent, information transmission between each subregion is only limitted between the subregion of same population gained, and each population of initialization, calculates the fitness value of each particle;
3) carry out the migration in systems in which of particle according to the fitness value of each particle, follow in transition process, particle is to the minimum node migrates of the minimum fitness value of expense; To each particle, the desired positions Pbest of its adaptive value and its process is made comparisons, if better, then it can be used as current desired positions Pbest;
4) to each particle, its adaptive value and all particle desired positions Gbest are made comparisons, if better, then it can be used as current desired positions Gbest;
5) new speed and the position of particle is obtained according to formula iteration; When occurring that the load value of certain node of moving into reaches threshold values system total load value/nodes, now this node removes moving in set of node automatically, finds optimum and move into node in particle set of node of moving at this moment;
6) end condition: iterations reaches the maximum iteration time of setting or reaches the load balancing degrees detected value/entropy of setting.
The system model set up in described step 1) is:
Whole storage system is N dimension, searches for, forms initial population X by m subregion;
；
The positional information of each particle i represents with N dimensional vector:
；
Each particle i migration velocity be:
；
In above-mentioned formula, m is Population Size, i=1,2,3 ..., m.
Described step 2) in, fitness value is calculated by fitness function, and this function is specially:
Fi(x1，x2，x3)=x1^2+x2^2+x3^2；
Wherein:
X1 represents the throughput of network, i.e. the amount of network transmitting data in the unit interval;
X2 represents the size of partition data amount;
X3 represents the distance of node of the moving out subregion obtained and node of moving into;
I represents node of moving out.
In described step 5), after finding Pbest, Gbest two optimal solutions, the iterative formula that particle i obtains new speed and position is:
Wherein,
the speed being subregion i in kth time iteration and position;
C1 and c2 is Studying factors or claims accelerator coefficient, is respectively regulated to the maximum step-length of Pbest and Gbest direction migration, and reflection subregion individual experience and colony's empirical log, according to the impact of traveling locus, reflect the information interchange between each subregion;
Rand1 and rand2 is the random number between ［ 0,1 ］, increases the randomness of particle flight;
the position of subregion i at individual extreme point;
it is the position of the global extremum point of whole population.
In described step 6), when reaching stopping criterion for iteration termination of iterations step, if the maximal rate restriction vmax of particle: for preventing subregion away from search volume, every one dimension speed of particle should meet [-vmax, + vmax] between, then search volume is defined as interval [-xmax ,+xmax];
As vi >=vmax, make vi=xmax;
As vi≤-vmax, make vi=-xmax.
A kind of cloud data migration method of the present invention, has the following advantages:
A kind of cloud data migration method that the present invention proposes, works out Data Migrating Strategy based on equilibrium degree, chooses optimum data migration operation, to reach the load balancing of system; Be reduced in duration in data migration process, reduce the impact on systematic function, improve load balancing degrees; Effective reduction load tilts, thus reduce the overhead in data migration process, practical, be easy to promote.
Accompanying drawing explanation
Accompanying drawing 1 is realization flow figure of the present invention.
Embodiment
Below in conjunction with the drawings and specific embodiments, the invention will be further described.
The invention provides a kind of cloud data migration method, as shown in Figure 1, its specific implementation process is:
For more effectively identifying hot spot data, reduce Data Migration amount as far as possible, in each memory node, data are split with the form tissue of subregion simultaneously, namely the data of a memory node can be divided into some subregions, and subregion is the base unit of Data Migration and load monitoring;
All memory nodes are included into according to the relation of the load value after standardization and 1/n the node set In_set and the node set Out_set that moves out that moves into respectively, In_set comprises the node of all load values lower than 1/n, and Out_set then comprises the node of all load values higher than 1/n;
Carry out cloud Data Migration, to move in In_set on each node by the partial data in node each in Out_set, and the average shortest time of moving between node by calculating each data partition controls the overall situation and Local Search, make the equilibrium being reached system load in cloud environment between each node by Data Migration.
The detailed process of described cloud Data Migration is:
1) first set following parameter, set up storage system model: population, dimension size, the detected value/entropy of the load balancing that maximum iteration time Tmax or needs reach, the size of inertia weight Wmax, Wmix and optimal solution set P;
2) subregion obtained using each node of moving out is as a population, then occur in system that each population of multiple population is relatively independent, information transmission between each subregion is only limitted between the subregion of same population gained, and each population of initialization, calculates the fitness value of each particle;
3) carry out the migration in systems in which of particle according to the fitness value of each particle, follow in transition process, particle is to the minimum node migrates of the minimum fitness value of expense; To each particle, the desired positions Pbest of its adaptive value and its process is made comparisons, if better, then it can be used as current desired positions Pbest;
4) to each particle, its adaptive value and all particle desired positions Gbest are made comparisons, if better, then it can be used as current desired positions Gbest;
5) new speed and the position of particle is obtained according to formula iteration; When occurring that the load value of certain node of moving into reaches threshold values system total load value/nodes, now this node removes moving in set of node automatically, finds optimum and move into node in particle set of node of moving at this moment;
6) end condition: iterations reaches the maximum iteration time of setting or reaches the load balancing degrees detected value/entropy of setting.
The system model set up in described step 1) is:
Whole storage system is N dimension, searches for, forms initial population X by m subregion;
；
The positional information of each particle i represents with N dimensional vector:
；
Each particle i migration velocity be:
；
In above-mentioned formula, m is Population Size, i=1,2,3 ..., m.
Described step 2) in, fitness value is calculated by fitness function, and this function is specially:
Fi(x1，x2，x3)=x1^2+x2^2+x3^2；
Wherein:
X1 represents the throughput of network, i.e. the amount of network transmitting data in the unit interval;
X2 represents the size of partition data amount;
X3 represents the distance of node of the moving out subregion obtained and node of moving into;
I represents node of moving out.
In described step 5), after finding Pbest, Gbest two optimal solutions, the iterative formula that particle i obtains new speed and position is:
Wherein,
the speed being subregion i in kth time iteration and position;
C1 and c2 is Studying factors or claims accelerator coefficient, is respectively regulated to the maximum step-length of Pbest and Gbest direction migration, and reflection subregion individual experience and colony's empirical log, according to the impact of traveling locus, reflect the information interchange between each subregion;
Rand1 and rand2 is the random number between ［ 0,1 ］, increases the randomness of particle flight;
the position of subregion i at individual extreme point;
it is the position of the global extremum point of whole population.
In described step 6), when reaching stopping criterion for iteration termination of iterations step, be generally set to maximum iteration time Tmax, computational accuracy
or the maximum stagnation step number △ t of optimal solution, in the model, with maximum iteration time Tmax for stopping criterion for iteration.If the maximal rate restriction vmax of particle: for preventing subregion away from search volume, every one dimension speed of particle should meet between [-vmax ,+vmax], then search volume is defined as interval [-xmax ,+xmax];
As vi >=vmax, make vi=xmax;
As vi≤-vmax, make vi=-xmax.
If vmax is too large, perhaps subregion can fly over outstanding region, if this value is too little, then particles possibly cannot detect the region beyond the optimal region of local fully, and are absorbed in local optimum.
Above-mentioned embodiment is only concrete case of the present invention; scope of patent protection of the present invention includes but not limited to above-mentioned embodiment; claims of any a kind of cloud data migration method according to the invention and the those of ordinary skill of any described technical field to its suitable change done or replacement, all should fall into scope of patent protection of the present invention.
Claims (7)
1. a cloud data migration method, is characterized in that, its specific implementation process is,
In each memory node, data are split with the form tissue of subregion, and namely the data of a memory node can be divided into some subregions, and subregion is the base unit of Data Migration and load monitoring;
All memory nodes are included into according to the relation of the load value after standardization and 1/n the node set In_set and the node set Out_set that moves out that moves into respectively, In_set comprises the node of all load values lower than 1/n, and Out_set then comprises the node of all load values higher than 1/n;
Carry out cloud Data Migration, to move in In_set on each node by the partial data in node each in Out_set, and the average shortest time of moving between node by calculating each data partition controls the overall situation and Local Search, make the equilibrium being reached system load in cloud environment between each node by Data Migration.
2. a kind of cloud data migration method according to claim 1, is characterized in that, the detailed process of described cloud Data Migration is:
1) first set following parameter, set up storage system model: population, dimension size, the detected value/entropy of the load balancing that maximum iteration time Tmax or needs reach, the size of inertia weight Wmax, Wmix and optimal solution set P;
2) subregion obtained using each node of moving out is as a population, then occur in system that each population of multiple population is relatively independent, information transmission between each subregion is only limitted between the subregion of same population gained, and each population of initialization, calculates the fitness value of each particle;
3) carry out the migration in systems in which of particle according to the fitness value of each particle, follow in transition process, particle is to the minimum node migrates of the minimum fitness value of expense; To each particle, the desired positions Pbest of its adaptive value and its process is made comparisons, if better, then it can be used as current desired positions Pbest;
4) to each particle, its adaptive value and all particle desired positions Gbest are made comparisons, if better, then it can be used as current desired positions Gbest;
5) new speed and the position of particle is obtained according to formula iteration; When occurring that the load value of certain node of moving into reaches threshold values system total load value/nodes, now this node removes moving in set of node automatically, finds optimum and move into node in particle set of node of moving at this moment;
6) end condition: iterations reaches the maximum iteration time of setting or reaches the load balancing degrees detected value/entropy of setting.
3. a kind of cloud data migration method according to claim 2, it is characterized in that, the system model set up in described step 1) is:
Whole storage system is N dimension, searches for, forms initial population X by m subregion;
；
The positional information of each particle i represents with N dimensional vector:
；
Each particle i migration velocity be:
；
In above-mentioned formula, m is Population Size, i=1,2,3 ..., m.
4. a kind of cloud data migration method according to claim 2, is characterized in that, described step 2) in, fitness value is calculated by fitness function, and this function is specially:
Fi(x1，x2，x3)=x1^2+x2^2+x3^2；
Wherein:
X1 represents the throughput of network, i.e. the amount of network transmitting data in the unit interval;
X2 represents the size of partition data amount;
X3 represents the distance of node of the moving out subregion obtained and node of moving into;
I represents node of moving out.
5. a kind of cloud data migration method according to claim 3 or 4, is characterized in that, in described step 5), after finding Pbest, Gbest two optimal solutions, the iterative formula that particle i obtains new speed and position is:
Wherein,
the speed being subregion i in kth time iteration and position;
C1 and c2 is Studying factors or claims accelerator coefficient, is respectively regulated to the maximum step-length of Pbest and Gbest direction migration, and reflection subregion individual experience and colony's empirical log, according to the impact of traveling locus, reflect the information interchange between each subregion;
Rand1 and rand2 is the random number between ［ 0,1 ］, increases the randomness of particle flight;
the position of subregion i at individual extreme point;
it is the position of the global extremum point of whole population.
6. a kind of cloud data migration method according to claim 2, it is characterized in that, in described step 6), when reaching stopping criterion for iteration termination of iterations step, if the maximal rate restriction vmax of particle: for preventing subregion away from search volume, every one dimension speed of particle should meet between [-vmax ,+vmax], then search volume is defined as interval [-xmax ,+xmax];
As vi >=vmax, make vi=xmax;
As vi≤-vmax, make vi=-xmax.
7. a kind of cloud data migration method according to claim 5, it is characterized in that, in described step 6), when reaching stopping criterion for iteration termination of iterations step, if the maximal rate restriction vmax of particle: for preventing subregion away from search volume, every one dimension speed of particle should meet between [-vmax ,+vmax], then search volume is defined as interval [-xmax ,+xmax];
As vi >=vmax, make vi=xmax;
As vi≤-vmax, make vi=-xmax.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510429337.9A CN105049508A (en) | 2015-07-21 | 2015-07-21 | Cloud data migration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510429337.9A CN105049508A (en) | 2015-07-21 | 2015-07-21 | Cloud data migration method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105049508A true CN105049508A (en) | 2015-11-11 |
Family
ID=54455711
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510429337.9A Pending CN105049508A (en) | 2015-07-21 | 2015-07-21 | Cloud data migration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105049508A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111580959A (en) * | 2020-04-26 | 2020-08-25 | Oppo（重庆）智能科技有限公司 | Data writing method, data writing device, server and storage medium |
| CN117376403A (en) * | 2023-10-08 | 2024-01-09 | 无锡安鑫卓越智能科技有限公司 | Cloud data migration method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103593347A (en) * | 2012-08-14 | 2014-02-19 | 中兴通讯股份有限公司 | Load balancing method and distributed database system |
| CN104536828A (en) * | 2014-12-26 | 2015-04-22 | 湖南强智科技发展有限公司 | Cloud computing task scheduling method and system based on quantum-behaved particle swarm algorithm |
-
2015
- 2015-07-21 CN CN201510429337.9A patent/CN105049508A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103593347A (en) * | 2012-08-14 | 2014-02-19 | 中兴通讯股份有限公司 | Load balancing method and distributed database system |
| CN104536828A (en) * | 2014-12-26 | 2015-04-22 | 湖南强智科技发展有限公司 | Cloud computing task scheduling method and system based on quantum-behaved particle swarm algorithm |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111580959A (en) * | 2020-04-26 | 2020-08-25 | Oppo（重庆）智能科技有限公司 | Data writing method, data writing device, server and storage medium |
| CN111580959B (en) * | 2020-04-26 | 2023-02-28 | Oppo（重庆）智能科技有限公司 | Data writing method, data writing device, server and storage medium |
| CN117376403A (en) * | 2023-10-08 | 2024-01-09 | 无锡安鑫卓越智能科技有限公司 | Cloud data migration method and system |
| CN117376403B (en) * | 2023-10-08 | 2024-05-14 | 上海知享家信息技术服务有限公司 | Cloud data migration method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lu et al. | A load-adapative cloud resource scheduling model based on ant colony algorithm | |
| CN105227645A (en) | A kind of cloud data migration method | |
| CN104104621B (en) | A kind of virtual network resource dynamic self-adapting adjusting method based on Nonlinear Dimension Reduction | |
| Wang et al. | Hybrid pulling/pushing for i/o-efficient distributed and iterative graph computing | |
| CN103916438A (en) | Cloud testing environment scheduling method and system based on load forecast | |
| Gu et al. | A multi-objective fog computing task scheduling strategy based on ant colony algorithm | |
| Li et al. | ASCAR: Automating contention management for high-performance storage systems | |
| CN110597598B (en) | Control method for virtual machine migration in cloud environment | |
| CN106209261B (en) | The mobile data collection method of three-dimensional UASNs based on probability neighborhood grid | |
| Khelghatdoust et al. | GLAP: Distributed dynamic workload consolidation through gossip-based learning | |
| CN105049508A (en) | Cloud data migration method | |
| CN106201658A (en) | A kind of migration virtual machine destination host multiple-objection optimization system of selection | |
| Xu et al. | Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems | |
| Sepúlveda et al. | 3DMIA: A multi-objective artificial immune algorithm for 3D-MPSoC multi-application 3D-NoC mapping | |
| Liu et al. | Optimal site selection of China railway data centers by the PSO algorithm | |
| Li et al. | Joint optimization of auto-scaling and adaptive service placement in edge computing | |
| Majumder et al. | EAAM: Energy-aware application management strategy for FPGA-based IoT-Cloud environments | |
| CN113096396A (en) | Path selection method based on gravitational field theory | |
| CN113377523A (en) | Heterogeneous sensing stream graph partitioning method | |
| Selvarani et al. | Enhanced two stage heuristics algorithm for VM scheduling | |
| Patel et al. | Dynamic load balancing techniques for improving performance in cloud computing | |
| Ray et al. | Adaptive data center network traffic management for distributed high speed storage | |
| CN106231636B (en) | The mobile data collection method of three-dimensional UASNs based on probability neighborhood covering collection | |
| Wang et al. | Data storage algorithms based on clustering in wireless sensor networks | |
| CN105955795B (en) | A kind of virtual machine migration method of Oriented Green cloud computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20151111 |
|
| WD01 | Invention patent application deemed withdrawn after publication |