CN109343793B - Data migration method and device - Google Patents
Data migration method and device Download PDFInfo
- Publication number
- CN109343793B CN109343793B CN201811054720.0A CN201811054720A CN109343793B CN 109343793 B CN109343793 B CN 109343793B CN 201811054720 A CN201811054720 A CN 201811054720A CN 109343793 B CN109343793 B CN 109343793B
- Authority
- CN
- China
- Prior art keywords
- migration
- migrated
- server
- maximum
- data segment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
One or more embodiments of the present specification disclose a data migration method and apparatus, which are used to improve the migration success rate and migration performance of a data segment. The method comprises the following steps: when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server; determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount; and migrating the data segment to be migrated to the second server according to the migration strategy.
Description
Technical Field
The present specification relates to the field of data processing technologies, and in particular, to a data migration method and apparatus.
Background
In a traditional cache cluster for solving data balance through a hash slot principle, data segments which should be stored by a single server are often calculated according to the total number of servers and the total number of hash slots, and each server maintains a data segment managed by the server. When a new server is added, the data segment required to be maintained by each server is recalculated, the data segment is adjusted according to the calculation result, after the data segment is migrated, the old server maintains the data segment required to be migrated and the data segment after migration, and the new server maintains the data segment required to be migrated.
In the prior art, data segments maintained by a server are generally migrated in a data equal distribution manner. For example, assuming that the number of cluster servers is n, the number of data segments maintained by each server before migration is h/n, and when the servers need to be expanded, assuming that t servers are added, the number of data segments maintained by each server is h/(n + t), and the number of data segments that need to be migrated by each original server is (h/n) -h/(n + t). In the migration process of the data segments, if the number of the data segments to be migrated increases, the stability index of the server is likely to change obviously, and even the server stops responding.
Disclosure of Invention
One or more embodiments of the present disclosure provide a data migration method and apparatus, so as to improve the migration success rate and migration performance of data segments.
To solve the above technical problem, one or more embodiments of the present specification are implemented as follows:
in one aspect, one or more embodiments of the present specification provide a data migration method, including:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount;
and migrating the data segment to be migrated to the second server according to the migration strategy.
In one embodiment, the preset condition includes at least one of:
the maximum migration amount is less than or equal to a first preset threshold value;
when the data segment to be migrated with the maximum migration amount is migrated, a variation value of a designated parameter related to the first server does not exceed a second preset threshold.
In one embodiment, the determining a migration policy for the data segment to be migrated according to the first number and the maximum migration amount includes:
if the first quantity is smaller than or equal to the maximum migration quantity, determining that the migration strategy is to directly migrate the data segment to be migrated to the second server;
and if the first quantity is greater than the maximum migration quantity, determining the migration times and/or the single migration quantity of the data segment to be migrated according to the first quantity and the maximum migration quantity.
In one embodiment, the determining the number of times of migration and/or the single migration amount of the data segment to be migrated according to the first number and the maximum migration amount includes:
calculating a quotient value and/or a remainder of the first number and the maximum migration amount;
and determining the migration times according to the quotient value and/or the remainder.
In one embodiment, after determining the number of times of migrating the data segment to be migrated according to the first number and the maximum migration amount, the migrating the data segment to be migrated to the second server according to the migration policy includes:
creating a plurality of migration tasks for migrating the data segment to be migrated;
determining a single migration volume corresponding to each migration task, wherein the single migration volume is less than or equal to the maximum migration volume;
and respectively executing the migration tasks according to the single migration amount.
In another aspect, one or more embodiments of the present specification provide a data migration apparatus, including:
the first determining module is used for determining a first number of the data segments to be migrated when the data segments to be migrated in the first server are migrated to the second server; determining the maximum migration volume meeting preset conditions corresponding to the first server;
a second determining module, configured to determine a migration policy for the data segment to be migrated according to the first number and the maximum migration amount; wherein the migration strategy at least comprises the migration times and/or the single migration amount;
and the migration module is used for migrating the data segment to be migrated to the second server according to the migration strategy.
In one embodiment, the preset condition includes at least one of:
the maximum migration amount is less than or equal to a first preset threshold value;
when the data segment to be migrated with the maximum migration amount is migrated, a variation value of a designated parameter related to the first server does not exceed a second preset threshold.
In one embodiment, the second determining module comprises:
a first determining unit, configured to determine that the migration policy is to directly migrate the data segment to be migrated to the second server if the first number is less than or equal to the maximum migration amount;
and a second determining unit, configured to determine, if the first number is greater than the maximum migration volume, the number of times of migration and/or single migration of the data segment to be migrated according to the first number and the maximum migration volume.
In one embodiment, the second determination unit is further configured to:
calculating a quotient value and/or a remainder of the first number and the maximum migration amount;
and determining the migration times according to the quotient value and/or the remainder.
In one embodiment, the migration module comprises:
a creating unit, configured to create a plurality of migration tasks for migrating the data segment to be migrated after determining the number of migrations of the data segment to be migrated according to the first number and the maximum migration amount;
a third determining unit, configured to determine a single migration amount corresponding to each migration task, where the single migration amount is less than or equal to the maximum migration amount;
and the execution unit is used for respectively executing the migration tasks according to the single migration amount.
In yet another aspect, one or more embodiments of the present specification provide a data migration apparatus, including:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount;
and migrating the data segment to be migrated to the second server according to the migration strategy.
In yet another aspect, an embodiment of the present application provides a storage medium for storing computer-executable instructions, where the computer-executable instructions, when executed, implement the following processes:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount;
and migrating the data segment to be migrated to the second server according to the migration strategy.
By adopting the technical scheme of one or more embodiments of the present specification, when the data segment to be migrated in the first server is migrated to the second server, the first number of the data segment to be migrated and the maximum migration amount corresponding to the first server and meeting the preset condition are determined, the migration policy for the data segment to be migrated is further determined according to the first number and the maximum migration amount, and the data segment to be migrated is migrated to the second server according to the migration policy. Wherein, the migration strategy at least comprises the migration times and/or the single migration amount. Therefore, when the data segments are migrated, the technical scheme does not simply finish migrating all the data segments at one time, but fully considers the number of the data segments to be migrated and the maximum migration amount of the server, so that the problems of unstable server performance and even response stop caused by migrating all the data segments at one time can be avoided, and the migration success rate of the data segments is improved.
Furthermore, the technical scheme can create a plurality of migration tasks, and the single migration volume of each migration task is less than or equal to the maximum migration volume of the server, so that the problems of unstable server performance and even response stop caused by migrating the data segments exceeding the maximum migration volume in the same migration task can be avoided, and more migration tasks can be started because the task volume of a single migration task is reduced, thereby improving the migration performance of the data segments.
Drawings
In order to more clearly illustrate one or more embodiments or technical solutions in the prior art in the present specification, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in one or more embodiments of the present specification, and other drawings can be obtained by those skilled in the art without inventive exercise.
FIG. 1 is a schematic flow chart diagram of a method of data migration in accordance with one embodiment of the present description;
FIG. 2 is a schematic flow chart diagram of a method of data migration in accordance with a specific embodiment of the present description;
FIG. 3 is a schematic block diagram of a data migration apparatus in accordance with an embodiment of the present description;
FIG. 4 is a schematic block diagram of a data migration apparatus according to an embodiment of the present description.
Detailed Description
One or more embodiments of the present disclosure provide a data migration method and apparatus, so as to improve the migration success rate and migration performance of a data segment.
In order to make those skilled in the art better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in one or more embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all embodiments. All other embodiments that can be derived by a person skilled in the art from one or more of the embodiments of the present disclosure without making any creative effort shall fall within the protection scope of one or more of the embodiments of the present disclosure.
Fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present specification, as shown in fig. 1, the method including:
s102, when the data segments to be migrated in the first server are migrated to the second server, determining a first number of the data segments to be migrated; and determining the maximum migration volume meeting the preset conditions corresponding to the first server.
And S104, determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity.
Wherein, the migration strategy at least comprises the migration times and/or the single migration amount.
And S106, migrating the data segment to be migrated to the second server according to the migration strategy.
By adopting the technical scheme of one or more embodiments of the present specification, when the data segment to be migrated in the first server is migrated to the second server, the first number of the data segment to be migrated and the maximum migration amount corresponding to the first server and meeting the preset condition are determined, the migration policy for the data segment to be migrated is further determined according to the first number and the maximum migration amount, and the data segment to be migrated is migrated to the second server according to the migration policy. Wherein, the migration strategy at least comprises the migration times and/or the single migration amount. Therefore, when the data segments are migrated, the technical scheme does not simply finish migrating all the data segments at one time, but fully considers the number of the data segments to be migrated and the maximum migration amount of the server, so that the problems of unstable server performance and even response stop caused by migrating all the data segments at one time can be avoided, and the migration success rate of the data segments is improved.
The improved data migration method of the above embodiment is described in detail below.
When the data segments to be migrated in the first server are migrated to the second server, firstly, a first number of the data segments to be migrated is determined, and a maximum migration amount meeting preset conditions corresponding to the first server is determined. Wherein the preset conditions include at least one of (1) - (2):
(1) the maximum migration amount is less than or equal to a first preset threshold value. In other words, if the first preset threshold data segment is migrated, the stability of the performance of the first server should not be affected, and it may be determined whether the stability of the performance of the first server is affected by referring to changes in parameters such as the operating speed, the memory size, and the duration of the response task of the first server during the migration process.
(2) When the maximum migration amount of the data segments to be migrated is migrated, the variation value of the specified parameter related to the first server does not exceed a second preset threshold value. The designated parameter may be any one or more of the operating speed of the first server, the memory size, the duration of the response task, and the like, which may affect the stability of the server.
The preset conditions in the above (1) - (2) can avoid that the stability of the performance of the server is affected due to the large number of data segment transitions to a certain extent.
In one embodiment, the first number of data segments to be migrated may be determined according to the following: if the number of cluster servers is n, the number of data segments maintained by each server before migration is h/n, and when the servers need capacity expansion, if t servers are added, the number of data segments maintained by each server is h/(n + t), the original number of data segments required to be migrated by each server is (h/n) -h/(n + t), that is, the first number of data segments to be migrated is (h/n) -h/(n + t).
After the first quantity of the data segments to be migrated and the maximum migration quantity corresponding to the first server and meeting the preset conditions are determined, a migration strategy for the data segments to be migrated is determined according to the first quantity and the maximum migration quantity. The migration policy may include the number of migrations and/or the amount of single migration.
In one embodiment, if the first number of the data segments to be migrated is less than or equal to the maximum migration amount, the migration policy is determined to directly migrate the data segments to be migrated to the second server. In this embodiment, the first number of the to-be-migrated data segments is less than or equal to the maximum migration amount, which indicates that the stability of the performance of the first server is not affected when all the to-be-migrated data segments are migrated to the second server at one time.
And if the first quantity of the data segments to be migrated is greater than the maximum migration volume, determining the migration times and/or the single migration volume of the data segments to be migrated according to the first quantity and the maximum migration volume. In this embodiment, the first number of the to-be-migrated data segments is greater than the maximum migration amount, and the stability of the performance of the first server is affected when the maximum migration amount of the to-be-migrated data segments are migrated at one time, so that in this case, the to-be-migrated data segments need to be migrated for multiple times, and the single migration amount is less than or equal to the maximum migration amount.
In one embodiment, when determining the migration times of the data segment to be migrated according to the first number and the maximum migration amount, a quotient value and/or a remainder of the first number and the maximum migration amount may be calculated, and then the migration times of the data segment to be migrated may be determined according to the quotient value and/or the remainder. Specifically, if the remainder is zero, the quotient of the first number and the maximum migration amount is the migration frequency of the data segment to be migrated, and the single migration amount is the maximum migration amount; if the remainder is not zero, the value obtained by adding 1 to the quotient of the first number and the maximum migration amount is the migration times of the data segment to be migrated, wherein the single migration amount to be migrated at one time is equal to the remainder, and the single migration amounts to be migrated at other times are the maximum migration amounts.
For example, the number of cluster servers is n, the number of data segments maintained by each server before migration is h/n, and the maximum migration volume of the first server is q. When the server needs to expand capacity, assuming that t servers are added, the number of data segments maintained by each server is h/(n + t), and the number of original data segments that each server needs to migrate is (h/n) -h/(n + t), that is, the first number of data segments to be migrated is (h/n) -h/(n + t).
When q > (h/n) -h/(n + t) or q ═ h/n) -h/(n + t), the (h/n) -h/(n + t) pieces of data to be migrated can be migrated directly to the second server.
When q < (h/n) -h/(n + t), the quotient and/or remainder of the first quantity and the maximum migration quantity is calculated, i.e. the value [ (h/n) -h/(n + t) ]/q is calculated. If the value of [ (h/n) -h/(n + t) ]/q is an integer, namely the remainder is zero, determining that the migration number of the data segment to be migrated is [ (h/n) -h/(n + t) ]/q, and the single migration amount is q; if the value of [ (h/n) -h/(n + t) ]/q is not an integer, that is, the remainder is not zero (assuming that the remainder is x), the migration number of the data segment to be migrated is determined to be [ (h/n) -h/(n + t) ]/q +1, wherein the single migration amount to be migrated at one time is x, and the single migration amounts to be migrated at other times are q.
In one embodiment, when the single migration amount of the data segment to be migrated is determined according to the first number and the maximum migration amount, the single migration amount of the data segment to be migrated may be determined to be a value less than or equal to the maximum migration amount. Optionally, it is determined that the single migration amount of the data segments to be migrated is equal to the maximum migration amount, and then when the data segments to be migrated are migrated, the data segments to be migrated with the maximum migration amount may be migrated each time until all the data segments to be migrated are migrated to the second server.
And after the migration strategy is determined, migrating the data segment to be migrated to the second server according to the migration strategy. Specifically, the data segment to be migrated can be migrated according to the following steps:
firstly, a migration task with a plurality of migration times is created for migrating the data segment to be migrated.
Secondly, determining single migration volume corresponding to each migration task.
Wherein the single migration amount is less than or equal to the maximum migration amount, and the maximum migration amount satisfies at least one of the following preset conditions:
(1) the maximum migration amount is less than or equal to a first preset threshold value. In the method, a first preset threshold may be set with reference to parameters related to the first server, that is, if the first preset threshold is migrated, the stability of the performance of the first server should not be affected, and whether the stability of the performance of the first server is affected may be determined by referring to changes in parameters such as an operating speed, a memory size, and a duration of a response task of the first server in the migration process.
(2) When the maximum migration amount of the data segments to be migrated is migrated, the variation value of the specified parameter related to the first server does not exceed a second preset threshold value. The designated parameter may be any one or more of the operating speed of the first server, the memory size, the duration of the response task, and the like, which may affect the stability of the server.
And finally, respectively executing migration tasks according to the determined single migration amount.
In this step, the migration tasks may be sequentially executed according to a designated order, and specifically, the migration tasks may be sequentially executed according to a creation order of the migration tasks; or the migration tasks are sequentially executed according to the single migration amount corresponding to each migration task, and if the plurality of migration tasks correspond to the same single migration amount, the plurality of migration tasks can be sequentially executed according to a random order.
According to the embodiment, a plurality of migration tasks can be created, and the single migration volume of each migration task is less than or equal to the maximum migration volume of the server, so that the problems of unstable server performance and even response stop caused by migration of the data segments exceeding the maximum migration volume in the same migration task can be solved, and more migration tasks can be started because the task volume of the single migration task is reduced, and the migration performance of the data segments is improved.
Fig. 2 is a schematic flow chart of a data migration method according to a specific embodiment of the present description, as shown in fig. 2, the method including:
s201, when the data segments to be migrated in the first server are migrated to the second server, determining the number of the data segments to be migrated and the maximum migration amount corresponding to the first server.
Wherein, the maximum migration quantity meets at least one of the following preset conditions:
(1) the maximum migration amount is less than or equal to a first preset threshold value. In the method, a first preset threshold may be set with reference to parameters related to the first server, that is, if the first preset threshold is migrated, the stability of the performance of the first server should not be affected, and whether the stability of the performance of the first server is affected may be determined by referring to changes in parameters such as an operating speed, a memory size, and a duration of a response task of the first server in the migration process.
(2) When the maximum migration amount of the data segments to be migrated is migrated, the variation value of the specified parameter related to the first server does not exceed a second preset threshold value. The designated parameter may be any one or more of the operating speed of the first server, the memory size, the duration of the response task, and the like, which may affect the stability of the server.
This step may determine the number of data segments to be migrated as follows: if the number of cluster servers is n, the number of data segments maintained by each server before migration is h/n, and when the servers need capacity expansion, if t servers are added, the number of data segments maintained by each server is h/(n + t), the original number of data segments required to be migrated by each server is (h/n) -h/(n + t), that is, the number of data segments to be migrated is (h/n) -h/(n + t).
S202, judging whether the number of the data segments to be migrated is larger than the maximum migration amount corresponding to the first server or not. If yes, go to S203; if not, go to step S208.
S203, calculating the quotient and/or remainder of the number of the data segments to be migrated and the maximum migration amount.
Based on the calculated number of the data segments to be migrated being (h/n) -h/(n + t), assuming that the maximum migration amount corresponding to the first server is q, the quotient of the number of the data segments to be migrated and the maximum migration amount is [ (h/n) -h/(n + t) ]/q.
And S204, determining the migration times of the data segment to be migrated according to the quotient value and/or the remainder.
Based on the quotient of the number of the data segments to be migrated and the maximum migration amount being [ (h/n) -h/(n + t) ]/q, if the value of [ (h/n) -h/(n + t) ]/q is an integer, namely the remainder is zero, determining the migration number of the data segments to be migrated as [ (h/n) -h/(n + t) ]/q; if the value of [ (h/n) -h/(n + t) ]/q is not an integer, i.e., the remainder is not zero (assuming that the remainder is x), the number of times of migration of the data segment to be migrated can be determined to be [ (h/n) -h/(n + t) ]/q + 1.
S205, creating a plurality of migration tasks for migrating the data segment to be migrated.
And S206, determining the single migration amount corresponding to each migration task.
And the single migration volume is less than or equal to the maximum migration volume corresponding to the first server.
Continuing with the above example, if the migration number of the data segment to be migrated is [ (h/n) -h/(n + t) ]/q, the single migration amount is q; if the migration times of the data segment to be migrated is [ (h/n) -h/(n + t) ]/q +1, the single migration amount to be migrated at one time is x, and the single migration amounts to be migrated at other times are q.
And S207, respectively executing each migration task according to the determined single migration amount.
And S208, directly migrating the data segment to be migrated to the second server.
In this embodiment, when the data segment to be migrated in the first server is migrated to the second server, the number of the data segment to be migrated and the maximum migration amount corresponding to the first server may be determined, and then the migration frequency for the data segment to be migrated may be determined according to the number of the data segment to be migrated and the maximum migration amount, and the data segment to be migrated may be migrated to the second server according to the migration frequency. Therefore, when the data segments are migrated, the technical scheme does not simply finish migrating all the data segments at one time, but fully considers the number of the data segments to be migrated and the maximum migration amount of the first server, so that the problems of unstable performance and even response stop of the first server caused by migrating all the data segments at one time can be avoided, and the migration success rate of the data segments is improved.
In summary, particular embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may be advantageous.
Based on the same idea, the data migration method provided in one or more embodiments of the present specification further provides a data migration apparatus.
Fig. 3 is a schematic flow chart of a data migration apparatus according to an embodiment of the present disclosure, and as shown in fig. 3, the data migration apparatus 300 includes:
a first determining module 310, configured to determine a first number of data segments to be migrated when migrating a data segment to be migrated in a first server to a second server; determining the maximum migration volume meeting the preset conditions corresponding to the first server;
a second determining module 320, configured to determine a migration policy for the data segment to be migrated according to the first number and the maximum migration amount; wherein, the migration strategy at least comprises the migration times and/or the single migration amount;
the migration module 330 is configured to migrate the data segment to be migrated to the second server according to the migration policy.
In one embodiment, the preset conditions include at least one of:
the maximum migration amount is less than or equal to a first preset threshold value;
when the maximum migration amount of the data segments to be migrated is migrated, the variation value of the specified parameter related to the first server does not exceed a second preset threshold value.
In one embodiment, the second determining module 320 includes:
the first determining unit is used for determining that the migration strategy is to directly migrate the data segment to be migrated to the second server if the first quantity is smaller than or equal to the maximum migration quantity;
and the second determining unit is used for determining the migration times and/or single migration of the data segments to be migrated according to the first quantity and the maximum migration volume if the first quantity is greater than the maximum migration volume.
In one embodiment, the second determination unit is further configured to:
calculating a quotient value and/or a remainder of the first number and the maximum migration amount;
the number of migrations is determined based on the quotient and/or remainder.
In one embodiment, the migration module 330 includes:
the creating unit is used for creating a plurality of migration tasks for migrating the data segments to be migrated after determining the migration times of the data segments to be migrated according to the first quantity and the maximum migration quantity;
the third determining unit is used for determining the single migration volume corresponding to each migration task, wherein the single migration volume is less than or equal to the maximum migration volume;
and the execution unit is used for respectively executing the migration tasks according to the single migration amount.
By adopting the device in one or more embodiments of the present specification, when the data segment to be migrated in the first server is migrated to the second server, the first number of the data segment to be migrated and the maximum migration amount corresponding to the first server and meeting the preset condition are determined, and then the migration policy for the data segment to be migrated is determined according to the first number and the maximum migration amount, and the data segment to be migrated is migrated to the second server according to the migration policy. Wherein, the migration strategy at least comprises the migration times and/or the single migration amount. Therefore, when the data segments are migrated, the technical scheme does not simply finish migrating all the data segments at one time, but fully considers the number of the data segments to be migrated and the maximum migration amount of the server, so that the problems of unstable server performance and even response stop caused by migrating all the data segments at one time can be avoided, and the migration success rate of the data segments is improved.
It should be understood by those skilled in the art that the data migration apparatus can be used to implement the foregoing data migration method, and the detailed description thereof should be similar to the foregoing method, and therefore, in order to avoid complexity, no further description is provided herein.
Based on the same idea, one or more embodiments of the present specification further provide a data migration apparatus, as shown in fig. 4. The data migration apparatus may have a large difference due to different configurations or performances, and may include one or more processors 401 and a memory 402, where the memory 402 may store one or more storage applications or data. Wherein memory 402 may be transient or persistent. The application program stored in memory 402 may include one or more modules (not shown), each of which may include a series of computer-executable instructions for the data migration device. Still further, the processor 401 may be configured to communicate with the memory 402 to execute a series of computer-executable instructions in the memory 402 on the data migration device. The data migration apparatus may also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input-output interfaces 405, one or more keyboards 406.
In particular, in this embodiment, the data migration apparatus includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the data migration apparatus, and the one or more programs configured to be executed by the one or more processors include computer-executable instructions for:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount;
and migrating the data segment to be migrated to the second server according to the migration strategy.
Optionally, the preset condition comprises at least one of:
the maximum migration amount is less than or equal to a first preset threshold value;
when the data segment to be migrated with the maximum migration amount is migrated, a variation value of a designated parameter related to the first server does not exceed a second preset threshold.
Optionally, the computer executable instructions, when executed, may further cause the processor to:
if the first quantity is smaller than or equal to the maximum migration quantity, determining that the migration strategy is to directly migrate the data segment to be migrated to the second server;
and if the first quantity is greater than the maximum migration quantity, determining the migration times and/or the single migration quantity of the data segment to be migrated according to the first quantity and the maximum migration quantity.
Optionally, the computer executable instructions, when executed, may further cause the processor to:
calculating a quotient value and/or a remainder of the first number and the maximum migration amount;
and determining the migration times according to the quotient value and/or the remainder.
Optionally, the computer executable instructions, when executed, may further cause the processor to:
after determining the migration times of the data segments to be migrated according to the first quantity and the maximum migration quantity, creating a plurality of migration tasks for migrating the data segments to be migrated, wherein the migration times are a plurality of migration tasks;
determining a single migration volume corresponding to each migration task, wherein the single migration volume is less than or equal to the maximum migration volume;
and respectively executing the migration tasks according to the single migration amount.
One or more embodiments of the present specification also propose a computer-readable storage medium storing one or more programs, the one or more programs including instructions, which when executed by an electronic device including a plurality of application programs, enable the electronic device to perform the above-mentioned data migration method, and in particular to perform:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount;
and migrating the data segment to be migrated to the second server according to the migration strategy.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the various elements may be implemented in the same one or more software and/or hardware implementations in implementing one or more embodiments of the present description.
One skilled in the art will recognize that one or more embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
One or more embodiments of the present specification are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
One or more embodiments of the present description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is merely one or more embodiments of the present disclosure and is not intended to limit the present disclosure. Various modifications and alterations to one or more embodiments described herein will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present specification should be included in the scope of claims of one or more embodiments of the present specification.
Claims (12)
1. A method of data migration, comprising:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount; the single migration amount is less than or equal to the maximum migration amount; the preset conditions include: the variation value of the designated parameter related to the stability of the first server is less than or equal to a preset threshold value;
and migrating the data segment to be migrated to the second server according to the migration strategy.
2. The method of claim 1, the preset conditions comprising at least one of:
the maximum migration amount is less than or equal to a first preset threshold value;
when the data segment to be migrated with the maximum migration amount is migrated, a variation value of a designated parameter related to the first server does not exceed a second preset threshold.
3. The method of claim 1, said determining a migration policy for the data segment to be migrated in accordance with the first number and the maximum migration amount, comprising:
if the first quantity is smaller than or equal to the maximum migration quantity, determining that the migration strategy is to directly migrate the data segment to be migrated to the second server;
and if the first quantity is greater than the maximum migration quantity, determining the migration times and/or the single migration quantity of the data segment to be migrated according to the first quantity and the maximum migration quantity.
4. The method of claim 3, said determining a number of migrations and/or an amount of single migration of the data segment to be migrated in accordance with the first number and the maximum amount of migration, comprising:
calculating a quotient value and/or a remainder of the first number and the maximum migration amount;
and determining the migration times according to the quotient value and/or the remainder.
5. The method according to claim 3 or 4, wherein the migrating the data segment to be migrated to the second server according to the migration policy after determining the number of times of migrating the data segment to be migrated according to the first number and the maximum migration amount comprises:
creating a plurality of migration tasks for migrating the data segment to be migrated;
determining a single migration volume corresponding to each migration task, wherein the single migration volume is less than or equal to the maximum migration volume;
and respectively executing the migration tasks according to the single migration amount.
6. A data migration apparatus, comprising:
the first determining module is used for determining a first number of the data segments to be migrated when the data segments to be migrated in the first server are migrated to the second server; determining the maximum migration volume meeting preset conditions corresponding to the first server;
a second determining module, configured to determine a migration policy for the data segment to be migrated according to the first number and the maximum migration amount; wherein the migration strategy at least comprises the migration times and/or the single migration amount; the single migration amount is less than or equal to the maximum migration amount; the preset conditions include: the variation value of the designated parameter related to the stability of the first server is less than or equal to a preset threshold value;
and the migration module is used for migrating the data segment to be migrated to the second server according to the migration strategy.
7. The apparatus of claim 6, the preset conditions comprising at least one of:
the maximum migration amount is less than or equal to a first preset threshold value;
when the data segment to be migrated with the maximum migration amount is migrated, a variation value of a designated parameter related to the first server does not exceed a second preset threshold.
8. The apparatus of claim 6, the second determination module comprising:
a first determining unit, configured to determine that the migration policy is to directly migrate the data segment to be migrated to the second server if the first number is less than or equal to the maximum migration amount;
and a second determining unit, configured to determine, if the first number is greater than the maximum migration volume, the number of times of migration and/or single migration of the data segment to be migrated according to the first number and the maximum migration volume.
9. The apparatus of claim 8, the second determination unit further to:
calculating a quotient value and/or a remainder of the first number and the maximum migration amount;
and determining the migration times according to the quotient value and/or the remainder.
10. The apparatus of claim 8 or 9, the migration module comprising:
a creating unit, configured to create a plurality of migration tasks for migrating the data segment to be migrated after determining the number of migrations of the data segment to be migrated according to the first number and the maximum migration amount;
a third determining unit, configured to determine a single migration amount corresponding to each migration task, where the single migration amount is less than or equal to the maximum migration amount;
and the execution unit is used for respectively executing the migration tasks according to the single migration amount.
11. A data migration apparatus comprising:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount; the single migration amount is less than or equal to the maximum migration amount; the preset conditions include: the variation value of the designated parameter related to the stability of the first server is less than or equal to a preset threshold value;
and migrating the data segment to be migrated to the second server according to the migration strategy.
12. A storage medium storing computer-executable instructions that, when executed, implement the following:
when a data segment to be migrated in a first server is migrated to a second server, determining a first number of the data segment to be migrated; determining the maximum migration volume meeting preset conditions corresponding to the first server;
determining a migration strategy aiming at the data segment to be migrated according to the first quantity and the maximum migration quantity; wherein the migration strategy at least comprises the migration times and/or the single migration amount; the single migration amount is less than or equal to the maximum migration amount; the preset conditions include: the variation value of the designated parameter related to the stability of the first server is less than or equal to a preset threshold value;
and migrating the data segment to be migrated to the second server according to the migration strategy.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111188799.8A CN113961145A (en) | 2018-09-11 | 2018-09-11 | Data migration method and device |
| CN201811054720.0A CN109343793B (en) | 2018-09-11 | 2018-09-11 | Data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811054720.0A CN109343793B (en) | 2018-09-11 | 2018-09-11 | Data migration method and device |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111188799.8A Division CN113961145A (en) | 2018-09-11 | 2018-09-11 | Data migration method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109343793A CN109343793A (en) | 2019-02-15 |
| CN109343793B true CN109343793B (en) | 2021-09-07 |
Family
ID=65304856
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811054720.0A Active CN109343793B (en) | 2018-09-11 | 2018-09-11 | Data migration method and device |
| CN202111188799.8A Pending CN113961145A (en) | 2018-09-11 | 2018-09-11 | Data migration method and device |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111188799.8A Pending CN113961145A (en) | 2018-09-11 | 2018-09-11 | Data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN109343793B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111857544B (en) * | 2019-04-26 | 2024-05-17 | 富联精密电子(天津)有限公司 | Storage resource management device and management method |
| CN110555014B (en) * | 2019-09-06 | 2022-04-15 | 中国联合网络通信集团有限公司 | Data migration method and system, electronic device and storage medium |
| CN111142788B (en) * | 2019-11-29 | 2021-10-15 | 浪潮电子信息产业股份有限公司 | Data migration method and device and computer readable storage medium |
| CN111930724B (en) * | 2020-10-14 | 2021-03-16 | 腾讯科技（深圳）有限公司 | Data migration method and device, storage medium and electronic equipment |
| CN114546995B (en) * | 2022-04-25 | 2022-07-15 | 孔智科技（徐州）有限公司 | Dynamic data migration method and system based on graph database |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500073A (en) * | 2013-10-08 | 2014-01-08 | 浪潮(北京)电子信息产业有限公司 | Method and device for data block migration |
| CN105242967A (en) * | 2015-09-29 | 2016-01-13 | 上海新储集成电路有限公司 | Method for carrying out data migration on hybrid memory in multi-core system based on DVFS technology |
| CN105843820A (en) * | 2015-01-16 | 2016-08-10 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| CN106293509A (en) * | 2016-07-12 | 2017-01-04 | 乐视控股（北京）有限公司 | Date storage method and system |
| CN106502576A (en) * | 2015-09-06 | 2017-03-15 | 中兴通讯股份有限公司 | Migration strategy method of adjustment, capacity change suggesting method and device |
| CN108469982A (en) * | 2018-03-12 | 2018-08-31 | 华中科技大学 | A kind of online moving method of container |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9720606B2 (en) * | 2010-10-26 | 2017-08-01 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Methods and structure for online migration of data in storage systems comprising a plurality of storage devices |
| US9405355B2 (en) * | 2012-08-21 | 2016-08-02 | Micron Technology, Inc. | Memory operation power management by data transfer time adjustment |
| CN103024034B (en) * | 2012-12-11 | 2016-12-07 | 华为技术有限公司 | A kind of dispatching method and device |
| CN105446790B (en) * | 2014-07-15 | 2019-10-18 | 华为技术有限公司 | A kind of virtual machine migration method and device |
| CN107229657A (en) * | 2016-03-25 | 2017-10-03 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| CN107870728B (en) * | 2016-09-23 | 2021-02-09 | 伊姆西Ip控股有限责任公司 | Method and apparatus for moving data |
| CN108228672B (en) * | 2016-12-22 | 2022-05-03 | 阿里巴巴集团控股有限公司 | Metadata migration method, device, system and equipment |
| CN107391031B (en) * | 2017-06-27 | 2020-05-08 | 北京邮电大学 | Data migration method and device in computing system based on hybrid storage |
-
2018
- 2018-09-11 CN CN201811054720.0A patent/CN109343793B/en active Active
- 2018-09-11 CN CN202111188799.8A patent/CN113961145A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500073A (en) * | 2013-10-08 | 2014-01-08 | 浪潮(北京)电子信息产业有限公司 | Method and device for data block migration |
| CN105843820A (en) * | 2015-01-16 | 2016-08-10 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| CN106502576A (en) * | 2015-09-06 | 2017-03-15 | 中兴通讯股份有限公司 | Migration strategy method of adjustment, capacity change suggesting method and device |
| CN105242967A (en) * | 2015-09-29 | 2016-01-13 | 上海新储集成电路有限公司 | Method for carrying out data migration on hybrid memory in multi-core system based on DVFS technology |
| CN106293509A (en) * | 2016-07-12 | 2017-01-04 | 乐视控股（北京）有限公司 | Date storage method and system |
| CN108469982A (en) * | 2018-03-12 | 2018-08-31 | 华中科技大学 | A kind of online moving method of container |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109343793A (en) | 2019-02-15 |
| CN113961145A (en) | 2022-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109343793B (en) | Data migration method and device | |
| CN110768912B (en) | API gateway current limiting method and device | |
| CN109005125B (en) | Dynamic current limiting method, device and system | |
| CN109658238B (en) | Data processing method and device | |
| CN110287197B (en) | Data storage method, migration method and device | |
| CN106897103B (en) | Method for quickly and thermally updating version of mobile terminal network game | |
| CN109032803B (en) | Data processing method and device and client | |
| CN107578338B (en) | Service publishing method, device and equipment | |
| CN110018878B (en) | Distributed system data loading method and device | |
| CN106445677A (en) | Load balancing method and device | |
| CN108449378B (en) | Information pushing method, device and equipment | |
| CN109145053B (en) | Data processing method and device, client and server | |
| CN115617799A (en) | Data storage method, device, equipment and storage medium | |
| CN109446271B (en) | Data synchronization method, device, equipment and medium | |
| CN111338803A (en) | Thread processing method and device | |
| US10901972B2 (en) | Table partition configuration method, apparatus and system for database system | |
| CN112559118A (en) | Application data migration method and device, electronic equipment and storage medium | |
| CN116304212A (en) | Data processing system, method, equipment and storage medium | |
| CN115033551A (en) | Database migration method and device, electronic equipment and storage medium | |
| CN109345300B (en) | Data statistical method and device | |
| CN109101514A (en) | Data lead-in method and device | |
| CN109753351B (en) | Time-limited task processing method, device, equipment and medium | |
| CN110059075B (en) | Method, device and equipment for migrating database and computer readable medium | |
| CN110659105A (en) | Virtualized resource processing method, device, equipment and storage medium | |
| CN109582938B (en) | Report generation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200924 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200924 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |