WO2018177107A1 - Procédé de migration de données, serveur de migration, et support d'informations - Google Patents

Procédé de migration de données, serveur de migration, et support d'informations Download PDF

Info

Publication number
WO2018177107A1
WO2018177107A1 PCT/CN2018/078398 CN2018078398W WO2018177107A1 WO 2018177107 A1 WO2018177107 A1 WO 2018177107A1 CN 2018078398 W CN2018078398 W CN 2018078398W WO 2018177107 A1 WO2018177107 A1 WO 2018177107A1
Authority
WO
WIPO (PCT)
Prior art keywords
relationship
data
task
migration
relationship chain
Prior art date
Application number
PCT/CN2018/078398
Other languages
English (en)
Chinese (zh)
Inventor
刘军
方锦亮
赵重庆
温伟飞
李良必
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Publication of WO2018177107A1 publication Critical patent/WO2018177107A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the embodiments of the present invention relate to the field of network technologies, and in particular, to a data migration method, a migration server, and a storage medium.
  • PB Packetabyte
  • IDC Internet Data Center
  • the service cluster when it performs service processing, it creates a corresponding computing task for the service and allocates corresponding computing resources for the computing task, and executes the computing task to execute the processing process of the service data. Because the various services are usually related to each other, in order to avoid affecting the related other services when migrating the business data of one service, the data of the service cluster is usually migrated as a whole. After all computing tasks (ie, stopping providing services to all services), all business data is migrated to the new service cluster, and then the computing tasks are reconfigured and the corresponding computing resources are allocated in the new service cluster, after which the reconfigured computing tasks are initiated. This completes the data migration by re-serving all services.
  • all computing tasks ie, stopping providing services to all services
  • the embodiment of the invention provides a data migration method, a migration server and a storage medium, which can solve the problems of the related art.
  • the technical solution is as follows:
  • a data migration method comprising:
  • the computing tasks indicated by the relationship chains that are not migrated among the plurality of relationship chains are normally run.
  • a data migration apparatus comprising:
  • a first acquiring unit configured to acquire, according to a computing task log of the original service cluster, a plurality of relationship chains, where the computing task log is used to record an association relationship between the computing task and the service data in the original service cluster, where each relationship chain is used. For indicating a set of computing tasks and business data having an association relationship;
  • a migration unit configured to sequentially migrate the service data and the computing task indicated by the multiple relationship chains to the target service cluster in units of relationship chains;
  • the computing tasks indicated by the relationship chains that are not migrated among the plurality of relationship chains are normally run.
  • a migration server comprising: a processor and a memory, the memory storing at least one instruction loaded by the processor and executed to:
  • the computing tasks indicated by the relationship chains that are not migrated among the plurality of relationship chains are normally run.
  • a computer readable storage medium stores at least one instruction loaded by a processor and executed to implement a method as performed by a migration server The action performed in .
  • the business data and the calculation task with the association relationship are represented by a relationship chain, so that the relationship chain being migrated will not be the other in the process of data migration in the relationship chain unit.
  • the relationship chain has an impact, and the computing tasks indicated by the relationship chain that has not been migrated can still be normally operated, so as not to affect the normal use of the service indicated by the relationship chain that has not been migrated.
  • FIG. 1A is a schematic diagram of an implementation scenario provided by an embodiment of the present invention.
  • FIG. 1B is a structural diagram of a migration server according to an embodiment of the present invention.
  • 2A is a flowchart of a data migration method according to an embodiment of the present invention.
  • 2B is a schematic diagram of a relationship chain according to an embodiment of the present invention.
  • 2C is a schematic diagram of a relationship chain splitting according to an embodiment of the present invention.
  • 2D is a schematic diagram of a relationship chain splitting according to an embodiment of the present invention.
  • 2E is a schematic diagram of a relationship chain splitting according to an embodiment of the present invention.
  • 2F is a schematic diagram of accessing key service data by a split relationship chain according to an embodiment of the present invention.
  • 2G is a schematic diagram of a process related to a double write table mechanism according to an embodiment of the present invention.
  • 2H is a schematic diagram of a migration state in a relationship chain migration process according to an embodiment of the present invention.
  • FIG. 3 is a block diagram of a data migration apparatus according to an embodiment of the present invention.
  • FIG. 4 is a block diagram of a data migration apparatus according to an embodiment of the present invention.
  • FIG. 1A is a schematic diagram of an implementation scenario of data migration according to an embodiment of the present invention.
  • the implementation scenario includes an original service cluster, a target service cluster, and a migration server.
  • the original service cluster is a service cluster that needs to migrate business data
  • the target service cluster is a service cluster to which service data is migrated.
  • the service cluster may include multiple storage clusters and multiple computing clusters, the storage clusters are used to store business data, and the computing clusters are used to run computing tasks and store related data of computing tasks, such as computing resource size of computing tasks and location of computing resources.
  • the storage cluster and the computing cluster may be deployed on different servers or on the same server, which is not limited in this embodiment.
  • the service cluster when it performs business processing, it creates a corresponding computing task for the service and allocates corresponding computing resources for the computing task, and executes the computing task to execute one or more business processing processes, for example, from The service cluster reads a certain service data, processes the service data, and writes another service data that is output to the service cluster.
  • the computing task has a certain running periodicity, and the running period may be several hours, several days, several weeks, or several months, for example, a computing task with a running period of one hour, and runs every hour.
  • the operation period of the different computing tasks may be the same or different, and the type of the computing task is related to the processing speed of the service data, which is not limited in this embodiment.
  • the service cluster also maintains a data path mapping table, which is used for the correspondence between the service data identifier and the storage path of the service data.
  • the computing task can determine the storage path of the read or written service data through the data path mapping table in the service cluster, thereby completing the process of reading or writing the service data according to the obtained storage path.
  • the business data read by one computing task may be written by other computing tasks, and the business data written by one computing task may be read by other computing tasks, so that there is a relationship between the computing task and the business data. Certain input and output relationships.
  • the migration server is used to migrate the data of the service cluster and manage the data migration process.
  • the migration server can be deployed in the original service cluster or deployed in the target service cluster. Of course, it can also be deployed in the original service cluster. On a different server than the target service cluster that can communicate with both.
  • the migration server needs to migrate data in the original service cluster to the target service cluster, and the migrated data relates to service data and calculation tasks in the original service cluster.
  • the migration server may include multiple modules, each of which plays a different role during the data migration process.
  • FIG. 1B is an architectural diagram of a migration server, which includes multiple functional modules. The following describes the functions of each functional module:
  • the analyzing module is configured to perform the process of acquiring multiple relationship chains according to the computing task log indicated by the following steps 201 to 203; the splitting module is configured to perform the process of splitting the relationship chain indicated by the following step 204; The module is configured to perform the process of consistency checking of the migration subtask and the relationship chain in step 206 below.
  • the migration module is configured to perform the process of the service data migration and the calculation task migration in the following steps 205 to 208, wherein after the migration of the service data indicated by the relationship chain is completed, the migration module executes the storage path of the data path mapping table.
  • the switching process corresponds to step 207.
  • the migration process of the computing task refers to the process of switching the configuration task configuration information, and the configuration information of the computing task can be obtained from the configuration database, and the process corresponds to step 208. If the migrated relationship chain is a split relationship chain, the key service data needs to be synchronized, and the process corresponds to step a in step 206.
  • the data path mapping table synchronization refers to adding a target storage path of the business data migrated to the target service cluster to the path mapping table.
  • the migration server foreground can be used to manage the relationship chain migration process, for example, can display various information of the relationship chain, the connection relationship of each node in the relationship chain, the migration state of the relationship chain in the migration process, and the relationship chain.
  • the configuration library is configured to store configuration information of the computing resource of the computing task, such as the size of the computing resource and the location information.
  • the configuration repository may also store the original storage path of the service data in the original service cluster and the target storage path of the target service cluster.
  • the task relationship chain is used to store multiple relationship chains generated by the analysis module.
  • the migration task library is used to store information about the migration subtasks, such as the number of the migration subtask, the indicated service data, the original storage path and the target storage path of the service data, and the amount of data of the service data.
  • This embodiment is applied to the scenario of service data migration.
  • the owner of the original service cluster opens the service data to the new owner, causing the service data to be migrated to the target service cluster of the new owner, or the size of the original service cluster is already
  • the service needs cannot be met.
  • the target service cluster needs to be deployed in a larger area of the equipment room to migrate the service data to the target service cluster.
  • the original service clusters may be placed in the IDC1 equipment room, and the target service clusters may all be placed in the IDC2 equipment room.
  • the geographic locations of the IDC1 equipment room and the IDC2 equipment room are different.
  • the size of the target service cluster is larger than the size of the original service cluster.
  • the number of servers that can be accommodated in the corresponding IDC2 room is greater than the number of servers that can be accommodated in the IDC1 room.
  • the original service cluster or the target service cluster may be placed in different IDC equipment rooms, which is not limited in this embodiment.
  • FIG. 2A is a flowchart of a data migration method according to an embodiment of the present invention.
  • the method process provided by the embodiment of the present invention includes:
  • the calculation task log is generated in the running process.
  • the calculation task log is used to record the relationship between the computing task and the service data in the original service cluster.
  • the computing task log includes a task identifier of the computing task, a storage path of the business data, an input/output relationship between the computing task and the business data, and other information, and the input and output relationship between the computing task and the business data is used to indicate the input service of the computing task.
  • the data and the output service data, the other information may include the service information to which the task belongs, such as the service identifier, the user information to which the service belongs, and the like.
  • the storage path of the service data may be used to indicate the service data, and the same storage path is used to indicate the same service data, and the calculation task accesses the service data through the storage path of the service data.
  • the migration server can obtain the calculation task log from the original service cluster, and extract multiple input and output records from the calculation task log.
  • the input and output records are used to indicate the task identifier of the computing task, the storage path of the business data, and the input/output relationship between the computing task and the business data. As shown in Table 1, an input and output recording table is shown.
  • the migration server may analyze the plurality of input and output records extracted from the calculation task log to obtain a plurality of relationship chains for indicating the relationship between the computing task and the service data, where the multiple relationship chains are acquired.
  • the process is detailed in steps 202 to 204 below.
  • the migration server adds the same relationship chain identifier to the input and output records having the association relationship
  • the process of adding different relationship chain identifiers to the input and output records having no association relationship may be:
  • Each input and output record is traversed, for the first input and output record currently traversed, if the traversed input and output record includes a second input and output record having an association relationship with the first input and output record, the first input is The output record adds the same relationship chain identifier as the second input-output record; if the traversed input-output record does not include the second input-output record having an association relationship with the first input-output record, the first input-output record is Add a relationship chain identifier that is different from the traversed input and output records.
  • the relationship between the first input/output record and the second input/output record means that the computing task indicated by the first input/output record has an input/output relationship with the service data indicated by the second input/output record, or The business data indicated by an input/output record has an input-output relationship with the computing task indicated by the second input/output record.
  • each input and output record of Table 1 is traversed.
  • a relationship chain identifier 1001 is added for the input and output record, assuming that the input and output records of the current traversal are The second input and output record "task 2, IN, storage path 1", then the "storage path 1" in the first input and output record that has been traversed has an input relationship, and then the input and output records that have been traversed are determined.
  • a relationship chain identifier 1001 identical to the first input and output record is added for the second input and output record.
  • the input and output records of the current traversal are the last input and output records in Table 1 "Task 5, IN, Storage Path 5", and there is no input or output between the service data indicated by "Task 5" and all the input and output records that have been traversed. Relationship, and there is no input-output relationship between the "storage path 5" and the calculated task indicated by all the input and output records that have been traversed. Therefore, the second input-output record is not included in the traversed input-output record, therefore, for the current
  • the last input and output record of the traversal adds a different relationship chain identifier than the traversed input and output record.
  • the different relationship chain identifier may be 1002 or the like. After adding the relationship chain identifier for all input and output records, the relationship list shown in Table 2 can be obtained.
  • the migration server may abstract the input and output records having the same relationship chain identifier into a first relationship chain, where the first relationship chain includes a task node for indicating a computing task and data for indicating service data.
  • the task node includes a task identifier of the computing task in the first relationship chain
  • the data node includes a storage path of the service data.
  • the generated first relationship chain is as shown in FIG. 2B, and the service data and the calculation task indicated by the relationship chain identifier 1001 corresponding to the input and output records are shown in FIG. 2B.
  • the connection from the task node 1 to the data node 1 is used to instruct the computing task 1 to write service data to the storage path 1, that is, the service data is the output data of the computing task 1.
  • the connection from the data node 1 to the task node 2 is used to instruct the computing task 2 to read the service data from the storage path 2, that is, the service data is the input data of the computing task 2.
  • the service data or the computing task indicated by any of the first relationship chains does not have an association relationship with the computing tasks or service data indicated by the other first relationship chains. Therefore, the service data and the calculation task in the original service cluster can be migrated in units of the first relationship chain, and the other first relationship chains are not affected when the service data and the calculation task indicated by one relationship chain are migrated. Indicates the normal operation of the computing task.
  • the time of data migration will be constrained by the amount of data migrated and the bandwidth of the network, and usually the network bandwidth is limited.
  • the migration process is further reduced.
  • the first relationship chain corresponding to the larger amount of data may be further split in this embodiment. For details, refer to step 204.
  • first relationship chain includes the second relationship chain
  • the first threshold may be set by the migration server according to a preset migration time and a network bandwidth of the relationship chain. For example, if the network bandwidth is 2 GB/s (gigabytes per second) and the preset migration time is 2 minutes, the first threshold is A threshold value is at most 120 GB. Of course, the first threshold may also be smaller than the 120 GB to avoid impact on network bandwidth due to unstable network environment.
  • the preset migration time may be preset by the migration server, or may be set according to the service requirement of the user, and is not limited in this embodiment.
  • the migration server may obtain the data volume of the service data indicated by the first relationship chain according to the storage path indicated by the data node in the first relationship chain. If the service data indicated by the first relationship chain exceeds the first threshold, determining that the first relationship chain is a second relationship chain, and determining that the second relationship chain needs to be split, the splitting process may include the following Steps 204a to 204c:
  • Step 204a Acquire weights of multiple data nodes in the second relationship chain.
  • the weight of each data node is used to indicate the degree of association of the data node in the second relationship chain. The higher the weight, the higher the degree of association of the data node.
  • the process of obtaining the weights of the plurality of data nodes may be: for each of the plurality of data nodes, a product of the number of task nodes associated with the data node and the data amount of the service data indicated by the data node, Determined as the weight of the data node.
  • the task node associated with the data node 1 includes task nodes 1 to 4, and the number of task nodes is 4, assuming the data amount of the service data indicated by the data node 1 If it is 100 GB, the data node 1 has a weight of 4*100 equal to 400.
  • first relationship chain splitting is to split the relationship chain with a large amount of data into a relational chain with a small amount of data, and for any data node in the relationship chain, if the data node is associated with the data node The more associated task nodes, the more the number of relational chains that can be split based on the data node, so that the data volume of the service data indicated by each of the split relationship chains is balanced and does not cause The amount of data of a certain relational chain is too large. Therefore, when determining the weight of the data node, it is necessary to consider two factors of the number of task nodes associated with the data node and the amount of data of the service data indicated by the data node.
  • Step 204b Obtain a key data node from a plurality of data nodes according to a sequence of weights from high to low and a position of the plurality of data nodes on the second relationship chain, where the key data node is the first one in the sequence
  • the two relationship chains are split into data nodes of at least two third relation chains.
  • the migration server analyzes each data node according to the order of weights from high to low. For example, the migration server is based on the data node to the second relationship.
  • the chain is pre-split, and the number of the third relationship chain that can be obtained by splitting the second relationship chain is determined. If the number of the third relationship chains obtained by the splitting is less than 2, the order of the weights is from high to low. Performing a pre-split process on the next data node; if the number of the third relationship chains obtained by the splitting is not less than 2, the data node is determined as a key data node, and the second relationship chain is split based on the key data node Minute. After the key data nodes are obtained from the plurality of data nodes, the migration server no longer performs a pre-split process on the data nodes subsequent to the critical data node in the above-mentioned arrangement order.
  • the method for determining the number of the third relationship chains that can be split according to the pre-separation of the second relationship chain by the data node may be: disconnecting the data node and the task node associated with the data node.
  • the process of determining connectivity between nodes other than the data node in the second relationship chain may be: traversing a node other than the data node, for example, arbitrarily selecting one node as a starting point, if Each node can traverse to determine that nodes other than the data node are connected, otherwise, it is determined that nodes other than the data node are disconnected.
  • the foregoing process of pre-splitting the second relationship chain is not a process of actually splitting the second relationship chain, but the migration server assumes that the second relationship chain can be split into multiples based on the corresponding data node.
  • the analysis process of the third relationship chain is not a process of actually splitting the second relationship chain, but the migration server assumes that the second relationship chain can be split into multiples based on the corresponding data node.
  • Step 204c Split, according to the key data node, a plurality of task nodes associated with the key data node in the second relationship chain into the plurality of third relationship chains.
  • the process of splitting the second relationship chain into multiple third relationship chains based on the key data node by the migration server may be classified into the following three cases:
  • the node in which the task node has an association relationship is determined to be the third relationship chain.
  • the key data node is included in each of the third relationship chains. It is still assumed that the key data node in the relationship chain shown in FIG. 2B is the data node 1, as shown in FIG. 2C, which is a plurality of thirds obtained by splitting the relationship chain shown in FIG. 2B based on the data node 1 in this case. Schematic diagram of the relationship chain.
  • the critical data node is determined as a third relationship chain, and each of the plurality of task nodes associated with the critical data node is other than the critical data node and The node with the associated relationship of the task node acts as a third relational chain.
  • the key data node is used alone as a third relationship chain.
  • the key data node is first split from the second relationship chain as a third relationship chain.
  • traversing the task node as a starting point, and determining, as the starting point, all the nodes that can be traversed are associated with the task node. node.
  • the key data node in the relationship chain shown in FIG. 2B is the data node 1, as shown in FIG. 2D, in this case, based on the data node 1, the relationship chain shown in FIG. 2B is split to obtain a plurality of third relationship chains.
  • Schematic diagram It should be noted that FIG. 2B is only shown as an example, and does not represent the actual splitting result.
  • a third relationship chain other than the key data node should include multiple nodes instead of Will only include one task node.
  • the key data node, the at least one task node directly associated with the key data node, and the node associated with the at least one task node are split into a third relationship chain, and the split relationship is The task node and the data node outside the third relationship chain are split into at least one third relationship chain.
  • the task node directly associated with the key data node refers to a task node that is a child node or a parent node of the key service data.
  • the key data node is split into a third relationship chain with at least one task node directly associated with it.
  • the process of splitting the task node and the data node other than the split third relationship chain into at least one third relationship chain, and in addition to the key data node in the second case The process in which the task node has the associated relationship is the same as the process of the third relational chain, and will not be described here.
  • the key data node in the relationship chain shown in FIG. 2B is the data node 1, as shown in FIG. 2E, which is obtained by splitting the relationship chain shown in FIG. 2B based on the data node 1 in this case.
  • a schematic diagram of the third relationship chain is shown in FIG. 2B.
  • the third relationship chain when the third relationship chain is migrated, if the key data node is detected in the relationship chain, the target storage path of the critical data node is written into the data path mapping table.
  • the second case and the third case After the split, the key business data can be copied to the target service cluster.
  • the migration server adds different relationship chain identifiers to the multiple third relationship chains.
  • the second relationship chain is formally split into a plurality of third relationship chains, and the third relationship chain may be the third in order to distinguish the third relationship chain obtained by the splitting from the first relationship chain that is not split.
  • the relationship chain adds a split identifier, which can be reflected in the relationship chain identifier. For example, the first two digits in the relationship chain identifier are used as the split identifier.
  • the format of the relationship chain identifier may be xx_yyyy, where xx is used to represent the split identifier, such as 00 represents the un-separated first relationship chain, and 01 represents the third relationship chain obtained by the split. Where yyyy is used to indicate the number of the relationship chain.
  • the computing task in the process of data migration in the relationship chain, the computing task can still run, and new service data is generated in the running process. Due to the limitation of the network bandwidth, when the relationship chain is too large, it is likely to be generated. The speed of the new business data is greater than the speed of the migration of the business data, which will cause the relationship chain to never be migrated. Therefore, splitting the large relationship chain into small relationship chains can ensure the realization of the relationship chain in the case of normal running computing tasks. Indicates the migration of business data.
  • step 203 and step 204 are processes for generating a plurality of relationship chains according to the association relationship between the calculation task and the service data indicated by the input and output records identified by the same relationship chain, and each relationship chain includes a task for indicating the calculation task.
  • the above steps 202 to 204 are steps of acquiring a plurality of relationship chains according to the calculation task log of the original service cluster.
  • Each relationship chain is used to indicate a set of computing tasks and business data with an associated relationship.
  • the migration server can sequentially migrate the service data and the calculation task indicated by the multiple relationship chains to the target service cluster in units of relationship chains.
  • sequential migration means that data migration can be performed only for one relationship chain at a time, or parallel migration can be performed for several relationship chains.
  • the computing tasks indicated by the relationship chains that are not migrated in the plurality of relationship chains are normally run.
  • the migration process of a relationship chain includes the following steps 205 to 208.
  • multiple migration subtasks may be generated according to multiple service data indicated by the relationship chain, and the process of generating multiple migration subtasks may be:
  • Each of the plurality of service data indicated by the relationship chain performs the following process: determining whether the data volume of the service data is less than a second threshold; if the data volume of the service data is less than the second threshold, generating a migration corresponding to the service data If the data volume of the service data is not less than the second threshold, the service data is divided into multiple sub-service data according to the second threshold, and the traffic sub-task is generated corresponding to each sub-service data.
  • the data volume of each sub-service data is less than a second threshold.
  • the second threshold may be preset or changed by the migration server, which is not limited in this embodiment.
  • the service cluster records the storage time of the service data, and the migration server can determine the generation time of the service data according to the stored storage time.
  • the migration server may add configuration information for each migration sub-task, and the configuration information may include an original storage path and a target storage path of the corresponding service data.
  • the service data shown in this embodiment refers to the service data stored in a storage path.
  • the migration subtask corresponding to the service data is used. Migrate the business data stored in a storage path.
  • the service data indicated by the relationship chain is migrated to the target service cluster according to multiple migration subtasks.
  • the migration server can migrate the business data to the target service cluster according to the original storage path and the target storage path.
  • a plurality of subtasks corresponding to a relationship chain may be executed in sequence or in parallel, which is not limited in this embodiment.
  • the granularity of data migration is reduced, and the multiple migration sub-tasks can be run in parallel, which improves the migration efficiency of the service data indicated by the relationship chain.
  • the computing task indicated by the relationship chain may continue to run. Therefore, the service data stored in the storage path in the relationship chain may occur.
  • Update For the service data stored in a storage path, the service data stored before the relationship chain is generated is referred to as historical service data, and the service data updated after the relationship chain is generated is referred to as new service data.
  • the service data generation time can be performed under a certain storage path from the first to the last. The migration of the service data is performed, that is, the historical service data is preferentially migrated to avoid the problem that the user needs to retransmit the service data when the user changes the service data, thereby reducing the migration efficiency.
  • this embodiment further provides a data verification mechanism for the migration subtask, and the data verification process may be: for each of the multiple migration subtasks, the service data corresponding to the migration subtask is all After the migration to the target service cluster, the service data corresponding to the migration sub-task in the target service cluster and the original service cluster is checked for consistency. If the consistency check is successful, the service data corresponding to the migration sub-task is successfully migrated. If the consistency check fails, it is determined that the migration of the service data corresponding to the migration subtask fails, and the migration subtask is re-executed. It should be noted that the configuration information of each migration sub-task may further include the data volume size of the corresponding service data.
  • the migration server detects that the data volume of the service data migrated to the target service cluster reaches the migration.
  • the amount of data indicated by the subtask is determined, it is determined that the service data corresponding to the migration subtask has all migrated to the target service cluster.
  • the scope of the consistency check on the service data corresponding to the migration sub-task includes: verification of the data volume of the service data, verification of the number of files included in the service data, and verification of the data content of the service data.
  • the migration server may perform a consistency check on the service data corresponding to the migration sub-task by using a preset algorithm, and the preset algorithm may be preset.
  • the preset algorithm may be a CRC (Cyclic Redundancy Check). Code) verification algorithm.
  • the timing of re-executing the migration sub-task may be performed immediately after determining that the consistency check of the corresponding service data fails, or may be performed after a preset time period after determining that the consistency check of the corresponding service data fails, and may also be performed.
  • the migration sub-tasks that are failed to be re-executed are re-executed, which is not limited in this embodiment.
  • the fine-grained verification of the service data is implemented, so that when the service data migration is in error, the data can be re-migrated at the granularity of the migration subtask, compared with the prior art.
  • the business data migration error occurs, all business data needs to be re-migrated, which reduces the cost of data errors during the migration process and improves the efficiency of data migration.
  • the related computing tasks are not stopped during the entire process of migrating the relationship chain, but in the service data. After migrating to a certain level, the computing task is migrated during the period in which the computing task stops running to minimize the time when the computing task stops running. In the process of migrating the service data indicated by the relationship chain, the following steps 206a to 206d may also be performed.
  • Step 206a In the process of migrating the service data indicated by the relationship chain, obtain a migration progress of the service data indicated by the relationship chain.
  • the migration server may obtain the migration progress of the service data indicated by the relationship chain according to the total data volume of the service data indicated by the relationship chain and the migrated data volume of the relationship chain service data.
  • the migration progress can be expressed as a ratio between the amount of migrated data and the total amount of data. For example, if the total data volume of the service data indicated by the relationship chain is 100 GB, and the migrated data volume is 60 GB, the relationship chain can be determined. Indicates that the migration progress of business data is 60%.
  • step 206b when the migration progress of the service data exceeds the preset progress, it is determined whether the computing task is in the stopped running state for each computing task indicated by the relationship chain, and if the computing task is in the stopped running state, step 206c is performed. If the computing task is in a running state, step 206d is performed.
  • the preset progress may be preset or modified by the migration server.
  • the preset progress may also be dynamically adjusted by the migration server according to the network bandwidth. For example, when the migration server detects that the network bandwidth is reduced, the preset may be appropriately increased. Set the value of the progress to reduce the time spent on the task of the migration task with the greatest possible reduction.
  • Step 206c If the computing task is in a stopped running state, maintaining the stopped running state of the computing task before the relationship chain completes the migration.
  • Step 206d If the computing task is in the running state, wait for the computing task to stop running, and maintain the stopped running state of the computing task before the relationship chain completes the migration.
  • the process of maintaining the stop running state of the computing task in step 206c and step 206d may be referred to as a freeze computing task process.
  • the message of the frozen computing task may be displayed to the enterprise user through the migration server before the computing task is frozen, and the process of freezing the computing task is performed after the enterprise user confirms the freezing.
  • the migration server may perform consistency check on the service data indicated by the relationship chain in a relationship chain unit, and the process may be performed.
  • the consistency check is performed on the service data indicated by the relationship chain in the target service cluster and the original service cluster; if the consistency check is successful, the subsequent steps 207 and 208 are performed; if the consistency check fails, according to The result of the consistency check determines the service data of the migration failure indicated by the relationship chain, and re-migrates the service data that failed to be migrated.
  • the consistency check of the service data may be performed one by one for each migration subtask, or may be performed for each storage path in the relationship chain.
  • the service data corresponding to the migration sub-task or the storage path is the service data of the migration failure.
  • the migration server can use the corresponding original migration sub-task or re-establish the migration sub-task to re-migrate the service data that failed to be migrated.
  • the specific migration process is the same as the above-mentioned data migration process based on the migration sub-task. Narration.
  • the migration server can distinguish whether the migrated relationship chain is the first relationship chain that has not been split or the third relationship chain that is obtained by splitting according to the relationship chain identifier.
  • the migration server may add a specified identifier to the key data node in each third relationship chain. The specified identifier is used to identify whether the key relationship is included in the migrated relationship chain, thereby determining whether the migrated relationship chain is the third relationship chain.
  • the multiple third relationships can be guaranteed in the process of performing data migration according to the third relationship chain.
  • the key service data indicated by the key shared data node is synchronized.
  • a double write table mechanism is adopted, and two storage paths of the key service data are stored in the data path mapping table, and one is a target storage in the target service cluster.
  • the path and the original storage path in the original service cluster may be: obtaining a target storage path of the key service data in the target service cluster, the key service data is the service data indicated by the key data node; and the data path mapping table Add the target storage path and keep the original storage path of the key business data in the original service cluster.
  • the process of adding the target storage path to the data path mapping table may be performed after the relationship chain is split, or may be performed before the multiple third relationship chain migration, which is not limited in this embodiment.
  • the process of migrating the third relationship chain includes the following steps. a to step c:
  • Step a Synchronize key service data in the target service cluster and the original service cluster according to the target storage path and the original storage path.
  • the migration server When detecting the update of the key service data of the original service cluster or the target service cluster, the migration server synchronizes the key service data in the target service cluster and the original service cluster according to the target storage path and the original storage path.
  • Step b If the service data and the calculation task indicated by the third relationship chain have all migrated to the target service cluster, when the computing task indicated by the third relationship chain is run, the target storage path is recorded according to the data path mapping table. Key business data.
  • Step c If the service data and the calculation task indicated by the third relationship chain are not all migrated to the target service cluster, when the computing task indicated by the third relationship chain is run, the original storage path is accessed according to the data path mapping table. Business data.
  • the storage path of the key service data in the corresponding service cluster may be obtained from the data path mapping table according to the identifier of the service cluster where the data indicated by the third relationship chain is located. For example, if the data indicated by the third relationship chain is in the original service cluster, that is, the data indicated by the third relationship chain has not been successfully migrated to the target service cluster, the computing task indicated by the third relationship chain is executed.
  • the original storage path of the key business data is obtained from the data path mapping table, and the key service data is accessed through the original storage path.
  • 2F is a schematic diagram of accessing key service data in the process of migrating the third relational chain obtained by splitting according to the relationship chain shown in FIG. 2B, wherein the data node 1 corresponds to key service data, and the task node 1
  • the third relationship chain has been migrated to the target service cluster, and the third relationship chain where task nodes 2 to 4 are located has not been migrated to the target service cluster.
  • the computing task indicated by the task node 1 accesses the key service data through the target storage path of the key service data, and the task nodes 2 to 4 access the key service data through the original storage path of the key service data.
  • the process involved in the double-write table mechanism includes the following process ( 1) to (4):
  • This process corresponds to the process of obtaining key data nodes in the second relationship chain.
  • the storage path of the key service data in the corresponding service cluster is obtained from the data path mapping table.
  • the computing task indicated by the third relationship chain can access the key service data in the target service cluster, that is, the third relationship chain is released.
  • the dependency of the original storage path of critical business data is released.
  • the migration of the data indicated by the relationship chain further includes the migration of the source data of the service
  • the source data includes data input by the user at the user terminal
  • the real-time generation of the user terminal is not synchronized to the original service cluster.
  • the data is typically used by computing tasks.
  • the source data may be obtained from a real-time data processing server through a specified interface, and the source data and the service data indicated by the relationship chain are migrated together to the target service cluster so as not to affect the normality of the computing task. run.
  • the migration server may record the target storage path of each service data indicated by the relationship chain, and all the service data indicated by the relationship chain are migrated to the target service cluster. For each service data, the migration server may replace the original storage path of the service data with the target storage path of the service data in the data path mapping table.
  • the migration server determines that the service data corresponding to all the third relationship chains related to the key service data is migrated to the target service cluster, from the data path mapping table. Delete the original storage path of the key business data and retain the target storage path of the key business data.
  • the process of migrating the computing task indicated by the relationship chain to the target service cluster may be: acquiring the first computing resource information and the second computing resource information of the computing task, and replacing the first computing resource information of the computing task with The second computing resource information.
  • the first computing resource information is computing resource information configured for the computing task in the original service cluster
  • the second computing resource information is computing resource information configured for the computing task in the target service cluster.
  • the migration server starts all computing tasks indicated by the relationship chain, thereby completing the migration of the relationship chain.
  • the embodiment may also implement incremental migration of data, and the incremental migration includes the following two layers:
  • the first level the migration of new data during the migration process.
  • the migration server may record the time stamp of the latest input and output record in the calculation task log after acquiring multiple relationship chains according to the calculation task log.
  • the migration server can obtain the new input and output records generated after the time stamp is obtained from the original service cluster calculation task log according to the time stamp of the record.
  • the migration server can update the relationship chain that has not been migrated according to the newly added input and output records, and the process can be: adding any input and output records to any of the items, if the relationship chain is not migrated, the new input is included. Outputting the associated fourth relationship chain, and updating the fourth relationship chain according to the newly added input and output record; if the fourth relationship chain is not included in the relationship chain that has not been migrated, according to the newly added input and output The relationship between the record and other newly added input and output records is generated, and a new relationship chain is generated.
  • the process of generating a new relationship chain is the same as the process of generating multiple relationship chains, and will not be described here.
  • the fourth relationship chain associated with the newly added input and output record means that the service data indicated by the fourth relationship chain is associated with the computing task indicated by the newly added input and output record, or is the fourth The computing task indicated by the relationship chain has an association relationship with the business data indicated by the newly added input and output record.
  • the migration server may perform the step of updating the relationship chain that has not been migrated in the process of the relationship chain being migrated, or may be performed after the completion of a relationship chain migration.
  • the embodiment does not limit this.
  • the migration server can periodically obtain new input and output records in the calculation task log to periodically update the relationship chain that has not been migrated.
  • the new computing task generated in the original service cluster may have an association relationship with the service data indicated by the migrated relationship chain, and therefore, when the relationship chain indicates After the data is migrated to the target service cluster, the new computing task needs to read and write the associated service data from the target service cluster, and the service data is not in the same IDC room as the original service cluster. The read and write will occupy a large network bandwidth. Therefore, in the first level, the migration server can update the relationship chain that has not been migrated according to the calculation task log in time, so that the relationship chain can add a comprehensive indication of the original service.
  • the migration server can also monitor the network bandwidth usage of all computing tasks in the original service cluster, and the network bandwidth usage is higher than the pre- For the bandwidth calculation task, the migration server preferentially migrates the relationship chain where the computing task is located to the target service cluster.
  • the breakpoint is resumed based on the data migration state at the time of the interruption.
  • the process of resuming the breakpoint may be: when the migration based on any one of the relationship chains is detected, when the migration interruption operation of the relationship chain is detected, the migration subtask of the uncompleted migration is recorded, and the migration process of the relational chain is stopped; When the continuation migration operation of the relationship chain is detected, the business data and the calculation task indicated by the relationship chain are migrated to the target service cluster according to the migration subtask of the uncompleted migration.
  • the migration server may record the number of the migration subtasks and perform the migration in the order of numbering. For different migration subtasks, the migration server can record the status of the migration subtask, such as the status of incomplete migration, migration, and migration.
  • the migration server detects a migration interruption operation for a relationship chain, you can record the number of the migration subtask that did not complete the migration.
  • a continuation migration operation for the relationship chain is detected, only the migration subtasks that have not completed the migration are performed to migrate the business data and computing tasks that were not migrated when the relationship chain is interrupted to the target service cluster.
  • the migration server may also adopt different migration states to control the migration process, and adopt the state machine to manage the migration process, thereby avoiding the loss of the migration state and ensuring that the migration state is lost.
  • the migration process of the relationship chain can be interrupted arbitrarily and then resumed.
  • FIG. 2H shows a schematic diagram of the migration state involved in the relationship chain during the migration process. The following takes a migration process of a relational chain as an example to introduce each migration state:
  • Start migration Start migrating the data indicated by the relationship chain.
  • the migration state After determining that the migration process of the relationship chain is started, the migration state can be entered. In the migration state, the migration server obtains the source data from the real-time data processing server through the specified interface.
  • the migration server determines that the original storage path and the target storage path of the key service data indicated by the third relationship chain are included in the data path mapping table. In the middle, it enters the migration state waiting for the user to confirm.
  • Freeze computing task In this migration state, the migration server performs the above steps 206b and 206c, and enters the next migration state when all computing tasks are in a stopped state.
  • Service data consistency check After the service data indicated by the relationship chain is completely migrated to the target service cluster, the migration state is entered.
  • Service data storage path switching After the consistency check of the relationship chain is successful, the migration state is entered, and the process of switching the service data storage path is performed.
  • Compute task migration After the service data storage path is completely switched from the original service cluster to the target service cluster, the migration state is entered and the process of calculating the task migration is performed.
  • Thawing calculation task After the completion of the task migration, the process of running all the computing tasks is executed. When all the computing tasks are running normally, the migration completion state is entered, thereby completing the migration of the business data and the computing tasks indicated by the relationship chain.
  • the migration server can provide a management interface in the foreground, and the terminal having the management authority can access the foreground of the migration server to display the management interface, and the management personnel can obtain the migration process by viewing the management interface.
  • the terminal having the management authority can access the foreground of the migration server to display the management interface, and the management personnel can obtain the migration process by viewing the management interface.
  • any terminal can log in to the migration server by using the administrator's username and password to obtain management rights and access the migration server foreground based on the obtained management rights.
  • the terminal connected to the target service cluster and connected to the target service cluster can obtain management rights and access the migration server foreground based on the obtained management rights.
  • the management interface may include the connection relationship of each node in the relationship chain, the migration status information of each node, the running status information of the computing task, the start migration time, the expected stop migration time, the migration status of the relationship chain, and the migration of the relationship chain. Progress and so on.
  • the management interface can also include one or more management options for managing the migration process.
  • the management interface may include a stop migration option and a continue migration option. When the administrator triggers the stop migration option, the migration server receives the stop migration instruction, and then suspends the current migration process until subsequent managers trigger the continue migration option. The migration server continues to migrate after receiving the Continue Migration command.
  • the method provided in this embodiment uses a relational chain representation of the service data and the calculation task of the association relationship according to the calculation task log in the original service cluster, so that the data migration is performed in the process of data migration in the relationship chain unit.
  • the relationship chain does not affect other relationship chains, and the computing tasks indicated by the relationship chain that is not migrated can still be run normally, so as not to affect the normal use of the services indicated by the relationship chain that has not been migrated.
  • both the computing task and the business data are migrated as nodes in the relationship chain, so that the computing tasks are not affected by the geographical location of the business data.
  • the relationship can be made large.
  • the small relationship chain can access the key service data flexibly regardless of whether it belongs to the original service cluster or the target service cluster, and realizes the decoupling between the related services and achieves the adoption.
  • Multiple small relationship chains gradually migrate complex businesses.
  • the service data indicated by the relationship chain is first migrated.
  • the calculation task can be performed in the gap where the calculation task stops running.
  • Migration greatly reducing the impact of data migration on the normal use of the business, and because the business data reaches the migration progress, the remaining business data volume can usually be completed in a short period of time, which can be less than the running cycle of the computing task, so The process of data migration does not affect the normal use of the business at all, and realizes user-unaware data migration.
  • the granularity of data migration is reduced, and the multiple migration sub-tasks can be run in parallel, which improves the migration efficiency of the service data indicated by the relationship chain.
  • the migration subtask needs to be re-migrated, and the service data of the entire service cluster does not need to be re-migrated, which reduces the cost of data errors during the migration process and improves the efficiency of data migration.
  • the business data and computing tasks in the original service cluster are gradually migrated to the target service cluster in units of multiple relationship chains by means of rounding to zero.
  • the service data and computing tasks in the original service cluster are performed.
  • the number of servers in the original service cluster can be removed and relocated to the target IDC room, so that server device resources can be reused, reducing the cost of data migration.
  • FIG. 3 is a block diagram of a data migration apparatus according to an embodiment of the present invention.
  • the apparatus includes a first acquisition unit 301 and a migration unit 302.
  • the first obtaining unit 301 is connected to the migration unit 302, and is configured to acquire a plurality of relationship chains according to the calculation task log of the original service cluster, where the calculation task log is used to record the relationship between the computing task and the service data in the original service cluster.
  • Each relationship chain is used to indicate a set of computing tasks and business data having an association relationship; and the migration unit 302 is configured to sequentially migrate the business data and the computing tasks indicated by the plurality of relationship chains to the target in a relationship chain unit.
  • a service cluster wherein, when the migration is based on any one of the relationship chains, the computing tasks indicated by the relationship chains that are not migrated in the plurality of relationship chains are normally run.
  • the first obtaining unit 301 is configured to add the same relationship chain identifier to the input and output records with the associated relationship according to the multiple input and output records recorded in the computing task log, so as not to have an association.
  • the input and output records of the relationship add different relationship chain identifiers; according to the association relationship between the computing task and the business data indicated by the input and output records identified by the same relationship chain, multiple relationship chains are generated, and each relationship chain includes a function for indicating calculation A task node of a task, a data node for indicating business data, and an association relationship between the task node and the data node.
  • the first obtaining unit 301 includes:
  • Generating a subunit configured to generate a plurality of first relationship chains according to an association relationship between the computing task and the service data indicated by the input and output records identified by the same relationship chain;
  • splitting unit is configured to split the second relationship chain into a plurality of third relationship chains if the plurality of first relationship chains include the second relationship chain, where the second relationship chain is data of the indicated service data The first relationship chain that exceeds the first threshold.
  • the splitting unit is configured to obtain weights of multiple data nodes in the second relationship chain, and the weight of each data node is used to indicate that the data node is in the second relationship chain.
  • the degree of association the higher the weight is the higher the degree of association;
  • the key data nodes are obtained from the plurality of data nodes according to the order of the weights from high to low and the positions of the plurality of data nodes on the second relationship chain
  • the key data node is the first data node in the sequence capable of splitting the second relationship chain into at least two third relationship chains; and based on the key data node, the second relationship chain and the key data
  • the plurality of task nodes associated with the node are split into a plurality of third relationship chains.
  • the disassembling unit is used to:
  • the critical data node, the task node, and the key data node are disconnected from the task node and have an association relationship with the task node.
  • the node is determined to be the third relationship chain; or,
  • Determining the key data node as a third relationship chain Determining the key data node as a third relationship chain, and each of the plurality of task nodes directly associated with the key data node is associated with the task node except the key data node The node is determined to be the third relationship chain; or,
  • the splitting unit is configured to, for each of the plurality of data nodes, a number of task nodes associated with the data node and data of the service data indicated by the data node The product of the quantity is determined as the weight of the data node.
  • the device further includes:
  • a second acquiring unit configured to acquire a target storage path of the key service data in the target service cluster, where the key service data is the service data indicated by the key data node;
  • the migration unit 302 is configured to:
  • the key service data is synchronized in the target service cluster and the original service cluster according to the target storage path and the original storage path;
  • the target storage path recorded according to the data path mapping table when running the computing task indicated by the third relationship chain Access the key business data;
  • the migration unit 302 includes:
  • Generating a subunit configured to generate, for each of the plurality of relationship chains, a plurality of migration subtasks according to the plurality of service data indicated by the relationship chain, where each migration subtask is used to indicate corresponding service data The original storage path and the target storage path;
  • a first migration subunit configured to migrate the service data indicated by the relationship chain to the target service cluster according to the multiple migration subtasks
  • a second migration subunit configured to migrate the computing task indicated by the relationship chain to the target service cluster
  • the computing task indicated by the relationship chain is in a stopped state when the computing task indicated by the relationship chain is migrated.
  • the first migration subunit is configured to:
  • the service data is divided into multiple sub-service data according to the second threshold, and a migration sub-task is generated corresponding to each sub-service data.
  • the amount of data of each sub-service data is less than the second threshold.
  • the first migration subunit is further configured to:
  • the device further includes:
  • a first checking unit configured to: for each of the multiple migration subtasks, after the traffic data corresponding to the migration subtask is all migrated to the target service cluster, the target service cluster and the original service If the consistency check succeeds, the service data corresponding to the migration subtask is successfully migrated. If the consistency check fails, the migration subtask is determined. The corresponding business data migration fails and the migration subtask is re-executed.
  • the device further includes:
  • a second check unit configured to perform consistency check on the service data indicated by the relationship chain in the target service cluster and the original service cluster; if the consistency check succeeds, perform calculation indicated by the relationship chain If the task is migrated to the target service cluster, if the consistency check fails, the service data that failed to be migrated is determined according to the consistency check result, and the service data that failed the migration is re-migrated.
  • the second migration subunit is configured to acquire first computing resource information and second computing resource information of the computing task, where the first computing resource information is used in the original service cluster.
  • the computing resource information configured by the task, the second computing resource information is computing resource information configured for the computing task in the target service cluster; the first computing resource information of the computing task is replaced with the second computing resource information.
  • the device further includes:
  • a switching unit configured to switch, in the data path mapping table, the original storage path of the service data in the original service cluster to a target storage path in the target service cluster.
  • the migration unit 302 is further configured to: when the migration based on any one of the relationship chains is detected, when the migration interruption operation of the relationship chain is detected, the migration subtask that does not complete the migration is recorded, and the pair is stopped.
  • the device further includes:
  • the relationship chain update unit is configured to obtain an updated calculation task log; and update the relationship chain that has not been migrated according to the updated calculation task log.
  • the device provided in this embodiment uses a relational chain to represent the service data and the computing task with the associated relationship according to the computing task log in the original service cluster, so that the data migration process is performed in the process of data migration in the relationship chain unit.
  • the relationship chain does not affect other relationship chains, and the computing tasks indicated by the relationship chain that is not migrated can still be run normally, so as not to affect the normal use of the services indicated by the relationship chain that has not been migrated.
  • FIG. 4 is a block diagram of a data migration apparatus according to an embodiment of the present invention.
  • device 400 can be provided as a server.
  • apparatus 400 includes a processing component 422 that further includes one or more processors, and memory resources represented by memory 432 for storing instructions executable by processing component 422, such as an application.
  • An application stored in memory 432 may include one or more modules each corresponding to a set of instructions.
  • the processing component 422 is configured to execute instructions to perform the method performed by the migration server in the data migration method embodiment described above.
  • Device 400 may also include a power supply component 426 configured to perform power management of device 400, a wired or wireless network interface 450 configured to connect device 400 to the network, and an input/output (I/O) interface 458.
  • Device 400 may operate based on an operating system stored in the memory 432, for example, Windows Server TM, Mac OS X TM , Unix TM, Linux TM, FreeBSD TM or the like.
  • the data migration apparatus can be used to perform the operations performed by the migration server in the above embodiment.
  • non-transitory computer readable storage medium comprising instructions, such as a memory comprising instructions executable by a processor to perform the data migration method of the above embodiments.
  • the non-transitory computer readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.
  • the embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores at least one instruction loaded by a processor and executed to implement an operation performed by the migration server in the method of the foregoing embodiment.
  • the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.
  • a person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium.
  • the storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne un procédé de migration de données, un serveur de migration et un support d'informations, se rapportant au domaine technique des réseaux. Le procédé consiste : à obtenir de multiples chaînes de relations en fonction de journaux de tâches de calcul d'un groupe de services d'origine, les journaux de tâches de calcul étant utilisés pour enregistrer des associations entre des tâches de calcul et des données de service dans le groupe de services d'origine, et chaque chaîne de relation étant utilisée pour indiquer un groupe de tâches de calcul et de données de service présentant une association ; à migrer séquentiellement, en prenant une chaîne de relations en tant qu'unité, les données de service et les tâches de calcul indiquées par les multiples chaînes de relations à un groupe de services cible ; et pendant la migration sur la base de l'une quelconque des chaînes de relations, à exécuter normalement des tâches de calcul indiquées par des chaînes de relations, qui ne sont pas migrées, dans les multiples chaînes de relations. En utilisant une chaîne de relations pour indiquer des données de service et une tâche de calcul présentant une association, des tâches de calcul indiquées par des chaînes de relations qui ne sont pas migrées peuvent toujours être exécutées normalement pendant la migration de données ; ainsi, l'utilisation normale de services indiqués par les chaînes de relations qui ne sont pas migrées ne serait pas affectée.
PCT/CN2018/078398 2017-03-29 2018-03-08 Procédé de migration de données, serveur de migration, et support d'informations WO2018177107A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710197702.7 2017-03-29
CN201710197702.7A CN108664496B (zh) 2017-03-29 2017-03-29 数据迁移方法及装置

Publications (1)

Publication Number Publication Date
WO2018177107A1 true WO2018177107A1 (fr) 2018-10-04

Family

ID=63674187

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/078398 WO2018177107A1 (fr) 2017-03-29 2018-03-08 Procédé de migration de données, serveur de migration, et support d'informations

Country Status (2)

Country Link
CN (1) CN108664496B (fr)
WO (1) WO2018177107A1 (fr)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110399356A (zh) * 2019-06-14 2019-11-01 阿里巴巴集团控股有限公司 一种在线数据迁移方法、装置、计算设备及存储介质
CN111258985A (zh) * 2020-01-17 2020-06-09 中国工商银行股份有限公司 数据集群迁移方法及装置
CN111274230A (zh) * 2020-03-26 2020-06-12 北京奇艺世纪科技有限公司 数据迁移的管理方法、装置、设备及存储介质
CN111708763A (zh) * 2020-06-18 2020-09-25 北京金山云网络技术有限公司 分片集群的数据迁移方法、装置和分片集群系统
CN111708755A (zh) * 2020-05-20 2020-09-25 北京奇艺世纪科技有限公司 数据迁移方法、装置、系统、电子设备以及可读存储介质
CN112506606A (zh) * 2020-11-23 2021-03-16 北京达佳互联信息技术有限公司 集群中容器的迁移方法、装置、设备和介质
CN112653539A (zh) * 2020-12-29 2021-04-13 杭州趣链科技有限公司 一种待存储数据的存储方法、装置以及设备
WO2021098268A1 (fr) * 2019-11-22 2021-05-27 浪潮电子信息产业股份有限公司 Procédé, appareil et dispositif de migration de service mon, et support de stockage lisible
CN113051245A (zh) * 2019-12-26 2021-06-29 云丁网络技术(北京)有限公司 用于迁移数据的方法、装置及系统
CN113535087A (zh) * 2021-07-13 2021-10-22 咪咕互动娱乐有限公司 数据迁移过程中的数据处理方法、服务器及存储系统
CN114024956A (zh) * 2020-07-17 2022-02-08 北京达佳互联信息技术有限公司 数据迁移方法、装置、服务器及存储介质
CN114785796A (zh) * 2022-04-22 2022-07-22 中国农业银行股份有限公司 一种数据均衡方法和装置
CN116954870A (zh) * 2023-09-19 2023-10-27 苏州元脑智能科技有限公司 跨集群应用的迁移方法、恢复方法、装置及集群系统

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110503199A (zh) * 2019-08-14 2019-11-26 北京中科寒武纪科技有限公司 运算节点的拆分方法和装置、电子设备和存储介质
CN110490322A (zh) * 2019-08-14 2019-11-22 北京中科寒武纪科技有限公司 运算节点的拆分方法和装置、电子设备和存储介质
CN110597609A (zh) * 2019-09-17 2019-12-20 深圳市及响科技有限公司 一种集群迁移与自动恢复方法及系统
CN113438267B (zh) * 2020-03-23 2023-02-28 华为技术有限公司 一种分析流数据的方法与设备
CN111459411B (zh) * 2020-03-30 2023-07-21 北京奇艺世纪科技有限公司 数据迁移方法、装置、设备及存储介质
CN116049096B (zh) * 2022-05-05 2024-04-16 荣耀终端有限公司 一种数据迁移方法、电子设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102855299A (zh) * 2012-08-16 2013-01-02 上海引跑信息科技有限公司 不中断服务条件下,分布式数据库迭代迁移的方法
CN102982085A (zh) * 2012-10-31 2013-03-20 北京奇虎科技有限公司 数据迁移系统和方法
CN103324466A (zh) * 2013-05-24 2013-09-25 浪潮电子信息产业股份有限公司 一种数据相关性序列化io的并行处理方法
CN103970879A (zh) * 2014-05-16 2014-08-06 中国人民解放军国防科学技术大学 一种调整数据块存储位置的方法及系统

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6047323A (en) * 1995-10-19 2000-04-04 Hewlett-Packard Company Creation and migration of distributed streams in clusters of networked computers
EP1567929A2 (fr) * 2002-11-15 2005-08-31 Creo Inc. Procedes et systemes de partage de donnees
JP4856932B2 (ja) * 2005-11-18 2012-01-18 株式会社日立製作所 記憶システム及びデータ移動方法
CN102999537B (zh) * 2011-09-19 2017-01-18 阿里巴巴集团控股有限公司 一种数据迁移系统和方法
CN103164261B (zh) * 2011-12-15 2016-04-27 中国移动通信集团公司 多中心数据任务处理方法、装置及系统
CN103647849B (zh) * 2013-12-24 2017-02-08 华为技术有限公司 一种业务迁移方法、装置和一种容灾系统
CN104935618B (zh) * 2014-03-19 2018-01-19 福建福昕软件开发股份有限公司 一种集群部署方法
CN103955491B (zh) * 2014-04-15 2017-04-19 南威软件股份有限公司 一种定时数据增量同步的方法
CN104184813B (zh) * 2014-08-20 2018-03-09 杭州华为数字技术有限公司 虚拟机的负载均衡方法和相关设备及集群系统
CN105404474A (zh) * 2015-12-07 2016-03-16 上海爱数信息技术股份有限公司 一种异构分布式存储系统的数据迁移方法
CN106055670A (zh) * 2016-06-06 2016-10-26 中国工商银行股份有限公司 一种系统间数据迁移方法及装置
CN106202212A (zh) * 2016-06-28 2016-12-07 微梦创科网络科技（中国）有限公司 一种基于数据服务器集群实现数据拆分的方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102855299A (zh) * 2012-08-16 2013-01-02 上海引跑信息科技有限公司 不中断服务条件下,分布式数据库迭代迁移的方法
CN102982085A (zh) * 2012-10-31 2013-03-20 北京奇虎科技有限公司 数据迁移系统和方法
CN103324466A (zh) * 2013-05-24 2013-09-25 浪潮电子信息产业股份有限公司 一种数据相关性序列化io的并行处理方法
CN103970879A (zh) * 2014-05-16 2014-08-06 中国人民解放军国防科学技术大学 一种调整数据块存储位置的方法及系统

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110399356A (zh) * 2019-06-14 2019-11-01 阿里巴巴集团控股有限公司 一种在线数据迁移方法、装置、计算设备及存储介质
WO2021098268A1 (fr) * 2019-11-22 2021-05-27 浪潮电子信息产业股份有限公司 Procédé, appareil et dispositif de migration de service mon, et support de stockage lisible
CN113051245A (zh) * 2019-12-26 2021-06-29 云丁网络技术(北京)有限公司 用于迁移数据的方法、装置及系统
CN111258985A (zh) * 2020-01-17 2020-06-09 中国工商银行股份有限公司 数据集群迁移方法及装置
CN111274230A (zh) * 2020-03-26 2020-06-12 北京奇艺世纪科技有限公司 数据迁移的管理方法、装置、设备及存储介质
CN111274230B (zh) * 2020-03-26 2024-03-08 北京奇艺世纪科技有限公司 数据迁移的管理方法、装置、设备及存储介质
CN111708755A (zh) * 2020-05-20 2020-09-25 北京奇艺世纪科技有限公司 数据迁移方法、装置、系统、电子设备以及可读存储介质
CN111708763B (zh) * 2020-06-18 2023-12-01 北京金山云网络技术有限公司 分片集群的数据迁移方法、装置和分片集群系统
CN111708763A (zh) * 2020-06-18 2020-09-25 北京金山云网络技术有限公司 分片集群的数据迁移方法、装置和分片集群系统
CN114024956A (zh) * 2020-07-17 2022-02-08 北京达佳互联信息技术有限公司 数据迁移方法、装置、服务器及存储介质
CN114024956B (zh) * 2020-07-17 2024-03-12 北京达佳互联信息技术有限公司 数据迁移方法、装置、服务器及存储介质
CN112506606A (zh) * 2020-11-23 2021-03-16 北京达佳互联信息技术有限公司 集群中容器的迁移方法、装置、设备和介质
CN112653539A (zh) * 2020-12-29 2021-04-13 杭州趣链科技有限公司 一种待存储数据的存储方法、装置以及设备
CN112653539B (zh) * 2020-12-29 2023-06-20 杭州趣链科技有限公司 一种待存储数据的存储方法、装置以及设备
CN113535087A (zh) * 2021-07-13 2021-10-22 咪咕互动娱乐有限公司 数据迁移过程中的数据处理方法、服务器及存储系统
CN113535087B (zh) * 2021-07-13 2023-10-17 咪咕互动娱乐有限公司 数据迁移过程中的数据处理方法、服务器及存储系统
CN114785796A (zh) * 2022-04-22 2022-07-22 中国农业银行股份有限公司 一种数据均衡方法和装置
CN116954870A (zh) * 2023-09-19 2023-10-27 苏州元脑智能科技有限公司 跨集群应用的迁移方法、恢复方法、装置及集群系统
CN116954870B (zh) * 2023-09-19 2024-02-02 苏州元脑智能科技有限公司 跨集群应用的迁移方法、恢复方法、装置及集群系统

Also Published As

Publication number Publication date
CN108664496A (zh) 2018-10-16
CN108664496B (zh) 2022-03-25

Similar Documents

Publication Publication Date Title
WO2018177107A1 (fr) Procédé de migration de données, serveur de migration, et support d'informations
US20240012720A1 (en) Application migration between environments
US20200364092A1 (en) Managing partitions in a scalable environment
WO2019154394A1 (fr) Système en grappes de bases de données réparties, procédé de synchronisation de données et support de stockage
US11663085B2 (en) Application backup and management
US11481139B1 (en) Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity
US8341365B2 (en) Data backup system and method for virtual infrastructure
US7779298B2 (en) Distributed job manager recovery
US11966307B2 (en) Re-aligning data replication configuration of primary and secondary data serving entities of a cross-site storage solution after a failover event
US9170848B1 (en) Parallel processing of data
WO2020072338A1 (fr) Sauvegarde de données et récupération après sinistre entre environnements
CN109408115B (zh) 一种基于容器环境中迁移对象的方法及计算系统
WO2012113336A1 (fr) Système de gestion de ressources dans un environnement virtualisé et procédé de mise en œuvre correspondant
KR20190041033A (ko) 서비스의 2차 위치에서의 작업의 재생 기법
US9563478B1 (en) Scalable concurrent execution of distributed workflows sharing common operations
US11409711B2 (en) Barriers for dependent operations among sharded data stores
CN110347483B (zh) 物理机到虚拟机迁移方法、装置及存储介质
US7516181B1 (en) Technique for project partitioning in a cluster of servers
US20220269414A1 (en) Snapshotting a containerized application
US20210165768A1 (en) Replication Barriers for Dependent Data Transfers between Data Stores
US11176088B2 (en) Dynamic server pool data segmentation using dynamic ordinal partition key without locks
US20230023053A1 (en) Data loss recovery in a secondary storage controller from a primary storage controller
WO2017181430A1 (fr) Procédé et dispositif de duplication de base de données dans un système distribué
US11921596B2 (en) Virtual machine failover management for geo-redundant data centers
US9485308B2 (en) Zero copy volume reconstruction

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18777264

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18777264

Country of ref document: EP

Kind code of ref document: A1