WO2012051845A1

WO2012051845A1 - Data transfer method and system

Info

Publication number: WO2012051845A1
Application number: PCT/CN2011/073485
Authority: WO
Inventors: 陈典强; 郭斌; 韩银俊
Original assignee: 刘建
Priority date: 2010-10-21
Filing date: 2011-04-28
Publication date: 2012-04-26
Also published as: CN101997911A; CN101997911B

Abstract

A data transfer method is disclosed in the present invention, and the method includes: according to the user selection, a data transfer-out node is notified to perform data transfer; the data transfer-out node sends the transferred data and a redoing log to a data transfer-in node; the data transfer-in node receives the transferred data and stores it, receives the redoing log and performs verification and update for the received transferred data according to the redoing log. A data transfer system is also disclosed in the present invention. The method and system in the present invention can improve the data transfer speed and ensure the data consistency in the transfer process, while also ensuring the normal operation of the system.

Description

Method and system for data migration

The present invention relates to the field of data distributed caching, and more particularly to a method and system for data migration. Background technique

Cloud computing is the product of the integration of traditional computer technology and network technology such as grid computing, distributed computing, parallel computing, utility computing, network storage, virtualization, load balancing, etc. It aims to bring multiple relatively low costs through the network. Computational entities are integrated into a system with powerful computing power. Distributed caching is an area in the field of cloud computing. Its role is to provide distributed storage services for massive amounts of data in cloud computing and high-speed read and write access.

A distributed caching system that provides distributed caching functions consists of a number of server nodes, hereinafter referred to as nodes, and clients interconnected. In general, to ensure data security, data written to a node cannot be stored on a single node. Instead, multiple copies of the same data are stored on multiple nodes, which are backups of each other. The data consists of a key (Key) and a value (Value). The Key is equivalent to the index of the data, and the Value is the data content represented by the Key. Logically, Key and Value are one-to-one relationships.

Distributed cache system may have problems such as uneven node load, unstable nodes, and excessive system load during long-term work. In this case, some data on nodes with higher or unstable load needs to be migrated to On a node with a lower load; or add a new node to migrate part of the data in the system to the new node for maintenance. In a distributed cache system, how to migrate data without affecting the normal operation of the system is a key problem that is difficult to solve.

In the prior art, there are generally two solutions to data migration:

Solution 1, stop the normal operation of the distributed cache system, prompt the client to distribute the cache system Updating, and then data migration of the distributed cache system;

Solution 2: In the normal working condition of the distributed cache system, the client traverses the data in the node that reads the data to be migrated, and then writes the node to be moved into the data;

The method in the above scheme 1 stops the normal operation of the distributed cache system, and affects the normal operation of the distributed cache system; the method in the scheme 2, the data is read by the traversal method, which causes the data migration speed to be slow, and In the case that the distributed cache system works normally, the newly generated data may be missed, resulting in inconsistent data between the nodes that migrated data and the nodes that migrate data. Summary of the invention

In view of this, the main purpose of the present invention is to provide a data migration method and system, which can improve the data migration speed while ensuring the normal operation of the system, and ensure the consistency of data during the migration process.

In order to achieve the above object, the technical solution of the present invention is achieved as follows:

The present invention provides a method for data migration, the method comprising:

Data migration based on user selection notification data removal node;

The data removal node sends the migration data and the redo log directly to the data migration node; the data migration node receives the migration data and saves it, and performs the calibration and update of the migration data according to the redo log.

In the above solution, the notification includes: a data entry node address, and a virtual node address where the migration data is located.

In the above solution, the data removal node directly sends the migration data and the redo log to the data migration node, including: the data removal node moves the node address according to the data, and the migration data and the redo log in the virtual node address are in the form of data packets. The data is sent to the node, and after the data packet of the current redo is sent, the management platform is notified, and the control platform suspends the service of the data removal node. In the foregoing solution, the verifying and updating the migration data according to the redo log includes: the data migration node determines, according to the received redo log, that the data corresponding to the operation recorded in the redo log is different from the migration data, and is performed again. The operation.

In the above solution, after the verification and update of the migration data according to the redo log, the method further includes: the data migration node notifies the management platform that the re-operation is completed, and the control platform notifies all nodes and clients, and the data is moved into the node instead of the data. The removal node provides the client with the relevant services for migrating data, and at the same time clears the migration data from the data removal node.

The present invention also provides a system for data migration, the system comprising: a data migration management module, a data removal processing module, and a data migration processing module;

The data migration management module is configured to migrate the data according to the user selection notification data removal processing module;

The data removal processing module is configured to send the migration data and the redo log to the data migration processing module;

The data is moved into the processing module, set to receive the migrated data and saved, and the received migration data is checked and updated according to the redo log.

In the above solution, the data migration management module is configured to send a notification including the data entry node address and the virtual node address where the migration data is located to the data removal processing module according to the user selection.

In the above solution, the data removal processing module is configured to: according to the data movement into the node address, send the migration data and the redo log in the virtual node to the data migration processing module in the form of data packets, and determine the data packet of the current redo log. After the sending is completed, the data migration management module is notified; the data migration management module is configured to suspend the service of the data removal node according to the notification of the data removal processing module.

In the above solution, the data migration processing module is configured to notify the data migration management module after the re-operation is completed; correspondingly, the data migration management module is configured to notify all nodes and The client, by the data moving in node instead of the data removal node, provides the client with the relevant service for migrating data, and clears the migrated data in the data removal node.

It can be seen that, by using the method and system of the present invention, the migration data is directly sent from the data removal node to the data migration node, thereby improving the speed of data migration; the migration data is migrated after replication, thereby effectively ensuring the normal function of the system; The data moving in node performs re-operation according to the redo log to ensure the consistency of the data migration between the data removal node and the data moving in node when the client sends a request to the data migration node during the data migration. DRAWINGS

1 is a schematic flowchart of a method for implementing data migration according to the present invention;

FIG. 2 is a schematic diagram of a system composition for implementing data migration according to the present invention. detailed description

The basic idea of the present invention is: according to the user selection notification data removal node for data migration, the data removal node sends data and redo (Redo) logs to the data migration node, and the data migration node receives the migration data and saves, according to the redo log pair. The received migration data is checked and updated.

The invention will now be described in detail by way of specific embodiments and drawings.

A method of data migration, as shown in Figure 1, includes the following steps:

Step 101: When the system is overburdened or the load is uneven, the data is migrated according to the user selection notification data to move out of the node;

When the user finds that the system is overloaded or the load is uneven, the data migration needs to be started. The node that needs to migrate data, that is, the data removal node is selected through the management platform; and the node that receives the migration data, that is, the data is moved into the node, and the data is removed from the node for data. Migration; here, the system refers to a distributed cache system;

The notification is: the management platform sends data migration information to the data removal node. The data migration information includes a virtual node that needs to be migrated, a data move-in node address, and the like; the virtual node is a node that stores data in different virtual nodes. The data moving in node may be a lightly loaded node, or may be a newly added node, and the newly added node accesses the system, and sends its own information to the control platform. The control platform requests other nodes and new according to the information sent by the newly added node. The node is added to establish a connection, or the new node is required to establish a connection with other nodes; the self information includes its own address and the like.

Step 102: The data removal node directly sends the data and the Redo log to the data moving in node; the data removal node selects the data in the virtual node according to the notification of the management platform, and moves the data to the node address according to the data, and sends the data to the node. node.

If the client sends a request to the data removal node at this time, the data can still be served to the client because the data of the data removal node is still there. At the same time, because the client sends a request to the data removal node, the data of the data removal node changes, and the data removal node is sent according to the client during the data migration period during the data migration. The request, the data moves out of the response made by the node, that is, the operation of the data removal node is written into the Redo log, and after the data removal node sends the migration data to the data migration node, the Redo log is sent to the data migration node in the form of a data packet. The Redo is used to record the operation of the data removal node, and the operation of the data removal node includes the data removal node performing actions and corresponding data, such as deleting and deleting data, adding and adding data, modifying and modifying data, etc. The function of the Redo log can be selected by the management module to be enabled.

Because the client will send requests to the data removal node from time to time, the records in the Redo log will continue to increase, which will cause the migration to end. Therefore, the data removal node notifies the management platform when the current Redo packet transmission is completed. The control platform requires the data removal node to stop the service, the data removal node stops the service, and the Redo log generated from the notification management platform to the stop service is sent to the data migration node in the form of a data packet, and the stop service refers to stopping the service to the system. provide support. The data is stored in the form of Key and Value. A Key and the corresponding Value are stored in multiple nodes, one of which is called a collaborative server, that is, a collaborative node, and the other is called a replica server, that is, a replica node, a collaborative server. There is a connection relationship with the replica server, and the function of the collaboration node and the replica node are the same. The correspondence between the Key and the node address is stored in the routing table. One Key corresponds to multiple node addresses. By default, the first node address is selected as the coordination node address for processing the Key, and the other node addresses are used as the replica node address. When the client sends a request to the node, it will query the locally saved routing table according to the Key, obtain the collaborative node address, and send a request to the node according to the node address. If the coordinating node stops the service, the next node address corresponding to the key is searched according to the locally saved routing table, and the request sent by the client is sent to the next node, and the node processes the request related to the key as a collaborative server, and therefore, the data After the node is removed from the service, it does not affect the normal operation of the system and the normal use of the client.

Step 103: The data moving in node receives the migration data, and performs verification and update on the migration data according to the Redo log to ensure data consistency.

Here, the data moving in node receives the data sent by the data removal node and saves, receives the Redo log sent by the data removal node, and checks and updates the migration data according to the log record, where the checksum update refers to the data moving in the node according to Receive the Redo log, obtain the operation in the Redo log, and compare the data corresponding to the operation with the migrated data. If the migration data is the same, no operation is performed. If the migration data is different, the operation is performed again to ensure the data. After the consistency, the verification and the update are completed, the management and control platform is notified, and the control platform sends a node replacement notification message to all nodes and clients in the network, and the node replacement notification message refers to the service for migrating data previously provided by the data removal node. Now, the data is moved into the node; and, in the routing table, the node address corresponding to the key in the migrated data, the original data is moved out of the node address, replaced with the address of the data moving in node, and the data is moved into the node service. At the same time, the management platform clears the migration data from the data removal node. At this point, the data migration work is completed.

The present invention also provides a system for data migration, as described in FIG. 2, the system includes: According to the migration management module 201, the data removal processing module 202, and the data migration processing module 203, the data migration management module 201 is located in the management module, and is configured to perform data migration according to the user selection notification data migration processing module 202; the notification includes a data migration node. The address and the virtual node where the data is migrated;

The data removal processing module 202 is located at the data removal node, and is configured to

The Redo log is sent to the data migration processing module 203 in the form of a data packet;

The data movement processing module 203 is located at the data moving in node, and is configured to receive and save the data packet, receive the Redo log, and check and update the received migration data according to the Redo log.

The data removal processing module 202 is configured to receive a request from the client, and move the data out of the response of the node, that is, the operation of moving the data out of the node, and record the data in the Redo log.

The data removal processing module 202 is further configured to: after determining that the data transmission of the current Redo log is completed, notify the data migration management module 201, and send the data of the Redo log generated between the sending of the log and the stopping of the service to the data migration processing module 203. ;

The corresponding data migration management module 201 is configured to suspend the service of the data removal node.

The data migration processing module 203 is further configured to: according to the received Redo log, obtain an operation recorded in the Redo log, compare the data in the operation with the migrated data, and if the migration data is the same, do not perform any operation, if the data is migrated If the operation is performed again, the data consistency is ensured. After the operation of the Redo log record is performed, the data migration management module 201 is notified. The data migration management module 201 is configured to notify all the notifications according to the data migration processing module 203. The node and the client, the migration data related service originally provided by the data removal node is changed to be served by the data migration node, and the data in the node address corresponding to the Key in the routing table is moved out of the node address, and replaced by the data migration node address.

The data migration management module 201 is further configured to clear the migrated data in the data out of the node.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. All modifications, equivalent substitutions and improvements made within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims

Claim

1. A method of data migration, the method comprising:

Data migration based on user selection notification data removal node;

2. The method according to claim 1, wherein the notification comprises: a data movement into a node address, and a virtual node address where the migration data is located.

3. The method according to claim 2, wherein the data removal node directly transmitting the migration data and the redo log to the data migration node comprises:

The data removal node sends the migration data and the redo log in the virtual node address to the data migration node in the form of data packets according to the data movement into the node address, and after the data packet of the current redo log is sent, the management platform is notified, and the control platform is notified. Pause data to move out of the node's service.

4. The method according to claim 1, wherein the verifying and updating the migration data according to the redo log comprises:

The data migration node determines that the data corresponding to the operation recorded in the redo log is different from the migration data according to the received redo log, and performs the operation again.

The method according to any one of claims 1 to 4, wherein after the checking and updating the migration data according to the redo log, the method further includes:

The data moving in node notifies the management platform that the re-operation is completed, and the control platform notifies all nodes and clients that the data moving in node replaces the data removal node to provide the client with the relevant services for migrating data, and at the same time clears the migration data in the data removal node.

6. A data migration system, the system comprising: a data migration management module, a data removal processing module, and a data migration processing module;

The data migration management module is configured to migrate the data removal processing module according to the user selection notification Data

7. The system according to claim 6, wherein

The data migration management module is configured to send, to the data removal processing module, a notification that the data is moved into the node address and the virtual node address where the data is migrated according to the user selection.

8. The system according to claim 7, wherein

The data removal processing module is configured to: according to the data movement into the node address, send the migration data and the redo log in the virtual node to the data migration processing module in the form of a data packet, and determine that the data packet of the current redo log is sent, and the notification is complete. Data migration management module;

The data migration management module is configured to suspend the service of the data removal node according to the notification of the data removal processing module.

9. The system according to claim 6, wherein

The data movement processing module is configured to read an operation recorded in the received redo log, determine that the data corresponding to the operation is different from the migration data, and perform the operation again.

10. The system according to any one of claims 6 to 9, wherein

The data migration processing module is configured to notify the data migration management module after the re-operation is completed; correspondingly, the data migration management module is configured to notify all nodes and clients, and the data migration node replaces the data removal node to provide the client with The services related to migrating data, clearing the migration data from the data removal node.