CN113050890A - Data migration method and device - Google Patents
Data migration method and device Download PDFInfo
- Publication number
- CN113050890A CN113050890A CN202110324765.0A CN202110324765A CN113050890A CN 113050890 A CN113050890 A CN 113050890A CN 202110324765 A CN202110324765 A CN 202110324765A CN 113050890 A CN113050890 A CN 113050890A
- Authority
- CN
- China
- Prior art keywords
- data
- directory
- directory node
- migrated
- storage system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 53
- 230000005012 migration Effects 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000012544 monitoring process Methods 0.000 claims description 33
- 238000013500 data storage Methods 0.000 claims description 21
- 238000012217 deletion Methods 0.000 claims description 19
- 230000037430 deletion Effects 0.000 claims description 19
- 230000008859 change Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 239000000126 substance Substances 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 5
- 238000012423 maintenance Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
Abstract
The invention discloses a data migration method and device, and relates to the technical field of computers. One embodiment of the method comprises: determining a directory file corresponding to data to be migrated in a first storage system; traversing the directory file to obtain one or more directory node paths from a root directory to directory nodes; acquiring data to be migrated stored under each directory node path; according to the data format of a second storage system, assembling the directory node path and the data to be migrated; and storing the assembled directory node path and the data to be migrated to the second storage system. The implementation method can perform data migration among different storage systems, and realizes switching and upgrading of the storage system of the service framework.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data migration method and apparatus.
Background
For various services available under a service framework, the same storage system is often used to store data such as configuration information and domain names of the services, so as to implement unified management and maintenance of the different services under the service framework.
With the development and progress of storage systems, in order to improve the management and maintenance efficiency of services, it is necessary to switch or upgrade an existing storage system of a service framework. Because different storage systems adopt different data storage structures, the data in the storage systems need to be migrated before the storage systems can be switched or upgraded. However, at present, data migration between different storage systems cannot be directly realized, and thus, the storage system of the service framework cannot be directly switched or upgraded.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data migration method and apparatus, which can perform data migration between different storage systems, and implement switching and upgrading of a service framework storage system.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data migration method including:
determining a directory file corresponding to data to be migrated in a first storage system;
traversing the directory file to obtain one or more directory node paths from a root directory to directory nodes;
acquiring data to be migrated stored under each directory node path;
according to the data format of a second storage system, assembling the directory node path and the data to be migrated;
and storing the assembled directory node path and the data to be migrated to the second storage system.
Optionally, assembling the directory node path and the data to be migrated in a key-value form; the directory node path is key, and the data to be migrated is value.
Optionally, the method further comprises:
monitoring the directory node corresponding to each directory node path, so as to process the directory node path stored in the second storage system and the data to be migrated according to the monitored event.
Optionally, the event comprises any one or more of: the method comprises the following steps of creating an event of a directory node, deleting the event of the directory node, changing the data of the directory node and deleting the sub-directory node under the directory node.
Optionally, when a directory node data change event is monitored, the updated data to be migrated stored in the directory node path is obtained from the first storage system, so as to update the data to be migrated corresponding to the directory node path in the second storage system.
Optionally, when a directory node deletion event is monitored, monitoring of the directory node is cancelled, and a directory node path and data to be migrated corresponding to the directory node in the second storage system are deleted.
Optionally, when the directory node creation event is monitored, the directory node path and the data to be migrated corresponding to the created directory node are stored in the second storage system, and the created directory node is monitored.
Optionally, under the condition that a sub-directory node deletion event under a directory node is monitored, monitoring of the directory node and the sub-directory node under the directory node is cancelled;
and deleting all directory node paths containing the directory nodes and the data to be migrated corresponding to the directory node paths in the second storage system.
Optionally, the directory node paths are stored in an array variable, and the directory node corresponding to each directory node path is monitored by traversing the array variable.
Optionally, the first storage system is Zookeeper, and the second storage system is Etcd.
To achieve the above object, according to another aspect of an embodiment of the present invention, there is provided a data migration apparatus including: the system comprises a directory file determining module, a node path acquiring module, a migration data acquiring module, a data assembling module and a data storage module, wherein the directory file determining module comprises: wherein the content of the first and second substances,
the directory file determining module is used for determining a directory file corresponding to the data to be migrated in the first storage system;
the node path acquisition module is used for traversing the directory file to acquire one or more directory node paths from a root directory to directory nodes;
the migration data acquisition module is used for acquiring the data to be migrated stored under each directory node path;
the data assembling module is used for assembling the directory node path and the data to be migrated according to the data format of the second storage system;
and the data storage module is used for storing the assembled directory node path and the data to be migrated to the second storage system.
Optionally, the data assembling module is configured to assemble the directory node path and the data to be migrated in a key-value form; the directory node path is key, and the data to be migrated is value.
Optionally, the data storage module is further configured to,
monitoring the directory node corresponding to each directory node path, so as to process the directory node path stored in the second storage system and the data to be migrated according to the monitored event.
Optionally, the event comprises any one or more of: the method comprises the following steps of creating an event of a directory node, deleting the event of the directory node, changing the data of the directory node and deleting the sub-directory node under the directory node.
Optionally, the data storage module is further configured to,
and under the condition that a directory node data change event is monitored, acquiring updated data to be migrated stored under a directory node path from the first storage system so as to update the data to be migrated corresponding to the directory node path in the second storage system.
Optionally, the data storage module is further configured to,
and under the condition that a directory node deletion event is monitored, canceling the monitoring of the directory node, and deleting a directory node path and data to be migrated corresponding to the directory node in the second storage system.
Optionally, the data storage module is further configured to,
and under the condition that the creation event of the directory node is monitored, storing the directory node path corresponding to the created directory node and the data to be migrated to the second storage system, and monitoring the created directory node.
Optionally, under the condition that a sub-directory node deletion event under a directory node is monitored, monitoring of the directory node and the sub-directory node under the directory node is cancelled;
and deleting all directory node paths containing the directory nodes and the data to be migrated corresponding to the directory node paths in the second storage system.
Optionally, the data storage module is configured to store the directory node paths to an array variable, so as to monitor the directory node corresponding to each directory node path by traversing the array variable.
Optionally, the first storage system is Zookeeper, and the second storage system is Etcd.
To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided an electronic device for data migration, including: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out the method of any of the data migration methods described above.
To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements any one of the data migration methods described above.
One embodiment of the above invention has the following advantages or benefits: the data migration between the first storage system and the second storage system with different data storage structures is realized by traversing the directory file in the first storage system, acquiring the directory node path from the directory to the directory node and the data to be migrated stored under the directory node path, and assembling the directory node path and the data to be migrated according to the data format of the second storage system to store the data to the second storage system, so that the first storage system can be directly switched to or changed into the second storage system.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a data migration method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main flow of another data migration method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main modules of a data migration apparatus according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data migration method according to an embodiment of the present invention, and as shown in fig. 1, the data migration method may specifically include the following steps:
step S101, determining a directory file corresponding to data to be migrated in the first storage system.
That is, the first storage system stores the service configuration information, the service port number, and the service domain name waiting for migration data in a directory structure. Specifically, the first storage system is taken as an example of a Zookeeper, which is an important component of Hadoop and Hbase and provides services such as configuration maintenance, domain name service, distributed synchronization and the like for a plurality of application services, so that the directory file stored therein has one or more directory nodes including a root directory node, a child directory node, a descendant directory node and the like. More specifically, the directory structure indicated by the directory file corresponding to the data to be migrated in the Zookeeper is as follows:
root _ key:// root directory
// level _1_ key 1// first level directory node 1
// level _2_ key 1// second level directory node 1
Level _3_ key 1// third level directory node 1
// level _4_ key 1// fourth level directory node 1
// level _5_ key 1// fifth level directory node 1
// level _5_ key 2// fifth level directory node 2
// level _4_ key 2// fourth level directory node 2
// level _3_ key 2// third level directory node 2
// level _2_ key 2// second level directory node 2
// level _1_ key 2// first level directory node 2
Wherein, root _ key, level _1_ key1, level _1_ key2 and the like are directory nodes; further, root _ key is a root directory node, level _1_ key1 and level _1_ key2 are child directory nodes under the root directory, level _2_ key1 is a child directory node under level _1_ key1, and level _3_ key1 is a child directory node under level _2_ key1 or a child directory node under level _1_ key 1. Therefore, the same directory node can be a child directory node of another directory node or a descendant directory node of another directory node. That is, for any directory node, it may or may not have a next level of child directory nodes; in addition, service configuration information, a service port number, and service domain name data to be migrated may be stored in each directory node, and the corresponding data to be migrated may also be empty.
Step S102, traversing the directory file to obtain one or more directory node paths from the root directory to the directory nodes.
Specifically, still taking the directory file in the Zookeeper as an example for description, a directory node path traversing from a root directory node to any one directory node may be obtained. If the directory node level _1_ key1 is taken as an example for description, the corresponding directory node path is/root _ key/level _1_ key1, and the directory node level _2_ key1 is taken as an example for description, the corresponding directory node path is/root _ key/level _1_ key1/level 2_1_ key1, and further, one or more directory node paths obtained after traversing the directory file are taken as examples as follows:
/root_key
/root_key/level_1_key1
/root_key/level_1_key1/level_2_key1
/root_key/level_1_key1/level_2_key1/level_3_key1
/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1
/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1/level_5_key1
/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1/level_5_key2
/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key2
/root_key/level_1_key1/level_2_key1/level_3_key2
/root_key/level_1_key1/level_2_key2
/root_key/level_1_key2
step S103, acquiring data to be migrated stored in each directory node path.
Specifically, the corresponding data to be migrated may be obtained by directly reading the stored data in the directory node path. Since service configuration information, a service port number and service domain name data to be migrated may be stored under each directory node, and the corresponding data to be migrated may also be null, in order to ensure normal management and maintenance of the service after migration, when the data to be migrated is read to be null, the corresponding directory path node is still reserved, and the corresponding data to be migrated is directly recorded as null or blank.
And step S104, assembling the directory node path and the data to be migrated according to the data format of the second storage system.
Specifically, the second storage system is taken as the Etcd for illustration, and compared with Zookeeper, in terms of function implementation, the Etcd can capture all events only by calling watch operation once, so that the workload of a client developer is greatly simplified; in the aspect of code development, the Etcd uses Go language and the gPC protocol of Google, and the ZooKeeper uses Java language and its TCP protocol, so that a developer is limited by the SDK provided by the ZooKeeper official, and can only directly use Java and C languages for development, and the flexibility is low, so that the fact that the Etcd is adopted to replace the ZooKeeper to store configuration information and the like for managing and maintaining services is considered under some business scenes. Since the Etcd adopts a distributed key-value pair (key-value) database to store metadata such as configuration information and the like to perform coordinated management on the service, the directory node path and the data to be migrated are assembled in a key-value form under the condition that the Etcd is adopted to replace a Zookeeper; the directory node path is key, and the data to be migrated is value. Based on this, the assembled directory node path and the data to be migrated are exemplified as follows:
(/root_key,root_key_value)
(/root_key/level_1_key1,level_1_key1_value)
(/root_key/level_1_key1/level_2_key1,level_2_key1_value)
(/root_key/level_1_key1/level_2_key1/level_3_key1,level_3_key1_value)
(/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1,level_4_key1_value)
(/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1/level_5_key1,level_5_key1_value)
(/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1/level_5_key2,level_5_key2_value)
step S105, storing the assembled directory node path and the to-be-migrated data to the second storage system.
That is to say, the migration of data such as configuration information and the like stored in the Zookeeper of the first storage system to the second storage system Etcd is realized by synchronizing the assembled directory node path in the key-value form and the total amount of data to be migrated to the second storage system Etcd, and the application service can be managed and maintained by adopting the etod to replace the Zookeeper based on the migration.
It is understood that during the process of migrating data in the first storage system to the second storage system at a time or before the first storage system is switched to the second storage system, the directory node and the data to be migrated in the first storage system may still be changed, and therefore, in order to ensure high availability of the second storage system, in an alternative embodiment, the method further includes: monitoring the directory node corresponding to each directory node path, so as to process the directory node path stored in the second storage system and the data to be migrated according to the monitored event.
Wherein the event comprises any one or more of the following: the method comprises the following steps of creating an event of a directory node, deleting the event of the directory node, changing data of the directory node and deleting the event of a sub-directory node under the directory node. Specifically, the events that can be listened to (watch) in Zookeeper share the following:
EventNodeCreated: directory node creation event
EventNodeled: directory node delete event
EventNodeDataChanged: directory node data change events
EventNodeChildrenChanged: sub-directory node change event under directory node
The sub-directory node change event under the directory node specifically comprises sub-directory node creation, sub-directory node deletion and sub-directory node data change. However, in the embodiment of the present invention, each directory node, including but not limited to a root directory, a child directory node, a descendant directory node, and the like, is monitored for the directory node creation event, the directory node deletion event, and the directory node data change event, so as to avoid the duplication with the directory node lower child directory node change event, monitoring for the directory node lower child directory node deletion event is only reserved.
More specifically, one or more directory node paths obtained by traversing the directory file may be stored in the array variable, so as to monitor the directory node corresponding to each directory node path by traversing the array variable. It will be appreciated that monitoring may also be performed by writing directory node paths to a file or table, etc.
In an optional implementation manner, when a directory node data change event is monitored, the updated data to be migrated stored in the directory node path is obtained from the first storage system, so as to update the data to be migrated corresponding to the directory node path in the second storage system.
In an optional implementation manner, when a directory node deletion event is monitored, monitoring of the directory node is cancelled, and a directory node path and data to be migrated corresponding to the directory node in the second storage system are deleted.
In an optional implementation manner, when a directory node creation event is monitored, a directory node path corresponding to the created directory node and data to be migrated are stored in the second storage system, and the created directory node is monitored.
In an optional implementation manner, under the condition that a sub-directory node deletion event under a directory node is monitored, monitoring on the directory node and the sub-directory node under the directory node is cancelled; and deleting all directory node paths containing the directory nodes and the data to be migrated corresponding to the directory node paths in the second storage system.
Based on the embodiment, data migration between the first storage system and the second storage system with different data storage structures is realized by traversing the directory file in the first storage system, acquiring the directory node path from the directory to the directory node and the data to be migrated stored under the directory node path, and assembling the directory node path and the data to be migrated according to the data format of the second storage system to store the data to the second storage system, so that the first storage system can be directly switched to or used as the second storage system.
Referring to fig. 2, on the basis of the foregoing embodiment, an embodiment of the present invention provides another data migration method, which may specifically include the following steps:
step S201, determining a directory file corresponding to the data to be migrated in the first storage system.
Step S202, traverse the directory file to obtain one or more directory node paths from the root directory to the directory nodes.
Step S203, acquiring data to be migrated stored in each directory node path.
Step S204, the directory node path and the data to be migrated are assembled according to the data format of the second storage system.
Step S205, storing the assembled directory node path and the data to be migrated to the second storage system.
Step S206, monitoring the directory node corresponding to each directory node path. Specifically, when the directory node data change event is monitored, the following step S207 is continuously executed; in the case of monitoring the directory node deletion event, continuing to execute the following step S208; in the case of monitoring the directory node creation event, the following step S209 is continuously executed; in the case of monitoring the deletion event of the sub-directory node under the directory node, the following step S210 is continuously executed.
Step S207, obtaining the updated data to be migrated stored under the directory node path from the first storage system, so as to update the data to be migrated corresponding to the directory node path in the second storage system.
Specifically, for an example of monitoring the change of the directory node level _1_ key1 data corresponding to the directory node path/root _ key/level _1_ key1, the latest data to be migrated (for example, 152) stored in the directory node path/root _ key/level _1_ key1 is obtained from the first storage system, and the value in the key value pair (/ root _ key/level _1_ key1, level _1_ key1_ value) stored in the second storage system is updated to 152.
Step S208, canceling monitoring on the directory node, and deleting the directory node path and the data to be migrated corresponding to the directory node in the second storage system.
Specifically, taking an example of monitoring a directory node deletion event corresponding to the directory node level _2_ key1 as an example, the key value pairs (/ root _ key/level _1_ key1/level _2_ key1, level _2_ key1_ value) are deleted from the second storage system.
Step S209, store the directory node path corresponding to the created directory node and the data to be migrated to the second storage system, and monitor the created directory node.
Taking the example of monitoring the creation of a new directory node level _1_ key3 under the directory node root _ key as an example, the directory node level _1_ key3 is monitored for events such as a directory node creation event, a directory node deletion event, a directory node data change, a sub-directory node deletion event under the directory node, and the like, and at the same time, key value pairs (/ root _ key/level _1_ key3, level _1_ key3_ value) corresponding to the directory node level _1_ key3 are stored in the second storage system.
Step S210, canceling monitoring on the directory node and the sub-directory nodes under the directory node, and deleting all directory node paths including the directory node and the to-be-migrated data corresponding to the directory node paths in the second storage system.
Explaining by taking the example of monitoring the sub-directory node deletion event under the directory node corresponding to level _1_ key1, because directory nodes level _1_ key1 and the corresponding sub-directory nodes level _2_ key1, level _3_ key1, level _4_ key1, level _5_ key1 and other corresponding directory node paths all include directory node level _1_ key1, then monitoring of directory nodes level _1_ key1 and the corresponding sub-directory nodes level _2_ key1, level _3_ key1, level _4_ key1, level _5_ key1 and other directory nodes is cancelled; meanwhile, deleting the key-value pairs corresponding to the directory nodes level _1_ key1, level _2_ key1, level _3_ key1, level _4_ key1, level _5_ key1 and the like in the second storage system, namely deleting the following key-value pairs from the second storage system:
(/root_key/level_1_key1,level_1_key1_value)
(/root_key/level_1_key1/level_2_key1,level_2_key1_value)
(/root_key/level_1_key1/level_2_key1/level_3_key1,level_3_key1_value)
(/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1,level_4_key1_value)
(/root_key/level_1_key1/level_2_key1/level_3_key1/level_4_key1/level_5_key1,level_5_key1_value)
therefore, after the data in the first storage system is synchronized to the second storage system in a one-time full mode, the real-time performance and the effectiveness of the directory node path and the data to be migrated in the second storage system are guaranteed through monitoring the directory nodes, and the high availability of the second storage system is improved.
Referring to fig. 3, on the basis of the above embodiment, an embodiment of the present invention provides a data migration apparatus 300, including: a directory file determination module 301, a node path acquisition module 302, a migration data acquisition module 303, a data assembly module 304, and a data storage module 305: wherein the content of the first and second substances,
the directory file determining module 301 is configured to determine a directory file corresponding to data to be migrated in the first storage system;
the node path obtaining module 302 is configured to traverse the directory file to obtain one or more directory node paths from a root directory to directory nodes;
the migration data obtaining module 303 is configured to obtain data to be migrated, which is stored in each directory node path;
the data assembling module 304 is configured to assemble the directory node path and the data to be migrated according to a data format of a second storage system;
the data storage module 305 is configured to store the assembled directory node path and the data to be migrated to the second storage system.
In an optional embodiment, the data assembling module 304 is configured to assemble the directory node path and the data to be migrated in a key-value form; the directory node path is key, and the data to be migrated is value.
In an alternative embodiment, the data storage module 305 is further configured to,
monitoring the directory node corresponding to each directory node path, so as to process the directory node path stored in the second storage system and the data to be migrated according to the monitored event.
In an alternative embodiment, the event comprises any one or more of: the method comprises the following steps of creating an event of a directory node, deleting the event of the directory node, changing the data of the directory node and deleting the sub-directory node under the directory node.
In an alternative embodiment, the data storage module 305 is further configured to,
and under the condition that a directory node data change event is monitored, acquiring updated data to be migrated stored under a directory node path from the first storage system so as to update the data to be migrated corresponding to the directory node path in the second storage system.
In an alternative embodiment, the data storage module 305 is further configured to,
and under the condition that a directory node deletion event is monitored, canceling the monitoring of the directory node, and deleting a directory node path and data to be migrated corresponding to the directory node in the second storage system.
In an alternative embodiment, the data storage module 305 is further configured to,
and under the condition that the creation event of the directory node is monitored, storing the directory node path corresponding to the created directory node and the data to be migrated to the second storage system, and monitoring the created directory node.
In an optional implementation manner, under the condition that a sub-directory node deletion event under a directory node is monitored, monitoring on the directory node and the sub-directory node under the directory node is cancelled; and deleting all directory node paths containing the directory nodes and the data to be migrated corresponding to the directory node paths in the second storage system.
In an optional implementation manner, the data storage module 305 is configured to store the directory node paths to an array variable, so as to perform snooping on a directory node corresponding to each directory node path by traversing the array variable.
In an optional implementation manner, the first storage system is Zookeeper, and the second storage system is Etcd.
Fig. 4 illustrates an exemplary system architecture 400 to which the data migration method or the data migration apparatus of the embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. Various applications, such as a visualization application, may be installed on the terminal devices 401, 402, and 403 to show the progress of data migration.
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server providing various services, such as a background management server processing a data migration request sent by a user using the terminal devices 401, 402, and 403, so as to feed back a data migration progress to the terminal devices.
It should be noted that the data migration method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the data migration apparatus is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprises a directory file determining module, a node path acquiring module, a migration data acquiring module, a data assembling module and a data storage module element. The names of the modules do not constitute a limitation to the modules themselves in some cases, for example, the directory file determination module may also be described as a "module for determining a directory file corresponding to data to be migrated in the first storage system".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: determining a directory file corresponding to data to be migrated in a first storage system; traversing the directory file to obtain one or more directory node paths from a root directory to directory nodes; acquiring data to be migrated stored under each directory node path; according to the data format of a second storage system, assembling the directory node path and the data to be migrated; and storing the assembled directory node path and the data to be migrated to the second storage system.
According to the technical scheme of the embodiment of the invention, the data migration between the first storage system and the second storage system with different data storage structures is realized by traversing the directory file in the first storage system, acquiring the data to be migrated stored under the directory node path and the directory node path from the directory to the directory node, and assembling the directory node path and the data to be migrated according to the data format of the second storage system to store the data to the second storage system, so that the first storage system can be directly switched or changed into the second storage system.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (13)
1. A method of data migration, comprising:
determining a directory file corresponding to data to be migrated in a first storage system;
traversing the directory file to obtain one or more directory node paths from a root directory to directory nodes;
acquiring data to be migrated stored under each directory node path;
according to the data format of a second storage system, assembling the directory node path and the data to be migrated;
and storing the assembled directory node path and the data to be migrated to the second storage system.
2. The data migration method according to claim 1,
assembling the directory node path and the data to be migrated in a key-value form; the directory node path is key, and the data to be migrated is value.
3. The data migration method according to claim 2, further comprising:
monitoring the directory node corresponding to each directory node path, so as to process the directory node path stored in the second storage system and the data to be migrated according to the monitored event.
4. The data migration method according to claim 3,
the event comprises any one or more of the following: the method comprises the following steps of creating an event of a directory node, deleting the event of the directory node, changing the data of the directory node and deleting the sub-directory node under the directory node.
5. The data migration method according to claim 4,
and under the condition that a directory node data change event is monitored, acquiring updated data to be migrated stored under a directory node path from the first storage system so as to update the data to be migrated corresponding to the directory node path in the second storage system.
6. The data migration method according to claim 4,
and under the condition that a directory node deletion event is monitored, canceling the monitoring of the directory node, and deleting a directory node path and data to be migrated corresponding to the directory node in the second storage system.
7. The data migration method according to claim 4,
and under the condition that the creation event of the directory node is monitored, storing the directory node path corresponding to the created directory node and the data to be migrated to the second storage system, and monitoring the created directory node.
8. The data migration method according to claim 4,
under the condition that a subdirectory node deletion event under a directory node is monitored, monitoring of the directory node and the subdirectory node under the directory node is cancelled;
and deleting all directory node paths containing the directory nodes and the data to be migrated corresponding to the directory node paths in the second storage system.
9. The data migration method according to claim 3,
and storing the directory node paths to an array variable, and monitoring the directory nodes corresponding to the directory node paths in a mode of traversing the array variable.
10. The data migration method according to claim 1,
the first storage system is a Zookeeper, and the second storage system is an Etcd.
11. A data migration apparatus, comprising: the system comprises a directory file determining module, a node path acquiring module, a migration data acquiring module, a data assembling module and a data storage module, wherein the directory file determining module comprises: wherein the content of the first and second substances,
the directory file determining module is used for determining a directory file corresponding to the data to be migrated in the first storage system;
the node path acquisition module is used for traversing the directory file to acquire one or more directory node paths from a root directory to directory nodes;
the migration data acquisition module is used for acquiring the data to be migrated stored under each directory node path;
the data assembling module is used for assembling the directory node path and the data to be migrated according to the data format of the second storage system;
and the data storage module is used for storing the assembled directory node path and the data to be migrated to the second storage system.
12. An electronic device for data migration, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-10.
13. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110324765.0A CN113050890A (en) | 2021-03-26 | 2021-03-26 | Data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110324765.0A CN113050890A (en) | 2021-03-26 | 2021-03-26 | Data migration method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113050890A true CN113050890A (en) | 2021-06-29 |
Family
ID=76515461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110324765.0A Pending CN113050890A (en) | 2021-03-26 | 2021-03-26 | Data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113050890A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113468509A (en) * | 2021-07-05 | 2021-10-01 | 曙光信息产业(北京)有限公司 | User authentication migration method, device, equipment and storage medium |
| CN117370310A (en) * | 2023-10-19 | 2024-01-09 | 中电云计算技术有限公司 | Distributed file system cross-cluster data increment migration method |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7672981B1 (en) * | 2007-02-28 | 2010-03-02 | Emc Corporation | Object classification and indexing of very large name spaces using grid technology |
| CN102667772A (en) * | 2010-03-01 | 2012-09-12 | 株式会社日立制作所 | File level hierarchical storage management system, method, and apparatus |
| US20160092119A1 (en) * | 2014-09-29 | 2016-03-31 | International Business Machines Corporation | Data migration between different types of storage systems |
| CN105593804A (en) * | 2013-07-02 | 2016-05-18 | 日立数据系统工程英国有限公司 | Method and apparatus for virtualization of file system, data storage system for virtualization of file system, and file server for use in data storage system |
| CN106933508A (en) * | 2017-02-14 | 2017-07-07 | 深信服科技股份有限公司 | The moving method and device of application container |
| CN106953893A (en) * | 2017-02-15 | 2017-07-14 | 北京百度网讯科技有限公司 | Data Migration between cloud storage system |
| CN107391629A (en) * | 2017-06-30 | 2017-11-24 | 北京奇虎科技有限公司 | Data migration method, system, server and computer-readable storage medium between cluster |
| CN109150936A (en) * | 2017-06-19 | 2019-01-04 | 北京宝兰德软件股份有限公司 | The sharing method and device of session object Session in a kind of distributed type assemblies |
| CN109753229A (en) * | 2017-11-08 | 2019-05-14 | 华为技术有限公司 | Data migration method, device, network attached storage equipment and storage medium |
| CN110019208A (en) * | 2017-11-14 | 2019-07-16 | 中国移动通信有限公司研究院 | A kind of data migration method, device and medium |
| US20190220528A1 (en) * | 2018-01-18 | 2019-07-18 | International Business Machines Corporation | Effective handling of hsm migrated files and snapshots |
| CN110543520A (en) * | 2019-08-30 | 2019-12-06 | 京东数字科技控股有限公司 | data migration method and device |
-
2021
- 2021-03-26 CN CN202110324765.0A patent/CN113050890A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7672981B1 (en) * | 2007-02-28 | 2010-03-02 | Emc Corporation | Object classification and indexing of very large name spaces using grid technology |
| CN102667772A (en) * | 2010-03-01 | 2012-09-12 | 株式会社日立制作所 | File level hierarchical storage management system, method, and apparatus |
| CN105593804A (en) * | 2013-07-02 | 2016-05-18 | 日立数据系统工程英国有限公司 | Method and apparatus for virtualization of file system, data storage system for virtualization of file system, and file server for use in data storage system |
| US20160092119A1 (en) * | 2014-09-29 | 2016-03-31 | International Business Machines Corporation | Data migration between different types of storage systems |
| CN105530273A (en) * | 2014-09-29 | 2016-04-27 | 国际商业机器公司 | Method and device for data migration |
| CN106933508A (en) * | 2017-02-14 | 2017-07-07 | 深信服科技股份有限公司 | The moving method and device of application container |
| CN106953893A (en) * | 2017-02-15 | 2017-07-14 | 北京百度网讯科技有限公司 | Data Migration between cloud storage system |
| CN109150936A (en) * | 2017-06-19 | 2019-01-04 | 北京宝兰德软件股份有限公司 | The sharing method and device of session object Session in a kind of distributed type assemblies |
| CN107391629A (en) * | 2017-06-30 | 2017-11-24 | 北京奇虎科技有限公司 | Data migration method, system, server and computer-readable storage medium between cluster |
| CN109753229A (en) * | 2017-11-08 | 2019-05-14 | 华为技术有限公司 | Data migration method, device, network attached storage equipment and storage medium |
| CN110019208A (en) * | 2017-11-14 | 2019-07-16 | 中国移动通信有限公司研究院 | A kind of data migration method, device and medium |
| US20190220528A1 (en) * | 2018-01-18 | 2019-07-18 | International Business Machines Corporation | Effective handling of hsm migrated files and snapshots |
| CN110543520A (en) * | 2019-08-30 | 2019-12-06 | 京东数字科技控股有限公司 | data migration method and device |
Non-Patent Citations (1)
| Title |
|---|
| 李东阳;刘鹏;田浪军;: "基于SSD的云存储主服务器元数据管理研究", 计算机技术与发展, no. 10, 24 July 2013 (2013-07-24) * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113468509A (en) * | 2021-07-05 | 2021-10-01 | 曙光信息产业(北京)有限公司 | User authentication migration method, device, equipment and storage medium |
| CN113468509B (en) * | 2021-07-05 | 2024-01-30 | 曙光信息产业(北京)有限公司 | User authentication migration method, device, equipment and storage medium |
| CN117370310A (en) * | 2023-10-19 | 2024-01-09 | 中电云计算技术有限公司 | Distributed file system cross-cluster data increment migration method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10951702B2 (en) | Synchronized content library | |
| US10484456B2 (en) | Sharing a content item | |
| US10936423B2 (en) | Enhanced application write performance | |
| CN111427701A (en) | Workflow engine system and business processing method | |
| CN113050890A (en) | Data migration method and device | |
| CN112965761B (en) | Data processing method, system, electronic equipment and storage medium | |
| CN115517009B (en) | Cluster management method, cluster management device, storage medium and electronic equipment | |
| CN111290871A (en) | Method and device for acquiring crash information of application program | |
| CN111857720A (en) | Method and device for generating user interface state information, electronic equipment and medium | |
| CN113364887A (en) | File downloading method based on FTP, proxy server and system | |
| CN110674082B (en) | Method and device for removing online document, electronic equipment and computer readable medium | |
| CN108810130B (en) | Method and device for planning distribution request | |
| CN115480877A (en) | External exposure method and device of application service in multi-cluster environment | |
| CN115442129A (en) | Method, device and system for managing cluster access authority | |
| CN113742376A (en) | Data synchronization method, first server and data synchronization system | |
| CN110058790B (en) | Method, apparatus and computer program product for storing data | |
| CN113761075A (en) | Method, device, equipment and computer readable medium for switching databases | |
| CN110851192A (en) | Method and device for responding to configuration of degraded switch | |
| CN113239038A (en) | Data processing method and device | |
| CN117389475A (en) | Data processing method and device | |
| CN114036321A (en) | Request processing method and device, electronic equipment and computer readable medium | |
| CN114490128A (en) | Data storage method and device under remote multi-active architecture | |
| CN113138975A (en) | Data migration method and device | |
| CN113760836A (en) | Wide table calculation method and device | |
| CN112882741A (en) | Application management system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |