CN115982101A - Machine room data migration method and device based on multi-machine room copy placement strategy - Google Patents
Machine room data migration method and device based on multi-machine room copy placement strategy Download PDFInfo
- Publication number
- CN115982101A CN115982101A CN202310258349.4A CN202310258349A CN115982101A CN 115982101 A CN115982101 A CN 115982101A CN 202310258349 A CN202310258349 A CN 202310258349A CN 115982101 A CN115982101 A CN 115982101A
- Authority
- CN
- China
- Prior art keywords
- machine room
- room
- copy
- data
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013508 migration Methods 0.000 title claims abstract description 105
- 230000005012 migration Effects 0.000 title claims abstract description 105
- 238000000034 method Methods 0.000 title claims abstract description 89
- 230000010076 replication Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a machine room data migration method and device based on a multi-machine room copy placement strategy, wherein the method comprises the following steps: modifying a copy placing strategy of a metadata node of an old machine room into a multi-machine-room copy placing strategy, and setting a machine room written in the multi-machine-room copy placing strategy as the old machine room; newly adding a data node in a new machine room, reporting the data node to a metadata node of the old machine room, and writing a copy placing strategy of the multiple machine rooms into the machine rooms to set the multiple machine rooms as the old machine room and the new machine room; migrating copies of all file blocks of the migrated file to data nodes of the new machine room based on the AzMover migration method; and taking the data nodes of the old machine room off line, and migrating the metadata nodes of the old machine room to the new machine room to finish the data migration of the machine room. According to the method and the system, a multi-machine-room copy placement strategy and an AzMover migration method are adopted, excessive cross-machine-room flow is avoided, and accurate control over bandwidth flow is achieved.
Description
Technical Field
The application relates to the technical field of computers, in particular to a computer room data migration method and device based on a multi-computer-room copy placement strategy.
Background
With the advent of the cloud-native era, multiple clouds and multiple machine rooms have become mainstream architectures of various internet and science and technology companies. The basic cloud service provided by the cloud manufacturer can enable a served party to avoid a large amount of basic operation and maintenance, so that the service is intensively researched and developed quickly. However, the cloud architecture also brings about many problems, such as the quality of service of cloud manufacturers is not too high, the price rises due to the contract and the renewal fee, and the like. In the face of such problems, a common solution is to replace a cooperative cloud manufacturer, but for a business company, all data and services need to be migrated to a machine room provided by a new cloud manufacturer.
At present, for a mainstream Distributed storage component HDFS (Hadoop Distributed File System), a Distcp tool is generally used for data migration, and performing data migration based on the Distcp tool mainly includes: and copying stock data of the old machine room to the new machine room by adopting a Distcp tool, stopping the service of the old machine room after the stock data is copied, copying the incremental data, and migrating all the services. However, the migration data based on Distcp has the following disadvantages: data consistency is difficult to guarantee, migrated data needs to be verified regularly for a long time, and users are prevented from being updated or deleted; when data is copied, cross-machine room flow is difficult to control, and special line bandwidth is easy to fill; when cluster cutting is carried out, the cutting needs to be carried out in a stopping way, and in the cutting, the time for copying the incremental data and migrating all the upper-layer services based on the HDFS is long, so that the stopping time is long, the risk is uncontrollable, and the cutting failure is easily caused.
Disclosure of Invention
The application provides a machine room data migration method and device based on a multi-machine room copy placement strategy, and aims to solve the problems that in the prior art, data consistency cannot be guaranteed, cross-machine room flow cannot be controlled, and the risk of outage is uncontrollable.
In a first aspect, the present application provides a machine room data migration method based on a multi-machine room copy placement policy, where the method includes:
modifying a copy placing strategy of a metadata node of an old machine room into a multi-machine-room copy placing strategy, and writing a machine room in the multi-machine-room copy placing strategy to be set as the old machine room;
newly adding a data node in a new machine room, reporting the data node to a metadata node of the old machine room, and writing a copy placing strategy of the multiple machine rooms into the machine rooms to set the multiple machine rooms as the old machine room and the new machine room; the number of the newly added data nodes in the new machine room is the number of the data nodes in the old machine room;
migrating copies of all file blocks of the migrated file to data nodes of the new machine room based on the AzMover migration method;
and taking the data nodes of the old machine room off the line, and migrating the metadata nodes of the old machine room to the new machine room to finish the data migration of the machine room.
According to the computer room data migration method based on the multi-computer-room copy placement strategy, the step of migrating the copies of the file blocks of the migrated file to the data nodes of the new computer room based on the AzMover migration method comprises the following steps: for each file block of the migrated file, determining a data node set where a copy of each file block is located, and determining a copy source node and a node of a copy to be deleted, wherein the node of the copy to be deleted is a data node with the smallest residual storage space in the data node set; determining a target node from the data nodes of the new machine room based on the principles of load balancing and storage balancing; copying the copies of the file blocks of the migrated file from the replication source node to the target node, and deleting the corresponding copies in the nodes of which the copies are to be deleted.
According to the computer room data migration method based on the multi-computer-room copy placement strategy, the method for determining the target node from the data nodes of the new computer room based on the principles of load balancing and storage balancing comprises the following steps: randomly selecting two data nodes from the data nodes of the new machine room as candidate nodes aiming at the HDFS single cluster with low concurrent migration file blocks, and determining the nodes with more residual storage spaces in the candidate nodes as target nodes; aiming at the HDFS single cluster with uniformly stored data nodes, arranging the data nodes of the new machine room in a descending order according to the remaining storage space at intervals, selecting a first number of data nodes with front positions as target nodes, and setting concurrence of the target nodes to be the same value; and aiming at the single HDFS cluster with the number of data nodes conforming to the large cluster scale, arranging the data nodes of the new computer room in a descending order according to the residual storage space, selecting a second number of data nodes at the front position as a candidate node set, randomly selecting a third number of data nodes from the candidate node set as target nodes, and setting the concurrence of the target nodes to be the same value.
According to the machine room data migration method based on the multi-machine room duplicate placement strategy, the duplication source node is provided with a flow configuration item, and the flow configuration item is used for controlling the flow of duplicate copies.
According to the computer room data migration method based on the multi-computer-room copy placement strategy, the method for determining the copy source node comprises the following steps: if the new computer room does not have a copy of the file block to be migrated, taking the data node with the lowest load in the data node set as a copy source node; and if the new machine room has the copy of the file block to be migrated, taking the data node of the copy of the file block to be migrated in the new machine room as a replication source node.
According to the computer room data migration method based on the multi-computer-room copy placement strategy, before the copy placement strategy of the metadata node of the old computer room is modified into the multi-computer-room copy placement strategy, the method further comprises the following steps: defining a network topology class, and constructing a network topology of the HDFS based on the network topology class; and the network topology of the HDFS is used for enabling the metadata node of the old machine room to sense the multi-machine room copy placement strategy.
According to the computer room data migration method based on the multi-computer-room copy placement strategy, the multi-computer-room copy placement strategy is used for selecting a data node of a target computer room to write in a copy according to the multi-computer-room copy placement strategy of a file to be deployed, deleting the copy of the data node of the target computer room according to the multi-computer-room copy placement strategy of the file to be deployed, and controlling the number of times of crossing a private line when the copy is written.
In a second aspect, the present application further provides a computer room data migration apparatus based on a multi-computer-room copy placement policy, where the apparatus includes:
the system comprises a first setting module, a second setting module and a third setting module, wherein the first setting module is used for modifying a copy placing strategy of a metadata node of an old machine room into a multi-machine-room copy placing strategy and writing a machine room in the multi-machine-room copy placing strategy into the old machine room;
the second setting module is used for newly adding a data node in a new machine room, reporting the newly added data node to a metadata node of the old machine room, and writing the multi-machine-room copy placing strategy into the machine room to set the machine room as the old machine room and the new machine room; the number of the newly added data nodes in the new machine room is the number of the data nodes in the old machine room;
the copy migration module is used for migrating the copies of all the file blocks of the migrated files to the data nodes of the new computer room based on the AzMover migration method;
and the service migration module is used for taking the data nodes of the old machine room off line and migrating the metadata nodes of the old machine room to the new machine room to finish the data migration of the machine room.
In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores a computer program, and when the processor runs the computer program, the processor executes the steps in any implementation manner of the computer room data migration method based on the multi-computer-room copy placement policy.
In a fourth aspect, an embodiment of the present application further provides a readable storage medium, where a computer program is stored in the readable storage medium, and when the computer program runs on a processor, the computer program executes the steps in any implementation manner of the computer room data migration method based on the multi-computer-room copy placement policy.
In summary, the machine room data migration method and device based on the multi-machine room copy placement strategy enable the HDFS to quickly sense the machine room where the copies are placed by constructing the network topology of the HDFS and modifying the copy placement strategy into the multi-machine room copy placement strategy; incremental data are written into a new machine room and an old machine room simultaneously, so that cross-machine-room flow is further avoided; by adopting the AzMover migration method, the single copy of each file block is firstly copied across the computer room, and then the other copies of each file block are copied in the same computer room, so that excessive flow across the computer room is avoided, and the data migration of the computer room is completed without service interruption; meanwhile, the copy source node, the node to be deleted the copy and the target node are determined based on load balancing and storage balancing, so that the load balancing and storage balancing of the HDFS are ensured; the flow configuration items of the replication source nodes are adjusted, so that the accurate control of bandwidth flow is realized, the bandwidth fullness can be avoided, and the migration efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a machine room data migration method based on a multi-machine room copy placement policy provided in the present application;
fig. 2 is a schematic flow diagram illustrating a process of migrating copies of file blocks of a migrated file to data nodes of the new machine room based on an AzMover migration method according to the present application;
fig. 3 is a schematic structural diagram of a machine room data migration apparatus based on a multi-machine room copy placement policy provided in the present application;
fig. 4 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Icon: 300-a machine room data migration apparatus; 310-a first setup module; 320-a second setup module; 330-copy migration module; 340-a service migration module; 400-an electronic device; 410-a memory; 420-a processor; 430-bus.
Detailed Description
To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Fig. 1 is a schematic flow diagram of a machine room data migration method based on a multi-machine-room copy placement policy, which is applied to an HDFS single cluster, where the HDFS single cluster includes a Client, a metadata node (NameNode), and a data node (DataNode), the metadata node is deployed in a main machine room of the HDFS single cluster, and the data node is deployed in multiple machine rooms. The metadata node is a main node in the HDFS single cluster, which is used for receiving metadata requests initiated by various client users, such as directory checking, directory creation, directory deletion and the like, and does not provide real data read-write service. The data nodes are nodes used for actually storing data in the HDFS single cluster, and receive requests of users for reading and writing data, such as reading files, writing files and the like. In addition, there are generally a plurality of metadata nodes in the HDFS single cluster, and an Active/Standby mode is adopted, that is, the metadata nodes are divided into a main metadata node (Active NameNode) and a plurality of Standby metadata nodes (Standby NameNode) to provide a high-availability service.
It should be noted that, when the HDFS stores a file, the file is divided into a plurality of file blocks by a fixed size (e.g., 64M or 128M), each file block copies a plurality of (generally three) identical copies, and the copies of each file corresponding to each file are stored in different data nodes according to a file copy placement policy, where the file blocks are independent of each other.
As shown in fig. 1, the computer room data migration method based on the multi-computer-room copy placement policy includes:
s101, modifying a copy placing strategy of a metadata node of an old machine room into a multi-machine-room copy placing strategy, and writing a copy in the multi-machine-room copy placing strategy into the machine room to set the copy as the old machine room.
The multi-machine room copy placement strategy is used for selecting a data node of a target machine room to write in a copy according to the multi-machine room copy placement strategy of the file to be deployed, deleting the copy of the data node of the target machine room according to the multi-machine room copy placement strategy of the file to be deployed, and controlling the number of times of crossing a special line when the copy is written.
Specifically, it can be understood that the multiple machine room copy placement strategy can sense the machine room where the copy is placed, and select the data node of the target machine room to write in the copy or delete the copy of the data node of the target machine room strictly according to the copy placement strategy of the file to be deployed, for example, the multiple machine room copy placement strategy of the file to be deployed is "az1, az1, az2", when the copy of the file to be deployed is written, the data node is determined and written in the machine room 1, and the machine room 2 strictly according to the order of the multiple machine room copy placement strategy in a pipeline form, so that the file is written across a dedicated line only once, and the number of times of crossing the dedicated line is reduced as much as possible; when the copies of the file are deleted, the file is deleted according to the multi-computer-room copy placing strategy, if the file currently has three copies, the copy placed in the computer room 2 is deleted when one copy needs to be deleted, and the copy placed in the computer room 2 is deleted firstly and then the copy in the computer room 1 is deleted one when two copies need to be deleted.
It should be noted that the copy placement policy of the HDFS single cluster original generally adopts a random placement manner, and the metadata node determines the target data node according to the zoosettarget method in the org. Therefore, before migrating the computer room data, the copy placement strategy needs to be modified into a multi-computer room copy placement strategy.
In particular, it can be further understood that in the HDFS default implementation, all data nodes may be placed in a default rack, and the metadata node cannot sense the rack in which each data node is located. Therefore, a network topology of the HDFS needs to be constructed, so that the metadata nodes are aware of the distribution of the data nodes, so as to implement a multi-chassis copy placement policy.
In some embodiments, before step S101, the method for migrating machine room data based on a multi-machine-room copy placement policy further includes: defining a network topology class, and constructing a network topology of the HDFS based on the network topology class; the network topology of the HDFS is used for enabling the metadata node of the old machine room to perceive the multi-machine-room copy placement strategy. The defining of the network topology class and the construction of the network topology of the HDFS based on the network topology class specifically comprise the following steps:
step a1, defining a network topology class LocalFileDNSToSwitchMapping;
step a2, writing the incidence relation between the data nodes and the rack and the machine room into a local file;
and a3, analyzing the local file by using the network topology class LocalFileDNSToSwitchMapping to obtain the network topology of the HDFS.
According to the method for constructing the network topology of the HDFS based on the network topology, the multi-machine room copy placement strategy is supported, meanwhile, the problems that the calling speed of the shell script is uncontrollable, the resolving performance is poor and the like in the network topology of constructing the HDFS by adopting the original script based mapping of the HDFS in the prior art are solved, the query is carried out through the local file storing the association relation between the data node and the rack as well as the machine room, even if the client node which cannot be resolved is encountered, the result can be returned quickly, and the follow-up resolution cannot be influenced.
In the implementation process of the method, the network topology of the HDFS is constructed through the defined network topology class, so that the HDFS can sense a rack and a machine room where each data node is located, and the query speed is improved; by modifying the copy placement strategy of the metadata node into a multi-computer room copy placement strategy, the HDFS can sense the computer room where the copy is placed, and subsequent operations such as copy migration are facilitated.
And S102, adding a data node in a new machine room, reporting to a metadata node of the old machine room, and writing copies in the multi-machine-room copy placing strategy into the machine room to set the old machine room and the new machine room.
And the number of the newly added data nodes in the new machine room is the number of the data nodes in the old machine room.
Specifically, it can be understood that, in order to migrate all data of the old machine room to the new machine room, data nodes with the same data volume are newly added to the new machine room, so as to receive data of the old machine room, and simultaneously report to the metadata nodes of the old machine room, so that the newly added data nodes of the new machine room and the old machine room are under the same HDFS single cluster.
Specifically, it can be further understood that when a data node is newly added to a new computer room, copies written in the computer room in the multi-computer-room copy placement strategy are set as the old computer room and the new computer room, so that in a computer room data migration process, copies of incremental data generated in the old computer room can be written in the new computer room and the old computer room at the same time, and the incremental data is prevented from being copied again after migration. It is worth noting that any task request in the process needs to be simultaneously operated in the new machine room and the old machine room so as to achieve the purpose that incremental data are simultaneously written into the two machine rooms, if the task request is a read request, the new machine room is required to read a file copy placed in the new machine room, and the old machine room is required to read a file copy placed in the old machine room, so that cross-machine room flow is prevented.
In the implementation process of the method, the copies in the multi-machine-room copy placement strategy are written into the machine rooms and are set as the old machine room and the new machine room, so that incremental data generated in the migration process can be written into the two machine rooms simultaneously, the copies of the machine rooms can be read during reading, and cross-machine-room flow is prevented.
And S103, migrating the copy of each file block of the migrated file to the data node of the new machine room based on the AzMover migration method.
The AzMover migration method is a method for performing copy migration on a file block level based on an AzMover tool designed and developed by a user, and can be used for migrating data of an old computer room to a new computer room, and also can be used for migrating file block copies of all files in one or more directories to data nodes of a target computer room according to a multi-computer-room copy placement strategy of the files. In addition, the AzMover migration method can stably and accurately control the bandwidth flow, especially the cross-machine room bandwidth, by adjusting the flow configuration items of the data nodes.
The AzMover tool includes a FileVistor component, a MoverExecutor actuator. Aiming at a certain HDFS single cluster, the AzMover migration method specifically comprises the following steps:
step b1, receiving at least one migration path, and analyzing by adopting a FileVistor assembly to obtain a file list of the migration file;
specifically, the migration path is a path of a file or a directory, and after the AzMover tool receives the migration path, the FileVistor component performs analysis to obtain a file list.
Step b2, acquiring the meta information of the migrated file from a meta data node based on the file list of the migrated file;
specifically, the metadata node records metadata of all files and directories, and the metadata of the corresponding file can be directly obtained from the metadata node based on the file list of the migrated file.
Step b3, analyzing by adopting a FileVistor component based on the meta information of the migration file to obtain a file block list of the migration file;
and b4, based on the file block list of the migrated file, migrating the copy of each file block of the migrated file by adopting a MoverExecutor executor.
Specifically, a data node set where a corresponding copy is located is determined for each file block to be migrated, the data node with the lowest load is used as a copy source node, and the data node with the smallest residual storage space is used as a node of the copy to be deleted; determining a target machine room according to a multi-machine-room copy placing strategy of a file corresponding to each file block to be migrated, and taking a data node with the largest storage space in the target machine room as a target node; and copying the copy of each file block to be migrated from the replication source node to the target node, and deleting the corresponding copy in the node of which the copy is to be deleted.
It should be noted that the AzMover migration method described above supports concurrency, and can greatly improve the migration efficiency of the copy. In addition, the azMover migration method can be used for migrating file block copies, migrating EC (Erasure Coding) copies corresponding to files, and modifying the number of copies when the number of file copies of an old machine room does not support excessive reading requests, so that the HDFS can better support the reading and writing requests of users. However, when the EC copies corresponding to the file are migrated, because the copies of each file block of the EC file are different, all the EC copies corresponding to the migrated file need to be copied, and at this time, cross-machine-room traffic cannot be optimized.
It should be noted that, in the process of migrating the copies of the file blocks of the migrated file to the data nodes of the new machine room, in order to ensure that the clients of the old machine room can read the files normally, it is necessary to ensure that at least one file copy, i.e., a corresponding copy of each file block, is left in the old machine room during the migration process, so that the clients of the old machine room can read the file copies of the machine room normally, thereby avoiding generating flow across the machine rooms.
In the implementation process of the method, the copy migration of the file blocks is carried out based on the AzMover migration method, and compared with the prior art that stock data of an old machine room is copied based on Distcp, the migration is carried out on the level of the file blocks without checking the consistency of files; determining a copy source node, a node to be copied and a target node based on load balancing and storage balancing, and ensuring the load balancing and storage balancing of the HDFS; meanwhile, the bandwidth flow can be accurately controlled.
And S104, taking the data node of the old machine room off the line, and migrating the metadata node of the old machine room to the new machine room to finish the data migration of the machine room.
Specifically, it can be understood that after all the file block copies of the data node in the old computer room are copied to the data node in the new computer room, the data node in the old computer room is offline. And simultaneously, migrating the metadata node to a new machine room to finish the data migration of the machine room. It is worth noting that in the process of migrating the metadata node, migration can be completed without stopping service by switching the main/standby mode of the metadata node.
In some embodiments, while taking the data node of the old machine room off-line, the method further comprises: and setting the machine room written in the multi-machine-room copy placing strategy as the new machine room.
According to the machine room data migration method based on the multi-machine room copy placement strategy, the network topology of the HDFS is built, and the copy placement strategy is modified into the multi-machine room copy placement strategy, so that the HDFS can quickly sense the machine room where the copies are placed; incremental data are written into a new machine room and an old machine room simultaneously, so that cross-machine-room flow is further avoided; the copy of the file block is migrated by adopting the AzMover migration method, the copy source node, the node to be copied and the target node are determined based on load balancing and storage balancing, the load balancing and storage balancing of the HDFS are ensured, the AzMover migration method can also accurately control the bandwidth flow, and the data migration of a machine room is completed under the condition of no service halt.
Fig. 2 is a schematic flow diagram illustrating a process of migrating copies of file blocks of a migrated file to data nodes of the new machine room based on the AzMover migration method provided in the present application. As shown in fig. 2, the migrating the copy of each file block of the migrated file to the data node of the new machine room based on the AzMover migration method includes:
s201, aiming at each file block of the migrated file, determining a data node set where each file block copy is located, and determining a copy source node and a node of the copy to be deleted.
The node of the to-be-deleted replica is a data node with the minimum residual storage space in the data node set; the replication source node is provided with a flow configuration item, and the flow configuration item is used for controlling the flow of the replica copy; determining a replication source node, comprising: if the new computer room does not have a copy of the file block to be migrated, taking the data node with the lowest load in the data node set as a copy source node; and if the new machine room has the copy of the file block to be migrated, taking the data node of the copy of the file block to be migrated in the new machine room as a replication source node.
Specifically, it can be understood that, since the data migration of the computer room is to migrate the data of the old computer room into the new computer room, the data node set of a certain file block is the data node set storing the copy of the file block in the old computer room, and the node from which the copy is to be deleted is the data node with the smallest remaining storage space.
In particular, it can also be understood that the meaning of the traffic configuration item is that each data node is the maximum bandwidth that can be used at the copy. Because the process of copy copying is divided into cross-machine-room copying and same-machine-room copying, if the process is cross-machine-room copying, the bandwidth flow cannot fill the special line bandwidth, and cannot be too small, and the too small bandwidth can affect the migration efficiency, the parameter value of the flow configuration item can be determined according to the special line bandwidth, and the specific calculation formula is as follows: the parameter value of the flow configuration item = target private line bandwidth/target node data; if the data nodes are copied in the same computer room, the copy bandwidth is almost unlimited, so that the values of the parameter values of the flow configuration items are not limited and are high enough under the condition that the load of each data node is not overhigh.
In some embodiments, if the replica copy is cross-machine-room copy and 128 target nodes are selected for writing, and assuming that the target dedicated line bandwidth is 512Gb/s, the parameter value of the traffic configuration item is set to 512MB/s, which can ensure that the dedicated line bandwidth maintains fluctuation at 512Gb/s when the replica copy is performed, thereby not only avoiding the full dedicated line bandwidth, but also ensuring the migration efficiency. Wherein, 1GB/s =8Gb/s, and 1GB/s =1024MB/s.
In some embodiments, if the duplicate copy is the same as the machine room copy and 1000 target nodes are selected for writing, and the concurrence of each target node is set to 1, assuming that the parameter value of the traffic configuration item is set to 1GB/s, through experiments of data migration of an actual machine room, it is found that the bandwidth traffic of the whole HDFS cluster can be maintained within 100 to 200gb/s, and data of more than 10PB can be copied in one day.
S202, determining a target node from the data nodes of the new machine room based on the principles of load balancing and storage balancing.
Specifically, it can be understood that the load balancing is to make the read-write requests of the data nodes as uniform as possible without affecting normal reading and writing of users, and the storage balancing is to migrate the copy of the file block to the data node with relatively free storage as possible. The method for determining the target node under different scenes is designed by analyzing the number and the storage condition of the data nodes of the HDFS single cluster, and specifically comprises the following steps:
(1) And aiming at the HDFS single cluster with low concurrent migration file blocks, randomly selecting two data nodes from the data nodes of the new computer room as candidate nodes, and determining the nodes with more residual storage spaces in the candidate nodes as target nodes.
Specifically, for the case that the concurrence of the azMover migration method is low, in order to ensure load balancing and storage balancing, two data nodes are randomly selected as candidate nodes, and then the data nodes with more remaining storage spaces are determined as target nodes.
(2) And aiming at the single HDFS cluster with uniformly stored data nodes, arranging the data nodes of the new computer room in a descending order according to the remaining storage space at intervals, selecting a first number of data nodes with front positions as target nodes, and setting concurrence of the target nodes to be the same value.
Specifically, for the condition that the data nodes are stored uniformly, the target nodes are determined at intervals, that is, the data nodes are arranged in a descending order according to the remaining storage space, and then a first number of data nodes at the front positions are selected as the target nodes, wherein the first number is not fixed, and can be adjusted according to the actual condition, for example, the number of the target nodes is determined according to the actual read-write requirement, so as to determine the first number. In the method, the load balance and the storage balance of each data node are ensured by rotating the target nodes.
In some embodiments, the period of time may be 1min or 2min, the target node may be determined at intervals, and the target node may be rotated every 1min or every 2min, and the period of time may be specifically adjusted according to an actual situation.
(3) And aiming at the single HDFS cluster with the number of data nodes conforming to the large cluster scale, arranging the data nodes of the new computer room in a descending order according to the residual storage space, selecting a second number of data nodes at the front position as a candidate node set, randomly selecting a third number of data nodes from the candidate node set as target nodes, and setting the concurrence of the target nodes to be the same value.
Specifically, the number of data nodes meeting the large cluster size means that the number of data nodes is higher than a certain threshold, for example, a cluster in which the number of data nodes is more than 1000 may be referred to as a large-scale cluster; and aiming at the condition that the number of the data nodes accords with the large cluster scale, arranging the data nodes of the new machine room in a descending order according to the size of the residual storage space, selecting a second number of data nodes at the front position as a candidate node set, and then randomly selecting a third number of data nodes from the candidate node set as target nodes.
The third number is not fixed, and can be adjusted according to actual conditions, for example, the number of target nodes is determined according to actual read-write requirements, and then the third number is determined; the second quantity is adjusted according to the number of data nodes of the single HDFS cluster, and is greater than the third quantity, in some embodiments, the second quantity may be 2/3 of the number of data nodes of the single HDFS cluster, for example, 3000 data nodes are total in the single HDFS cluster, the second quantity may be 2000, that is, after the data nodes are arranged in a descending order according to the remaining storage space, the first 2000 data nodes are selected to form a candidate node set, then a batch of data nodes is randomly selected from the 2000 data node sets to serve as a target node, and if the third quantity is 500, 500 data nodes may be randomly selected to serve as the target node.
It is worth noting that for the single HDFS cluster with the data node number meeting the large cluster scale, the concurrency value of the target node needs to be set within an allowable range, and normal read-write requests are prevented from being influenced.
S203, copying the copy of each file block of the migration file from the replication source node to the target node, and deleting the corresponding copy in the node of the copy to be deleted.
Specifically, it can be understood that, for the duplicates of each file block, a replication source node, a node to be replicated, and a target node corresponding to each file block are respectively determined, and then the duplicates are copied from the replication source node to the target node, and the corresponding duplicates in the node to be replicated are deleted, so that one-time duplicate migration is completed.
Specifically, it may also be understood that migrating the copy of each file block of the migration file to the data node of the new computer room specifically includes: firstly, migrating a single copy of each file block to a data node of the new machine room; and then sequentially migrating the rest copies of each file block to the data nodes of the new computer room.
It should be noted that the single copy of each file block is migrated as a cross-machine-room copy. At the moment, the data node which stores the copy of the file block to be migrated in the old machine room and has the lowest load is used as the copy source node, so that the normal reading and writing requirements of a user cannot be influenced in the data migration process of the machine room; the data node which stores the copy of the file block to be migrated in the old machine room and has the minimum residual storage space is used as the node of the copy to be deleted, so that the storage space of the data can be released; meanwhile, the bandwidth flow copied across the machine room can be accurately controlled by copying the flow configuration item of the source node.
It should be further noted that the remaining copies of each file block are migrated to the same computer room copy. Because the new computer room already has the copy of the file block to be migrated, the data node in the new computer room with the copy of the file block to be migrated is used as the copy source node, so that the generation of cross-computer-room traffic can be avoided, and at the moment, the node of the copy to be deleted is still the data node in the old computer room, which stores the copy of the file block to be migrated and has the smallest residual storage space. In addition, if there are multiple data nodes with copies of file blocks to be migrated in the new machine room, the data node with the lowest load may be selected as the copy source node.
According to the AzMover-based migration method, the copies of the file blocks of the migrated file are migrated to the data nodes of the new machine room, and the single copy of each file block is copied across the machine room first, and then the other copies of each file block are copied from the machine room, so that excessive cross-machine-room flow is avoided; the lowest load is used as a copy source node, the smallest residual storage space is used as a node of a copy to be deleted, and different target node selection methods are designed, so that the load balance and the storage balance of the HDFS are achieved as much as possible in the migration process based on the AzMover migration method; by adjusting the flow configuration item of the replication source node, the accurate control of the bandwidth flow is realized, the bandwidth fullness can be avoided, and the migration efficiency is improved.
Fig. 3 is a schematic structural diagram of a computer room data migration apparatus based on a multi-computer-room copy placement policy, which can be used to implement the method described in the foregoing embodiment. As shown in fig. 3, the apparatus includes:
a first setting module 310, configured to modify a copy placement policy of a metadata node of an old machine room into a multiple machine room copy placement policy, and set a machine room written in the multiple machine room copy placement policy as the old machine room;
a second setting module 320, configured to report the new data node in the new machine room to the metadata node of the old machine room, and write the multiple machine room copy placement policy into the machine room, where the multiple machine room copy placement policy is set as the old machine room and the new machine room; the number of the newly added data nodes in the new machine room is the number of the data nodes in the old machine room;
the copy migration module 330 is configured to migrate copies of the file blocks of the migrated file to the data nodes of the new machine room based on the AzMover migration method;
and the service migration module 340 is configured to take the data node of the old machine room off line, and migrate the metadata node of the old machine room to the new machine room to complete machine room data migration.
For a detailed description of the machine room data migration apparatus based on the multi-machine room copy placement policy, please refer to the description of the related method steps in the above embodiments, and repeated details are not repeated. The above-described apparatus embodiments are merely illustrative, and "modules" used herein as separate components may be a combination of software and/or hardware for implementing predetermined functions, and may or may not be physically separate. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Fig. 4 is a schematic structural diagram of an electronic device provided in the present application, and as shown in fig. 4, the electronic device includes a memory 410 and a processor 420, where the memory 410 stores a computer program, and the processor 420 executes, when running the computer program, steps in a computer room data migration method based on a multi-computer-room copy placement policy.
The embodiment of the application also provides a readable storage medium, wherein a computer program is stored in the readable storage medium, and when the computer program runs on a processor, the computer program executes the steps in the computer room data migration method based on the multi-computer-room copy placement strategy.
It should be understood that the electronic device may be a personal computer, tablet computer, smart phone, or other electronic device with logic computing functionality; the readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk, an optical disk, or the like.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not substantially depart from the spirit and scope of the present invention as set forth in the appended claims.
Claims (10)
1. A computer room data migration method based on a multi-computer room copy placement strategy is characterized by comprising the following steps:
modifying a copy placing strategy of a metadata node of an old machine room into a multi-machine-room copy placing strategy, and setting a machine room written in the multi-machine-room copy placing strategy as the old machine room;
newly adding a data node in a new machine room, reporting the data node to a metadata node of the old machine room, and writing a copy placing strategy of the multiple machine rooms into the machine rooms to set the multiple machine rooms as the old machine room and the new machine room; the number of the newly added data nodes in the new machine room is the number of the data nodes in the old machine room;
migrating copies of all file blocks of the migrated files to data nodes of the new machine room based on an AzMover migration method;
and taking the data nodes of the old machine room off the line, and migrating the metadata nodes of the old machine room to the new machine room to finish the data migration of the machine room.
2. The method according to claim 1, wherein migrating copies of file blocks of the migrated file to data nodes of the new computer room based on the AzMover migration method includes:
for each file block of the migrated file, determining a data node set where a copy of each file block is located, and determining a copy source node and a node of a copy to be deleted, wherein the node of the copy to be deleted is a data node with the smallest residual storage space in the data node set;
determining a target node from the data nodes of the new machine room based on the principles of load balancing and storage balancing;
copying the copies of the file blocks of the migrated file from the replication source node to the target node, and deleting the corresponding copies in the nodes of which the copies are to be deleted.
3. The method according to claim 2, wherein the determining a target node from the data nodes of the new computer room based on the principles of load balancing and storage balancing comprises:
randomly selecting two data nodes from the data nodes of the new machine room as candidate nodes aiming at the HDFS single cluster with low concurrent migration file blocks, and determining the nodes with more residual storage spaces in the candidate nodes as target nodes;
aiming at the HDFS single cluster with uniformly stored data nodes, arranging the data nodes of the new machine room in a descending order according to the remaining storage space at intervals, selecting a first number of data nodes with front positions as target nodes, and setting concurrence of the target nodes to be the same value;
and aiming at the single HDFS cluster with the number of data nodes conforming to the large cluster scale, arranging the data nodes of the new computer room in a descending order according to the residual storage space, selecting a second number of data nodes at the front position as a candidate node set, randomly selecting a third number of data nodes from the candidate node set as target nodes, and setting the concurrence of the target nodes to be the same value.
4. The method according to claim 2, wherein the replication source node is provided with a traffic configuration item for controlling traffic of the replica copies.
5. The method of claim 2, wherein determining a replication source node comprises:
if the new computer room does not have a copy of the file block to be migrated, taking the data node with the lowest load in the data node set as a copy source node;
and if the new machine room has the copy of the file block to be migrated, taking the data node of the copy of the file block to be migrated in the new machine room as a replication source node.
6. The method of claim 1, wherein before modifying the replica placement policy of the metadata node of the old computer room to the multi-computer-room replica placement policy, the method further comprises:
defining a network topology class, and constructing a network topology of the HDFS based on the network topology class; and the network topology of the HDFS is used for enabling the metadata node of the old machine room to sense the multi-machine room copy placement strategy.
7. The method according to claim 1, wherein the multi-machine room copy placement strategy is used for selecting a data node of a target machine room to write in a copy according to the multi-machine room copy placement strategy of the file to be deployed, deleting the copy of the data node of the target machine room according to the multi-machine room copy placement strategy of the file to be deployed, and controlling the number of times of crossing private lines during copy writing.
8. A machine room data migration device based on a multi-machine room copy placement strategy is characterized by comprising:
the system comprises a first setting module, a second setting module and a third setting module, wherein the first setting module is used for modifying a copy placing strategy of a metadata node of an old machine room into a multi-machine-room copy placing strategy and writing a machine room in the multi-machine-room copy placing strategy into the old machine room;
the second setting module is used for newly adding a data node in a new machine room, reporting the newly added data node to a metadata node of the old machine room, and writing the multi-machine-room copy placing strategy into the machine room to set the machine room as the old machine room and the new machine room; the number of the newly added data nodes in the new machine room is the number of the data nodes in the old machine room;
the copy migration module is used for migrating the copies of all the file blocks of the migrated files to the data nodes of the new computer room based on the AzMover migration method;
and the service migration module is used for taking the data nodes of the old machine room off line and migrating the metadata nodes of the old machine room to the new machine room to finish the data migration of the machine room.
9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method for migrating data of multiple computer rooms based on the multiple computer rooms duplicate placement strategy according to any one of claims 1 to 7.
10. A readable storage medium, in which a computer program is stored, which, when run on a processor, performs the method for multi-room replica placement policy based room data migration according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310258349.4A CN115982101B (en) | 2023-03-17 | 2023-03-17 | Machine room data migration method and device based on multi-machine room copy placement strategy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310258349.4A CN115982101B (en) | 2023-03-17 | 2023-03-17 | Machine room data migration method and device based on multi-machine room copy placement strategy |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115982101A true CN115982101A (en) | 2023-04-18 |
| CN115982101B CN115982101B (en) | 2023-06-02 |
Family
ID=85972488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310258349.4A Active CN115982101B (en) | 2023-03-17 | 2023-03-17 | Machine room data migration method and device based on multi-machine room copy placement strategy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115982101B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117742604A (en) * | 2023-12-20 | 2024-03-22 | 北京火山引擎科技有限公司 | Data storage control method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103631894A (en) * | 2013-11-19 | 2014-03-12 | 浪潮电子信息产业股份有限公司 | Dynamic copy management method based on HDFS |
| CN104468756A (en) * | 2014-11-27 | 2015-03-25 | 中国联合网络通信集团有限公司 | Method and device for achieving load distribution in PaaS platform |
| CN104615606A (en) * | 2013-11-05 | 2015-05-13 | 阿里巴巴集团控股有限公司 | Hadoop distributed file system and management method thereof |
| CN106055698A (en) * | 2016-06-14 | 2016-10-26 | 智者四海（北京）技术有限公司 | Data migration method, agent node and database instance |
| CN111078121A (en) * | 2018-10-18 | 2020-04-28 | 深信服科技股份有限公司 | Data migration method, system and related components of distributed storage system |
| US20210390078A1 (en) * | 2020-06-10 | 2021-12-16 | Wandisco Inc. | Methods, devices and systems for migrating an active filesystem |
| CN114385561A (en) * | 2022-01-10 | 2022-04-22 | 北京沃东天骏信息技术有限公司 | File management method and device and HDFS system |
-
2023
- 2023-03-17 CN CN202310258349.4A patent/CN115982101B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615606A (en) * | 2013-11-05 | 2015-05-13 | 阿里巴巴集团控股有限公司 | Hadoop distributed file system and management method thereof |
| CN103631894A (en) * | 2013-11-19 | 2014-03-12 | 浪潮电子信息产业股份有限公司 | Dynamic copy management method based on HDFS |
| CN104468756A (en) * | 2014-11-27 | 2015-03-25 | 中国联合网络通信集团有限公司 | Method and device for achieving load distribution in PaaS platform |
| CN106055698A (en) * | 2016-06-14 | 2016-10-26 | 智者四海（北京）技术有限公司 | Data migration method, agent node and database instance |
| CN111078121A (en) * | 2018-10-18 | 2020-04-28 | 深信服科技股份有限公司 | Data migration method, system and related components of distributed storage system |
| US20210390078A1 (en) * | 2020-06-10 | 2021-12-16 | Wandisco Inc. | Methods, devices and systems for migrating an active filesystem |
| CN114385561A (en) * | 2022-01-10 | 2022-04-22 | 北京沃东天骏信息技术有限公司 | File management method and device and HDFS system |
Non-Patent Citations (1)
| Title |
|---|
| 孙嘉冬: "交管信息系统分布式数据迁移研究与实现", 《中国优秀硕士论文电子期刊网》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117742604A (en) * | 2023-12-20 | 2024-03-22 | 北京火山引擎科技有限公司 | Data storage control method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115982101B (en) | 2023-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9542296B1 (en) | Disk replacement using a predictive statistical model | |
| US8984243B1 (en) | Managing operational parameters for electronic resources | |
| US10372561B1 (en) | Block storage relocation on failure | |
| US10242013B2 (en) | Snapshot management using heatmaps in a large capacity disk environment | |
| US20210255994A1 (en) | Intelligent file system with transparent storage tiering | |
| JP5118243B2 (en) | Realization of reliable access to non-local block data storage by executing programs | |
| US8930364B1 (en) | Intelligent data integration | |
| US9773012B2 (en) | Updating map structures in an object storage system | |
| US11645237B2 (en) | Replicating data utilizing a virtual file system and cloud storage | |
| US11262918B1 (en) | Data storage system with uneven drive wear reduction | |
| US20220075757A1 (en) | Data read method, data write method, and server | |
| US11099768B2 (en) | Transitioning from an original device to a new device within a data storage array | |
| US20150205531A1 (en) | Adding Storage Capacity to an Object Storage System | |
| CN115982101B (en) | Machine room data migration method and device based on multi-machine room copy placement strategy | |
| US9037762B2 (en) | Balancing data distribution in a fault-tolerant storage system based on the movements of the replicated copies of data | |
| CN115955488B (en) | Distributed storage copy cross-machine room placement method and device based on copy redundancy | |
| US11336723B1 (en) | Replicating data volume updates from clients accessing the data volume across fault tolerance zones | |
| CN114930281A (en) | Dynamic adaptive partition partitioning | |
| CN111400098B (en) | Copy management method and device, electronic equipment and storage medium | |
| CN115827590A (en) | Data migration and cloud method, computing equipment and computer storage medium | |
| CN112685130A (en) | Virtual machine backup method and device in distributed storage environment and storage medium | |
| CN114415978B (en) | Multi-cloud cluster data reading and writing method and device, storage medium and electronic equipment | |
| GB2443442A (en) | Automated redundancy control and recovery mechanisms in a clustered computing system | |
| CN116340262A (en) | Data processing method, system, equipment and medium | |
| Skipwith | Reliable Scalable Storage: Solution for cloud native applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |