CN115982101B - Machine room data migration method and device based on multi-machine room copy placement strategy - Google Patents
Machine room data migration method and device based on multi-machine room copy placement strategy Download PDFInfo
- Publication number
- CN115982101B CN115982101B CN202310258349.4A CN202310258349A CN115982101B CN 115982101 B CN115982101 B CN 115982101B CN 202310258349 A CN202310258349 A CN 202310258349A CN 115982101 B CN115982101 B CN 115982101B
- Authority
- CN
- China
- Prior art keywords
- machine room
- data
- nodes
- node
- copy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013508 migration Methods 0.000 title claims abstract description 137
- 230000005012 migration Effects 0.000 title claims abstract description 137
- 238000000034 method Methods 0.000 title claims abstract description 85
- 238000004590 computer program Methods 0.000 claims description 12
- 230000010076 replication Effects 0.000 claims description 6
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a machine room data migration method and device based on a multi-machine room copy placement strategy, wherein the method comprises the following steps: modifying a copy placement strategy of a metadata node of an old machine room into a multi-machine room copy placement strategy, and setting a write machine room in the multi-machine room copy placement strategy as the old machine room; reporting a new data node to the metadata node of the old machine room in a new machine room, and setting the write machine room in the multi-machine room copy placement strategy as the old machine room and the new machine room; based on an AzMover migration method, copies of all file blocks of a migration file are migrated to a data node of the new machine room; and the data nodes of the old machine room are offline, and the metadata nodes of the old machine room are migrated to the new machine room, so that the data migration of the machine room is completed. The multi-machine-room copy placement strategy and the AzMover migration method are adopted, so that excessive cross-machine-room traffic is avoided, and the accurate control of bandwidth traffic is realized.
Description
Technical Field
The application relates to the technical field of computers, in particular to a computer room data migration method and device based on a multi-computer-room copy placement strategy.
Background
With the advent of the cloud primary age, the multi-cloud multi-machine room has become a mainstream architecture of various large internet and science and technology companies. The basic cloud service provided by the cloud manufacturer can enable a served party to avoid a large number of basic operation and maintenance, so that research and development business is focused and development is rapid. However, the multi-cloud architecture also brings about a plurality of problems, such as poor service quality of cloud manufacturers, contract expiration renewal and price increase, and the like. In the face of such problems, a more common solution is to replace the cooperating cloud vendors, but this requires for the business company to migrate all data and services to the room provided by the new cloud vendor.
Currently, for the mainstream distributed storage component HDFS (Hadoop Distributed File System ), a Distcp tool is generally used for data migration, and data migration based on the Distcp tool mainly includes: and copying the stock data of the old machine room to the new machine room by adopting a Distcp tool, stopping the service of the old machine room after the copying of the stock data is completed, copying the incremental data and migrating all the services. However, migrating data based on Distcp has the following disadvantages: the consistency of the data is difficult to ensure, and the migrated data needs to be verified at a long time to prevent the user from updating or deleting; when copying data, the flow rate across the machine room is difficult to control, and the bandwidth of a special line is easy to fill; when the cluster is cut, the cluster needs to be cut off, and when the cluster is cut off, copying incremental data and migrating all upper-layer services based on the HDFS can take a long time, so that the cut off time is long, the risk is uncontrollable, and the cut off failure is easy to cause.
Disclosure of Invention
The application provides a machine room data migration method and device based on a multi-machine room copy placement strategy, which are used for solving the problems that data consistency cannot be guaranteed, flow across a machine room cannot be controlled, risk of stopping service cannot be controlled and the like in the prior art.
In a first aspect, the present application provides a machine room data migration method based on a multi-machine room copy placement policy, where the method includes:
modifying a copy placement strategy of a metadata node of an old machine room into a multi-machine room copy placement strategy, and setting a write machine room in the multi-machine room copy placement strategy as the old machine room;
reporting a new data node to the metadata node of the old machine room in a new machine room, and setting the write machine room in the multi-machine room copy placement strategy as the old machine room and the new machine room; the number of newly added data nodes in the new machine room is the number of data nodes in the old machine room;
based on an AzMover migration method, copies of all file blocks of a migration file are migrated to a data node of the new machine room;
and the data nodes of the old machine room are offline, and the metadata nodes of the old machine room are migrated to the new machine room, so that the data migration of the machine room is completed.
According to the machine room data migration method based on the multi-machine room copy placement strategy provided by the application, the AzMover migration method is used for migrating copies of all file blocks of a migration file to a data node of the new machine room, and comprises the following steps: determining a data node set where copies of all file blocks are located according to all file blocks of the migration file, and determining a copy source node and a node for deleting the copies, wherein the node for deleting the copies is a data node with the minimum residual storage space in the data node set; determining a target node from the data nodes of the new machine room based on the principles of load balancing and storage balancing; copying copies of all file blocks of the migration file from the copy source node to the target node, and deleting corresponding copies in the nodes of which the copies are to be deleted.
According to the machine room data migration method based on the multi-machine room copy placement strategy provided by the application, the method for determining the target node from the data nodes of the new machine room based on the load balancing and storage balancing principles comprises the following steps: for the HDFS single cluster of the low concurrency migration file block, two data nodes are randomly selected from the data nodes of the new machine room to serve as candidate nodes, and the nodes with more residual storage spaces in the candidate nodes are determined to serve as target nodes; aiming at the data nodes, storing uniform HDFS single clusters, arranging the data nodes of the new machine room in a descending order of the residual storage space at intervals, selecting a first number of data nodes with front positions as target nodes, and setting concurrency of the target nodes to be the same value; aiming at an HDFS single cluster with the data node number conforming to the large cluster scale, the data nodes of the new machine room are arranged in descending order according to the residual storage space, a second number of data nodes with the front positions are selected as a candidate node set, a third number of data nodes are randomly selected from the candidate node set as target nodes, and the concurrency of the target nodes is set to be the same value.
According to the machine room data migration method based on the multi-machine room copy placement strategy, the copy source node is provided with the flow configuration item, and the flow configuration item is used for controlling the flow of copy.
According to the machine room data migration method based on the multi-machine room copy placement strategy, the method for determining the copy source node comprises the following steps: if the new machine room does not have the copy of the file block to be migrated, taking the data node with the lowest load in the data node set as a copy source node; and if the new machine room has the copy of the file block to be migrated, taking the data node of the copy of the file block to be migrated in the new machine room as a copy source node.
According to the machine room data migration method based on the multi-machine room copy placement strategy, before the copy placement strategy of the metadata node of the old machine room is modified into the multi-machine room copy placement strategy, the method further comprises the following steps: defining a network topology class, and constructing the network topology of the HDFS based on the network topology class; the network topology of the HDFS is configured to enable metadata nodes of the old machine room to perceive the multi-machine-room copy placement policy.
According to the machine room data migration method based on the multi-machine room copy placement strategy, the multi-machine room copy placement strategy is used for selecting the data node write-in copy of the target machine room according to the multi-machine room copy placement strategy of the file to be deployed, deleting the copy of the data node of the target machine room according to the multi-machine room copy placement strategy of the file to be deployed, and controlling the number of times of crossing special lines when the copy is written.
In a second aspect, the present application further provides a machine room data migration apparatus based on a multi-machine room copy placement policy, where the apparatus includes:
the first setting module is used for modifying the copy placement strategy of the metadata nodes of the old machine room into a multi-machine room copy placement strategy, and setting a write-in machine room in the multi-machine room copy placement strategy as the old machine room;
the second setting module is used for adding a data node in a new machine room, reporting the data node to the metadata node of the old machine room, and setting the write-in machine room in the multi-machine room copy placement strategy as the old machine room and the new machine room; the number of newly added data nodes in the new machine room is the number of data nodes in the old machine room;
the copy migration module is used for migrating the copies of all file blocks of the migration file to the data nodes of the new machine room based on an AzMover migration method;
and the service migration module is used for downloading the data nodes of the old machine room, and migrating the metadata nodes of the old machine room to the new machine room to complete the data migration of the machine room.
In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores a computer program, and when the processor runs the computer program, the processor executes steps in any implementation manner of the computer room data migration method based on the multiple-computer room copy placement policy.
In a fourth aspect, an embodiment of the present application further provides a readable storage medium, where a computer program is stored, where the computer program executes steps in any implementation manner of the computer room data migration method based on the multi-room copy placement policy, when the computer program runs on a processor.
In summary, the computer room data migration method and device based on the multi-computer room copy placement strategy, by constructing the network topology of the HDFS and modifying the copy placement strategy into the multi-computer room copy placement strategy, the HDFS can quickly sense the computer room in which the copy is placed; the incremental data is written into the new machine room and the old machine room at the same time, so that the flow across the machine rooms is further avoided; by adopting the AzMover migration method, a single copy of each file block is copied across the machine room, and then the other copies of each file block are copied with the machine room, so that excessive flow across the machine room is avoided, and the machine room data migration is completed under the condition of no stopping service; meanwhile, a copy source node, a node of which the copy is to be deleted and a target node are determined based on load balancing and storage balancing, so that the load balancing and the storage balancing of the HDFS are ensured; the flow configuration items of the copy source nodes are adjusted, so that the accurate control of bandwidth flow is realized, the full bandwidth can be avoided, and the migration efficiency is improved.
Drawings
For a clearer description of the present application or of the prior art, the drawings that are used in the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description below are some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a computer room data migration method based on a multi-computer room copy placement strategy provided in the present application;
FIG. 2 is a schematic flow chart of migrating copies of each file block of a migration file to a data node of the new machine room based on the AzMover migration method provided by the application;
fig. 3 is a schematic structural diagram of a computer room data migration device based on a multi-computer room copy placement strategy provided in the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Icon: 300-a machine room data migration device; 310-a first setting module; 320-a second setting module; 330-copy migration module; 340-a service migration module; 400-an electronic device; 410-a memory; 420-a processor; 430-bus.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is apparent that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
Fig. 1 is a flow chart of a machine room data migration method based on a multi-machine room copy placement strategy, which is provided by the application, wherein the method is applied to an HDFS single cluster, the HDFS single cluster includes a Client, a metadata node (NameNode), and a data node (DataNode), the metadata node is disposed in a main machine room of the HDFS single cluster, and the data node is disposed in a plurality of machine rooms. The metadata node is a master node in the HDFS single cluster, which is used for receiving metadata requests initiated by users of each client, such as checking a directory, creating a new directory, deleting a directory, and the like, and does not provide real data read-write services. The data node is a node for actually storing data in the HDFS single cluster, and receives a request of a user for reading and writing the data, such as reading a file, writing the file and the like. In addition, there are typically a plurality of metadata nodes in the HDFS single cluster, and a primary-backup mode is adopted, that is, the metadata nodes are divided into a primary metadata node (Active NameNode) and a plurality of backup metadata nodes (Standby namenodes) to provide high available services.
It should be noted that when the HDFS stores a file, the file is split into a plurality of file blocks with a fixed size (e.g. 64M and 128M), each file block may copy a plurality of (typically three) identical copies, and copies of each file corresponding to each file are stored on different data nodes according to a copy placement policy of the file, where the file blocks are mutually independent.
As shown in fig. 1, the computer room data migration method based on the multi-computer room copy placement strategy includes:
s101, modifying a copy placement strategy of metadata nodes of an old machine room into a multi-machine room copy placement strategy, and setting a copy writing machine room in the multi-machine room copy placement strategy as the old machine room.
The multi-machine-room copy placement strategy is used for selecting the data node write copy of the target machine room according to the multi-machine-room copy placement strategy of the file to be deployed, deleting the copy of the data node of the target machine room according to the multi-machine-room copy placement strategy of the file to be deployed, and controlling the number of times of the copy write crossing special lines.
Specifically, it may be understood that the multi-machine-room copy placement policy can sense the machine room where the copy is placed, and select the data node of the target machine room to write the copy or delete the copy of the data node of the target machine room strictly according to the copy placement policy of the file to be deployed, for example, the multi-machine-room copy placement policy of the file to be deployed is "az1, az1, az2", when writing the copy of the file to be deployed, the data node needs to be determined and written in the machine room 1, and the machine room 2 strictly according to the order of the multi-machine-room copy placement policy in a pipeline form, so that the write of the file is only performed once by a special line, and the number of times of the special line is reduced as much as possible; when deleting the copies of the file, the copy is also required to be deleted according to a multi-machine-room copy placement strategy, for example, when deleting one copy, the copy placed in the machine room 2 is deleted, and when deleting two copies, the copy placed in the machine room 2 is deleted, and then the copy in the machine room 1 is deleted.
It should be noted that, the copy placement policy of the HDFS single cluster original generally adopts a random placement manner, and when writing a file, adding a copy of the file, and recovering the copy of the file, the metadata node determines the target data node according to the chooseTarget method in the org. Therefore, before migrating the computer room data, the copy placement policy needs to be modified into a multi-computer room copy placement policy.
Specifically, it is further understood that in the HDFS default implementation, all data nodes may be placed on a default rack, and the metadata node may not be aware of the rack on which each data node is located. Therefore, it is necessary to construct a network topology of HDFS so that metadata nodes perceive the distribution of data nodes in order to implement a multi-machine-room copy placement strategy.
In some embodiments, before step S101, the computer room data migration method based on the multi-computer room copy placement policy further includes: defining a network topology class, and constructing the network topology of the HDFS based on the network topology class; the network topology of the HDFS is configured to enable metadata nodes of the old machine room to perceive the multi-machine-room copy placement policy. The defining network topology class and constructing the network topology of the HDFS based on the network topology class specifically comprises the following steps:
step a1, defining a network topology class LocalFileDNSToSwitchmapping;
step a2, writing the association relation between the data node and the rack and the machine room into a local file;
and a step a3 of analyzing the local file by using the network topology class localfileDNSToSwitchmapping to obtain the network topology of the HDFS.
According to the method for constructing the network topology of the HDFS based on the network topology class, the problems that call speed of a shell script is uncontrollable, analysis performance is poor and the like in the network topology of the HDFS constructed by adopting the original script based mapping of the HDFS in the prior art are solved while a multi-machine-room copy placement strategy is supported, and query is carried out through a local file storing the association relation between a data node and a rack as well as a machine room, so that even if a client node incapable of being analyzed is encountered, a result can be returned quickly, and subsequent analysis is not affected.
In the implementation process of the method, the network topology of the HDFS is constructed through the defined network topology class, so that the HDFS can sense the rack and the machine room where each data node is located, and the query speed is improved; by modifying the copy placement strategy of the metadata node into the multi-machine room copy placement strategy, the HDFS can sense the machine room in which the copy is placed, and subsequent copy migration and other operations are facilitated.
S102, a new data node is added in a new machine room, the new data node is reported to the metadata node of the old machine room, and copies in the multi-machine room copy placement strategy are written into the machine room to be set as the old machine room and the new machine room.
The number of the newly added data nodes in the new machine room is the number of the data nodes in the old machine room.
Specifically, it can be understood that, in order to migrate all the data of the old machine room to the new machine room, the data nodes with the same data volume are newly added in the new machine room so as to receive the data of the old machine room, and meanwhile, the metadata nodes of the old machine room are reported, so that the newly added data nodes of the new machine room and the old machine room are under the same HDFS single cluster.
Specifically, it may be further understood that, while a new machine room adds a new data node, the copy writing machine room in the multi-machine room copy placement policy is set to the old machine room and the new machine room, so that during the data migration process of the machine room, the copy of the incremental data generated in the old machine room can be written into the new machine room and the old machine room at the same time, thereby avoiding copying the incremental data again after migration. It is worth noting that any task request in the process needs to run in the new machine room and the old machine room simultaneously so as to write incremental data into the two machine rooms simultaneously, and if the task request is a read request, the new machine room is required to read a file copy placed in the new machine room, and the old machine room is required to read a file copy placed in the old machine room, so that the flow across the machine rooms is prevented.
In the implementation process of the method, the copy writing machine room in the multi-machine room copy placement strategy is set to be the old machine room and the new machine room, so that incremental data generated in the migration process can be written into two machine rooms at the same time, and the copy of each machine room can be read during reading, so that the flow across the machine rooms is prevented.
S103, based on an AzMover migration method, copies of all file blocks of the migration file are migrated to the data nodes of the new machine room.
The AzMover migration method is a method for carrying out copy migration on a file block layer on the basis of an AzMover tool which is self-designed and developed, and can be used for migrating data of an old machine room to a new machine room and migrating file block copies of all files under one or more directories to data nodes of a target machine room according to a multi-machine-room copy placement strategy of the files. In addition, the AzMover migration method can stably and accurately control bandwidth flow, especially cross-machine room bandwidth by adjusting the flow configuration item of the data node.
The AzMover tool comprises a FileVistor component and a MoverExecutor executor. For a certain HDFS single cluster, the AzMover migration method specifically includes the following steps:
step b1, receiving at least one migration path, and analyzing by adopting a FileVistor component to obtain a file list of migration files;
specifically, the migration path is a path of a file or a directory, and after the AzMover tool receives the migration path, the FileVistor component analyzes the migration path to obtain a file list.
Step b2, acquiring meta information of the migration file from a metadata node based on the file list of the migration file;
specifically, the metadata node records the meta information of all files and directories, and the meta information of the corresponding file can be directly obtained from the metadata node based on the file list of the migration file.
Step b3, analyzing by adopting a FileVistor component based on the meta information of the migration file to obtain a file block list of the migration file;
and b4, based on the file block list of the migration file, migrating copies of all file blocks of the migration file by using a MoverExecutor executor.
Specifically, determining a data node set where a corresponding copy is located for each file block to be migrated, taking the data node with the lowest load as a copy source node, and taking the data node with the smallest residual storage space as a node for deleting the copy; determining a target machine room according to a multi-machine room copy placement strategy of a file corresponding to each file block to be migrated, and taking a data node with the largest storage space in the target machine room as a target node; copying the copy of each file block to be migrated from the copy source node to the target node, and deleting the corresponding copy in the nodes of which the copies are to be deleted.
It should be noted that, the AzMover migration method supports concurrency, and can greatly improve the migration efficiency of the copy. In addition, the AzMover migration method not only can be used for migrating file block copies, but also can be used for migrating EC (Erasure Coding) copies corresponding to files, and modifying the copy number when the file copy number of an old machine room does not support excessive read requests, so that the HDFS can better support the read and write requests of users. However, when the EC copies corresponding to the migration files are migrated, all the EC copies corresponding to the migration files need to be copied because the copies of each file block of the EC files are different, and at this time, the flow rate across the machine room cannot be optimized.
It should be noted that, in the process of migrating the copies of the file blocks of the migrated file to the data node of the new machine room, in order to ensure that the client of the old machine room can read each file normally, at least one file copy, that is, a corresponding copy of each file block, needs to be ensured to be left in the old machine room in the migration process, so that the client of the old machine room can read the copy of the file of the machine room normally, thereby avoiding the generation of flow across the machine rooms.
In the implementation process of the method, copy migration of the file blocks is performed based on an AzMover migration method, and compared with the prior art, the copy migration is performed on the file block level based on the inventory data of the Distcp copying old machine room, and the consistency of the files is not required to be checked; determining a copy source node, a node of a copy to be deleted and a target node based on load balancing and storage balancing, so that the load balancing and the storage balancing of the HDFS are ensured; meanwhile, the bandwidth flow can be accurately controlled.
S104, the data nodes of the old machine room are offline, and the metadata nodes of the old machine room are migrated to the new machine room, so that the data migration of the machine room is completed.
Specifically, it can be understood that after the file block copies of the data nodes of the old machine room are copied into the data nodes of the new machine room, the data nodes of the old machine room are taken off line. And meanwhile, the metadata nodes are migrated to a new machine room, and the data migration of the machine room is completed. It is noted that in the process of migrating the metadata node, the migration can be completed without stopping service by switching the active and standby modes of the metadata node.
In some embodiments, while the data node of the old machine room is being taken off line, the method further comprises: and setting the write-in machine room in the multi-machine room copy placement strategy as the new machine room.
According to the machine room data migration method based on the multi-machine room copy placement strategy, the network topology of the HDFS is built, and the copy placement strategy is modified into the multi-machine room copy placement strategy, so that the HDFS can quickly sense the machine room in which the copy is placed; the incremental data is written into the new machine room and the old machine room at the same time, so that the flow across the machine rooms is further avoided; the copy of the file block is migrated by adopting the AzMover migration method, and the copy source node, the node of which the copy is to be deleted and the target node are determined based on load balancing and storage balancing, so that the load balancing and storage balancing of the HDFS are ensured, and the AzMover migration method can also accurately control the bandwidth flow, thereby realizing the completion of computer room data migration under the condition of no stop.
Fig. 2 is a schematic flow chart of migrating copies of each file block of a migration file to a data node of the new machine room based on the AzMover migration method provided in the present application. As shown in fig. 2, the migration method based on AzMover migrates copies of file blocks of a migration file to a data node of the new machine room, including:
s201, determining a data node set where copies of all file blocks are located for all file blocks of the migration file, and determining a copy source node and a node of the copy to be deleted.
The nodes of the copies to be deleted are data nodes with the minimum remaining storage space in the data node set; the replication source node is provided with a flow configuration item, and the flow configuration item is used for controlling the flow of replication copy; determining a replication source node, comprising: if the new machine room does not have the copy of the file block to be migrated, taking the data node with the lowest load in the data node set as a copy source node; and if the new machine room has the copy of the file block to be migrated, taking the data node of the copy of the file block to be migrated in the new machine room as a copy source node.
Specifically, it can be understood that, since the machine room data migration is to migrate the data of the old machine room to the new machine room, the data node set for a certain file block is the data node set storing the copy of the file block in the old machine room, and the node to be deleted copy is the data node with the minimum remaining storage space.
In particular, it will also be appreciated that the meaning of the traffic profile is that each data node is the maximum bandwidth that can be used at the copy. Because the copying process is divided into copying across the machine room and copying with the same machine room, if copying across the machine room, the bandwidth flow cannot fill the special line bandwidth, and the migration efficiency is affected by too small bandwidth, so the parameter value of the flow configuration item can be determined according to the special line bandwidth, and the specific calculation formula is as follows: parameter value of flow configuration item = target dedicated line bandwidth/target node data; if the data nodes are copied in the same machine room, the copying bandwidth is almost unlimited, so that the parameter value of the flow configuration item is not limited and is high enough under the condition of ensuring that the load of each data node is not too high.
In some embodiments, if the copy is a copy across a machine room and 128 target nodes are selected for writing, and if the target private line bandwidth is 512Gb/s, the parameter value of the flow configuration item is set to 512MB/s, so that the private line bandwidth can be ensured to be kept to be fluctuated at 512Gb/s when the copy is copied, the private line bandwidth can be prevented from being full, and the migration efficiency is ensured. Wherein, 1 GB/s=8 Gb/s,1 GB/s=1024 MB/s.
In some embodiments, if the copy is the same machine room copy and 1000 target nodes are selected for writing, and the concurrence of each target node is 1, it is assumed that the parameter value of the flow configuration item is set to 1GB/s, and through the experiment of data migration of the actual machine room, it is found that the bandwidth flow of the whole HDFS cluster can be kept at 100-200 GB/s, and data above 10PB can be copied in one day.
S202, determining a target node from the data nodes of the new machine room based on the principles of load balancing and storage balancing.
Specifically, it can be understood that the load balancing is that the read-write requests of all data nodes are as uniform as possible and do not affect the normal read-write of a user, and the storage balancing is that copies of file blocks are migrated to data nodes with relatively idle storage as much as possible. The method for determining the target node under different scenes is designed by analyzing the number and the storage condition of the data nodes of the HDFS single cluster, and is specifically as follows:
(1) And for the HDFS single cluster of the low concurrency migration file block, randomly selecting two data nodes from the data nodes of the new machine room as candidate nodes, and determining the nodes with more residual storage spaces in the candidate nodes as target nodes.
Specifically, for the situation that the concurrency of the AzMover migration method is low, in order to ensure load balancing and storage balancing, two data nodes are selected randomly to serve as candidate nodes, and then the data nodes with more residual storage space are determined to serve as target nodes.
(2) And aiming at the data nodes, storing uniform HDFS single clusters, arranging the data nodes of the new machine room in a descending order of the residual storage space at intervals, selecting a first number of data nodes with front positions as target nodes, and setting the concurrency of the target nodes to be the same value.
Specifically, for the case that the data nodes are uniformly stored, determining the target nodes at intervals, namely, firstly arranging the data nodes in descending order according to the residual storage space, and then selecting a first number of data nodes with front positions as the target nodes, wherein the first number is not fixed, and can be adjusted according to actual conditions, such as determining the number of the target nodes according to actual read-write requirements, so as to determine the first number. In the method, the load balance and the storage balance of each data node are ensured through rotating the target node.
In some embodiments, the period of time may be 1min or 2min, and the target node may be determined once every period of time, and the target node may be rotated once every 1min or once every 2min, where the period of time may be specifically adjusted according to practical situations.
(3) Aiming at an HDFS single cluster with the data node number conforming to the large cluster scale, the data nodes of the new machine room are arranged in descending order according to the residual storage space, a second number of data nodes with the front positions are selected as a candidate node set, a third number of data nodes are randomly selected from the candidate node set as target nodes, and the concurrency of the target nodes is set to be the same value.
Specifically, the number of data nodes meeting the large-scale cluster refers to that the number of the data nodes is higher than a certain threshold, for example, clusters with more than 1000 data nodes can be called large-scale clusters; aiming at the situation that the number of data nodes accords with the large cluster scale, the data nodes of the new machine room are arranged in descending order according to the size of the residual storage space, the second number of data nodes with the front positions are selected as candidate node sets, and then the third number of data nodes are randomly selected from the candidate node sets to serve as target nodes.
The third number is not fixed, and can be adjusted according to actual conditions, for example, the number of target nodes is determined according to actual read-write requirements, and then the third number is determined; the second number is adjusted according to the number of data nodes of the HDFS single cluster, and the second number is greater than the third number, in some embodiments, the second number may be 2/3 of the number of data nodes of the HDFS single cluster, for example, 3000 data nodes in total of the HDFS single cluster may be 2000, that is, after the data nodes are arranged in descending order of the remaining storage space, the first 2000 data nodes are selected to form a candidate node set, then a batch of data nodes is randomly selected from the 2000 data node sets to be used as a target node, and if the third number is 500, 500 data nodes may be randomly selected to be used as a target node.
It is worth noting that, aiming at the HDFS single cluster with the number of data nodes conforming to the large cluster scale, the concurrency value of the target node needs to be set in the allowed range, so as to avoid affecting the normal read-write request.
And S203, copying the copies of the file blocks of the migration file from the copy source node to the target node, and deleting the corresponding copy in the node of which the copies are to be deleted.
Specifically, it can be understood that, for each copy of the file block, a copy source node, a node to be deleted and a target node corresponding to the copy source node are respectively determined, then the copy is copied from the copy source node to the target node, and the corresponding copy in the node to be deleted is deleted, so that copy migration is completed once.
Specifically, it may be further understood that the data node for migrating copies of file blocks of the migration file to the new machine room specifically includes: firstly, migrating a single copy of each file block to a data node of the new machine room; and then sequentially migrating the rest copies of each file block to the data nodes of the new machine room.
It should be noted that a single copy of each file block is migrated as a copy across the machine room. At the moment, the data node which stores the copy of the file block to be migrated and has the lowest load in the old machine room is used as the copy source node, so that the normal read-write requirement of a user can not be influenced in the data migration process of the machine room; the data node which stores the copy of the file block to be migrated and has the minimum residual storage space in the old machine room is used as the node for deleting the copy, so that the storage space of the data can be released; meanwhile, the bandwidth flow copied across the machine room can be accurately controlled by copying the flow configuration item of the source node.
It should also be noted that the remaining copies of each file block are migrated as identical house copies. Because the copy of the file block to be migrated exists in the new machine room, the data node with the copy of the file block to be migrated in the new machine room is used as the copy source node, so that the flow of the machine room can be avoided, and the node with the copy to be deleted is still the data node with the copy of the file block to be migrated stored in the old machine room and the residual storage space is the smallest. In addition, if there are multiple data nodes of the copy of the file block to be migrated in the new machine room, the data node with the lowest load may be selected as the copy source node.
According to the AzMover migration method, copies of all file blocks of a migration file are migrated to the data nodes of the new machine room, and the single copy of each file block is copied across the machine room first, and then the other copies of each file block are copied across the machine room, so that excessive flow across the machine room is avoided; the node with the lowest load and the smallest residual storage space is used as a copy to be deleted, and different target node selection methods are designed, so that load balancing and storage balancing of the HDFS are achieved as much as possible in the migration process based on the AzMover migration method; by adjusting the flow configuration items of the copy source nodes, the accurate control of the bandwidth flow is realized, the bandwidth filling can be avoided, and the migration efficiency is improved.
Fig. 3 is a schematic structural diagram of a computer room data migration apparatus based on a multi-computer room copy placement policy provided in the present application, which may be used to implement the method described in the foregoing embodiments. As shown in fig. 3, the apparatus includes:
a first setting module 310, configured to modify a copy placement policy of a metadata node of an old machine room into a multi-machine room copy placement policy, and set a write machine room in the multi-machine room copy placement policy as the old machine room;
the second setting module 320 is configured to add a new data node in a new machine room, report the new data node to the metadata node of the old machine room, and set the write machine room in the multi-machine room copy placement policy as the old machine room and the new machine room; the number of newly added data nodes in the new machine room is the number of data nodes in the old machine room;
the copy migration module 330 is configured to migrate, based on an AzMover migration method, a copy of each file block of the migration file to a data node of the new machine room;
and the service migration module 340 is configured to offline the data node of the old machine room, and migrate the metadata node of the old machine room to the new machine room, thereby completing data migration of the machine room.
For detailed description of the computer room data migration device based on the multi-computer room copy placement strategy, please refer to the description of the related method steps in the above embodiment, and the repetition is omitted. The apparatus embodiments described above are merely illustrative, wherein the "module" as illustrated as a separate component may or may not be physically separate, as may be a combination of software and/or hardware implementing the intended function. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Fig. 4 is a schematic structural diagram of an electronic device provided in the present application, as shown in fig. 4, where the electronic device includes a memory 410 and a processor 420, where the memory 410 stores a computer program, and the processor 420 executes steps in a computer room data migration method based on a multi-room copy placement policy when running the computer program.
The embodiment of the application also provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program executes steps in a computer room data migration method based on a multi-computer room copy placement strategy when running on a processor.
It should be understood that the electronic device may be an electronic device with a logic computing function, such as a personal computer, a tablet computer, a smart phone, etc.; the readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory ), a magnetic disk, an optical disk, or the like.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and they should not fall within the scope of the present invention.
Claims (8)
1. A machine room data migration method based on a multi-machine room copy placement strategy is characterized by comprising the following steps:
modifying a copy placement strategy of a metadata node of an old machine room into a multi-machine room copy placement strategy, and setting a write machine room in the multi-machine room copy placement strategy as the old machine room;
reporting a new data node to the metadata node of the old machine room in a new machine room, and setting the write machine room in the multi-machine room copy placement strategy as the old machine room and the new machine room; the number of newly added data nodes in the new machine room is the number of data nodes in the old machine room;
based on an AzMover migration method, copies of all file blocks of a migration file are migrated to a data node of the new machine room;
the data nodes of the old machine room are offline, and the metadata nodes of the old machine room are migrated to the new machine room, so that the data migration of the machine room is completed;
the AzMover migration method is used for migrating copies of all file blocks of a migration file to a data node of the new machine room, and comprises the following steps:
determining a data node set where copies of all file blocks are located according to all file blocks of the migration file, and determining a copy source node and a node for deleting the copies, wherein the node for deleting the copies is a data node with the minimum residual storage space in the data node set;
determining a target node from the data nodes of the new machine room based on the principles of load balancing and storage balancing;
copying copies of all file blocks of the migration file from the copy source node to the target node, and deleting corresponding copies in the nodes of which the copies are to be deleted;
the determining a target node from the data nodes of the new machine room based on the load balancing and storage balancing principles comprises the following steps:
for the HDFS single cluster of the low concurrency migration file block, two data nodes are randomly selected from the data nodes of the new machine room to serve as candidate nodes, and the nodes with more residual storage spaces in the candidate nodes are determined to serve as target nodes;
aiming at the data nodes, storing uniform HDFS single clusters, arranging the data nodes of the new machine room in a descending order of the residual storage space at intervals, selecting a first number of data nodes with front positions as target nodes, and setting concurrency of the target nodes to be the same value;
aiming at an HDFS single cluster with the data node number conforming to the large cluster scale, the data nodes of the new machine room are arranged in descending order according to the residual storage space, a second number of data nodes with the front positions are selected as a candidate node set, a third number of data nodes are randomly selected from the candidate node set as target nodes, and the concurrency of the target nodes is set to be the same value.
2. The method according to claim 1, wherein the replication source node is provided with a traffic configuration item for controlling traffic of the replication copy.
3. The method of claim 1, wherein determining the replication source node comprises:
if the new machine room does not have the copy of the file block to be migrated, taking the data node with the lowest load in the data node set as a copy source node;
and if the new machine room has the copy of the file block to be migrated, taking the data node of the copy of the file block to be migrated in the new machine room as a copy source node.
4. The method of claim 1, wherein prior to modifying the replica placement policy of the metadata node of the old machine room to a multiple machine room replica placement policy, the method further comprises:
defining a network topology class, and constructing the network topology of the HDFS based on the network topology class; the network topology of the HDFS is configured to enable metadata nodes of the old machine room to perceive the multi-machine-room copy placement policy.
5. The method of claim 1, wherein the multi-machine-room copy placement policy is used for selecting a data node write copy of a target machine room according to the multi-machine-room copy placement policy of the file to be deployed, deleting a copy of the data node of the target machine room according to the multi-machine-room copy placement policy of the file to be deployed, and controlling the number of times of cross-private lines when the copy is written.
6. A machine room data migration apparatus based on a multi-machine room copy placement policy, the apparatus comprising:
the first setting module is used for modifying the copy placement strategy of the metadata nodes of the old machine room into a multi-machine room copy placement strategy, and setting a write-in machine room in the multi-machine room copy placement strategy as the old machine room;
the second setting module is used for adding a data node in a new machine room, reporting the data node to the metadata node of the old machine room, and setting the write-in machine room in the multi-machine room copy placement strategy as the old machine room and the new machine room; the number of newly added data nodes in the new machine room is the number of data nodes in the old machine room;
the copy migration module is used for migrating the copies of all file blocks of the migration file to the data nodes of the new machine room based on an AzMover migration method;
the service migration module is used for downloading the data nodes of the old machine room and migrating the metadata nodes of the old machine room to the new machine room to finish the data migration of the machine room;
the AzMover migration method is used for migrating copies of all file blocks of a migration file to a data node of the new machine room, and comprises the following steps:
determining a data node set where copies of all file blocks are located according to all file blocks of the migration file, and determining a copy source node and a node for deleting the copies, wherein the node for deleting the copies is a data node with the minimum residual storage space in the data node set;
determining a target node from the data nodes of the new machine room based on the principles of load balancing and storage balancing;
copying copies of all file blocks of the migration file from the copy source node to the target node, and deleting corresponding copies in the nodes of which the copies are to be deleted;
the determining a target node from the data nodes of the new machine room based on the load balancing and storage balancing principles comprises the following steps:
for the HDFS single cluster of the low concurrency migration file block, two data nodes are randomly selected from the data nodes of the new machine room to serve as candidate nodes, and the nodes with more residual storage spaces in the candidate nodes are determined to serve as target nodes;
aiming at the data nodes, storing uniform HDFS single clusters, arranging the data nodes of the new machine room in a descending order of the residual storage space at intervals, selecting a first number of data nodes with front positions as target nodes, and setting concurrency of the target nodes to be the same value;
aiming at an HDFS single cluster with the data node number conforming to the large cluster scale, the data nodes of the new machine room are arranged in descending order according to the residual storage space, a second number of data nodes with the front positions are selected as a candidate node set, a third number of data nodes are randomly selected from the candidate node set as target nodes, and the concurrency of the target nodes is set to be the same value.
7. An electronic device comprising a memory storing a computer program and a processor executing the computer room data migration method according to any one of claims 1 to 5 based on a multi-room copy placement strategy.
8. A readable storage medium, characterized in that a computer program is stored in the readable storage medium, which computer program, when run on a processor, performs the machine room data migration method based on the multi-machine room copy placement strategy of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310258349.4A CN115982101B (en) | 2023-03-17 | 2023-03-17 | Machine room data migration method and device based on multi-machine room copy placement strategy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310258349.4A CN115982101B (en) | 2023-03-17 | 2023-03-17 | Machine room data migration method and device based on multi-machine room copy placement strategy |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115982101A CN115982101A (en) | 2023-04-18 |
| CN115982101B true CN115982101B (en) | 2023-06-02 |
Family
ID=85972488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310258349.4A Active CN115982101B (en) | 2023-03-17 | 2023-03-17 | Machine room data migration method and device based on multi-machine room copy placement strategy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115982101B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117742604A (en) * | 2023-12-20 | 2024-03-22 | 北京火山引擎科技有限公司 | Data storage control method and device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106055698A (en) * | 2016-06-14 | 2016-10-26 | 智者四海（北京）技术有限公司 | Data migration method, agent node and database instance |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615606B (en) * | 2013-11-05 | 2018-04-06 | 阿里巴巴集团控股有限公司 | A kind of Hadoop distributed file systems and its management method |
| CN103631894A (en) * | 2013-11-19 | 2014-03-12 | 浪潮电子信息产业股份有限公司 | Dynamic copy management method based on HDFS |
| CN104468756B (en) * | 2014-11-27 | 2018-01-26 | 中国联合网络通信集团有限公司 | The method and apparatus that load distribution is realized in PaaS platform |
| CN111078121B (en) * | 2018-10-18 | 2024-08-20 | 深信服科技股份有限公司 | Data migration method and system for distributed storage system and related components |
| US11487703B2 (en) * | 2020-06-10 | 2022-11-01 | Wandisco Inc. | Methods, devices and systems for migrating an active filesystem |
| CN114385561A (en) * | 2022-01-10 | 2022-04-22 | 北京沃东天骏信息技术有限公司 | File management method and device and HDFS system |
-
2023
- 2023-03-17 CN CN202310258349.4A patent/CN115982101B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106055698A (en) * | 2016-06-14 | 2016-10-26 | 智者四海（北京）技术有限公司 | Data migration method, agent node and database instance |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115982101A (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210255994A1 (en) | Intelligent file system with transparent storage tiering | |
| CN104615606B (en) | A kind of Hadoop distributed file systems and its management method | |
| US9773012B2 (en) | Updating map structures in an object storage system | |
| CN110413685B (en) | Database service switching method, device, readable storage medium and computer equipment | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| US11645237B2 (en) | Replicating data utilizing a virtual file system and cloud storage | |
| US11262918B1 (en) | Data storage system with uneven drive wear reduction | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN110825704B (en) | Data reading method, data writing method and server | |
| US11099768B2 (en) | Transitioning from an original device to a new device within a data storage array | |
| US20150205531A1 (en) | Adding Storage Capacity to an Object Storage System | |
| CN115982101B (en) | Machine room data migration method and device based on multi-machine room copy placement strategy | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN113204520B (en) | Remote sensing data rapid concurrent read-write method based on distributed file system | |
| US9037762B2 (en) | Balancing data distribution in a fault-tolerant storage system based on the movements of the replicated copies of data | |
| CN115955488B (en) | Distributed storage copy cross-machine room placement method and device based on copy redundancy | |
| US11336723B1 (en) | Replicating data volume updates from clients accessing the data volume across fault tolerance zones | |
| US11966637B1 (en) | Method and system for storing data in portable storage devices | |
| CN112748865A (en) | Method, electronic device and computer program product for storage management | |
| CN114930281A (en) | Dynamic adaptive partition partitioning | |
| CN111400098B (en) | Copy management method and device, electronic equipment and storage medium | |
| CN113986878A (en) | Data writing method, data migration device and electronic equipment | |
| CN115827590A (en) | Data migration and cloud method, computing equipment and computer storage medium | |
| US11121981B1 (en) | Optimistically granting permission to host computing resources | |
| CN112527751B (en) | Data processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |